使用Python 2.7 / Windows运行多处理作业

基于这个答案,我想用Python 2.7 / Windows来运行这个multiprocessing作业:

def main():
    import itertools as it
    from multiprocessing import Pool

    def dothejob(i, j, k):
        print i, j, k

    the_args = it.product(range(100), range(100), range(100))
    pool = Pool(4)

    def jobWrapper(args): 
        return dothejob(*args)

    res = pool.map(jobWrapper, the_args)

if __name__ == '__main__':
    main()    

main()和最后两行是必需的,因为如果没有它们,就有一个众所周知的bug:

这可能意味着你在Windows上,并且忘记在主模块中使用适当的习惯用法:

if __name__ == '__main__':
    ....

但即使如此,我得到这个错误:

在主res = pool.map(jobWrapper,the_args)文件“C: Users User Desktop test.py”中,第14行,在第251行的文件“C: Python27 lib multiprocessing pool.py”在map中返回self.map_async(func,iterable,chunksize).get()文件“C: Python27 lib multiprocessing pool.py”,第558行,在get raise self._value cPickle.PicklingError:Can not pickle :属性查找> 内置函数失败

哪里有一个cPickle在这里介入,为什么这个错误/如何解决它?


所有定义必须在模块范围内:

import itertools as it
from multiprocessing import Pool, freeze_support

def dothejob(i, j, k):
    print i, j, k

def jobWrapper(args): 
    return dothejob(*args)

def main():
    the_args = it.product(range(100), range(100), range(100))
    pool = Pool(4)
    res = pool.map(jobWrapper, the_args)

if __name__ == '__main__':
    freeze_support() #you need this in windows
    main()

您还需要在Windows中调用freeze_support

链接地址: http://www.djcxy.com/p/9423.html

上一篇: Run a multiprocessing job with Python 2.7 / Windows

下一篇: TypeError: 'pcapObject' object is not iterable