使用Python 2.7 / Windows运行多处理作业
基于这个答案,我想用Python 2.7 / Windows来运行这个multiprocessing
作业:
def main():
import itertools as it
from multiprocessing import Pool
def dothejob(i, j, k):
print i, j, k
the_args = it.product(range(100), range(100), range(100))
pool = Pool(4)
def jobWrapper(args):
return dothejob(*args)
res = pool.map(jobWrapper, the_args)
if __name__ == '__main__':
main()
main()
和最后两行是必需的,因为如果没有它们,就有一个众所周知的bug:
这可能意味着你在Windows上,并且忘记在主模块中使用适当的习惯用法:
if __name__ == '__main__':
....
但即使如此,我得到这个错误:
在主res = pool.map(jobWrapper,the_args)文件“C: Users User Desktop test.py”中,第14行,在第251行的文件“C: Python27 lib multiprocessing pool.py”在map中返回self.map_async(func,iterable,chunksize).get()文件“C: Python27 lib multiprocessing pool.py”,第558行,在get raise self._value cPickle.PicklingError:Can not pickle :属性查找> 内置函数失败
哪里有一个cPickle
在这里介入,为什么这个错误/如何解决它?
所有定义必须在模块范围内:
import itertools as it
from multiprocessing import Pool, freeze_support
def dothejob(i, j, k):
print i, j, k
def jobWrapper(args):
return dothejob(*args)
def main():
the_args = it.product(range(100), range(100), range(100))
pool = Pool(4)
res = pool.map(jobWrapper, the_args)
if __name__ == '__main__':
freeze_support() #you need this in windows
main()
您还需要在Windows中调用freeze_support