Run a multiprocessing job with Python 2.7 / Windows
Based on this answer, I'd like to run this multiprocessing job with Python 2.7 / Windows:
def main():
import itertools as it
from multiprocessing import Pool
def dothejob(i, j, k):
print i, j, k
the_args = it.product(range(100), range(100), range(100))
pool = Pool(4)
def jobWrapper(args):
return dothejob(*args)
res = pool.map(jobWrapper, the_args)
if __name__ == '__main__':
main()
The main() and the last two lines are necessary because without them, there's the well known bug:
This probably means that you are on Windows and you have forgotten to use the proper idiom in the main module:
if __name__ == '__main__':
....
But even with that, I get this error:
File "C:UsersUserDesktoptest.py", line 14, in main res = pool.map(jobWrapper, the_args) File "C:Python27libmultiprocessingpool.py", line 251, in map return self.map_async(func, iterable, chunksize).get() File "C:Python27libmultiprocessingpool.py", line 558, in get raise self._value cPickle.PicklingError: Can't pickle : attribute lookup > builtin .function failed
Where is there a cPickle involved here and why this error / how to solve it?
All definitions must be on module scope:
import itertools as it
from multiprocessing import Pool, freeze_support
def dothejob(i, j, k):
print i, j, k
def jobWrapper(args):
return dothejob(*args)
def main():
the_args = it.product(range(100), range(100), range(100))
pool = Pool(4)
res = pool.map(jobWrapper, the_args)
if __name__ == '__main__':
freeze_support() #you need this in windows
main()
You also need freeze_support call in windows