Python Multiprocessing Lib Error (AttributeError:
Am getting this error when using the pool.map(funct, iterable) :
AttributeError: __exit__
No Explanation, only stack trace to the pool.py file within the module.
using in this way:
with Pool(processes=2) as pool:
pool.map(myFunction, mylist)
pool.map(myfunction2, mylist2)
I suspect there could be a problem with the picklability (python needs to pickle , or transform list data into byte stream) yet I'm not sure if this is true or if it is how to debug.
EDIT: new format of code that produces this error :
def governingFunct(list):
#some tasks
def myFunction():
# function contents
with closing(Pool(processes=2)) as pool:
pool.map(myFunction, sublist)
pool.map(myFunction2, sublist2)
ERROR PRODUCED:
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
In Python 2.x and 3.0, 3.1 and 3.2, multiprocessing.Pool() objects are not context managers. You cannot use them in a with statement. Only in Python 3.3 and up can you use them as such. From the Python 3 multiprocessing.Pool() documentation:
New in version 3.3: Pool objects now support the context management protocol – see Context Manager Types. __enter__() returns the pool object, and __exit__() calls terminate().
For earlier Python versions, you could use contextlib.closing() , but take into account this'll call pool.close() , not pool.terminate() . Terminate manually in that case:
from contextlib import closing
with closing(Pool(processes=2)) as pool:
pool.map(myFunction, mylist)
pool.map(myfunction2, mylist2)
pool.terminate()
or create your own terminating() context manager:
from contextlib import contextmanager
@contextmanager
def terminating(thing):
try:
yield thing
finally:
thing.terminate()
with terminating(Pool(processes=2)) as pool:
pool.map(myFunction, mylist)
pool.map(myfunction2, mylist2)
with statement is for object that have __enter__ and __exit__ functions, ie Context Manager Types
multiprocessing.Pool is not Context Manager Type. try do the following:
pool = Pool(processes=2)
pool.map(myFunction, mylist)
pool.map(myfunction2, mylist2)