Multiprocessing : NULL result without error in PyObject
Here is a sample program where I use multiprocessing. The calculations are done with multiprocessing.Process and the results are collected using multiprocessing.Queue .
#THIS PROGRAM RUNS WITH ~40Gb RAM. (you can reduce a,b,c for less RAM
#but then it works for smaller values)
#PROBLEM OCCURS ONLY FOR HUGE DATA.
from numpy import *
import multiprocessing as mp
a = arange(0, 3500, 5)
b = arange(0, 3500, 5)
c = arange(0, 3500, 5)
a0 = 540. #random values
b0 = 26.
c0 = 826.
def rand_function(a, b, c, a0, b0, c0):
Nloop = 100.
def loop(Nloop, out):
res_total = zeros((700, 700, 700), dtype = 'float')
n = 1
while n <= Nloop:
rad = sqrt((a-a0)**2 + (b-b0)**2 + (c-c0)**2)
res_total += rad
n +=1
out.put(res_total)
out = mp.Queue()
jobs = []
Nprocs = mp.cpu_count()
print "No. of processors : ", Nprocs
for i in range(Nprocs):
p = mp.Process(target = loop, args=(Nloop/Nprocs, out))
jobs.append(p)
p.start()
final_result = zeros((700, 700, 700), dtype = 'float')
for i in range(Nprocs):
final_result = final_result + out.get()
p.join()
test = rand_function(a,b,c,a0, b0, c0)
Here is the error message :
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/queues.py", line 266, in _feed
send(obj)
SystemError: NULL result without error in PyObject_Call
I read here that it is a bug. But I am unable to understand. Can anyone please tell me any way out to calculate huge data using multiprocessing?
Thank you very much
The bug report your reference states that multiprocessing module is unable to push huge arguments to subprocess.
The reason is that it needs to pickle these arguments and store the pickled blob somewhere in memory.
You, however, don't need to pass arrays as arguments.
Possible causes:
loop as a target mp.Queue() as argument Please see http://stevenengelhardt.com/2013/01/16/python-multiprocessing-module-and-closures/ about converting your closure to a class.
Set up full state before you give control to multiprocessing.
链接地址: http://www.djcxy.com/p/78404.html上一篇: 使用CSS时Chrome上的模糊文本