out of memory error when reading csv file in chunk
I am processing a csv -file which is 2.5 GB big. The 2.5 GB table looks like this:
columns=[ka,kb_1,kb_2,timeofEvent,timeInterval]
0:'3M' '2345' '2345' '2014-10-5',3000
1:'3M' '2958' '2152' '2015-3-22',5000
2:'GE' '2183' '2183' '2012-12-31',515
3:'3M' '2958' '2958' '2015-3-10',395
4:'GE' '2183' '2285' '2015-4-19',1925
5:'GE' '2598' '2598' '2015-3-17',1915
And I want to groupby ka and kb_1 to get the result like this:
columns=[ka,kb,errorNum,errorRate,totalNum of records]
'3M','2345',0,0%,1
'3M','2958',1,50%,2
'GE','2183',1,50%,2
'GE','2598',0,0%,1
(definition of error Record: when kb_1 != kb_2 , the corresponding record is treated as abnormal record )
My computer, which is ubuntu 12.04, has 16 GB memory and free -m returns
total used free shared buffers cached
Mem: 112809 14476 98333 0 128 10823
-/+ buffers/cache: 3524 109285
Swap:
0 0 0
My python file is called bigData.py
import pandas as pd
import numpy as np
import sys,traceback,os
cksize=98333 # or 1024, either chunk size didn't work at all
try:
dfs = pd.DataFrame()
reader=pd.read_table('data/petaJoined.csv', chunksize=cksize)
for chunk in reader:#when executed this line,error occur!
pass
#temp=tb_createTopRankTable(chunk)
#dfs.append(temp)
#df=tb_createTopRankTable(dfs)
except:
traceback.print_exc(file=sys.stdout)
ipdb> pd.__version__
'0.16.0'
I use the following command to monitor the memory usage:
top
ps -C python -o %cpu,%mem,cmd
Since it takes about 2 seconds to crash, so I can see the mem usage had reached 90% some time, and CPU usage reached 100%
When I excecute python bigData.py , the following error generate:
/usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: UserWarning: Module dateutil was already imported from /usr/local/lib/python2.7/dist-packages/dateutil/__init__.pyc, but /usr/lib/python2.7/dist-packages is being added to sys.path
from pkg_resources import resource_stream
/usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: UserWarning: Module pytz was already imported from /usr/local/lib/python2.7/dist-packages/pytz/__init__.pyc, but /usr/lib/python2.7/dist-packages is being added to sys.path
from pkg_resources import resource_stream
Traceback (most recent call last):
File "bigData.py", line 10, in <module>
for chunk in reader:
File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 691, in __iter__
yield self.read(self.chunksize)
File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 715, in read
ret = self._engine.read(nrows)
File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 1164, in read
data = self._reader.read(nrows)
File "pandas/parser.pyx", line 758, in pandas.parser.TextReader.read (pandas/parser.c:7411)
File "pandas/parser.pyx", line 792, in pandas.parser.TextReader._read_low_memory (pandas/parser.c:7819)
File "pandas/parser.pyx", line 833, in pandas.parser.TextReader._read_rows (pandas/parser.c:8268)
File "pandas/parser.pyx", line 820, in pandas.parser.TextReader._tokenize_rows (pandas/parser.c:8142)
File "pandas/parser.pyx", line 1758, in pandas.parser.raise_parser_error (pandas/parser.c:20728)
CParserError: Error tokenizing data. C error: out of memory
Segmentation fault (core dumped)
or
/usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: UserWarning: Module dateutil was already imported from /usr/local/lib/python2.7/dist-packages/dateutil/__init__.pyc, but /usr/lib/python2.7/dist-packages is being added to sys.path
from pkg_resources import resource_stream
/usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: UserWarning: Module pytz was already imported from /usr/local/lib/python2.7/dist-packages/pytz/__init__.pyc, but /usr/lib/python2.7/dist-packages is being added to sys.path
from pkg_resources import resource_stream
Traceback (most recent call last):
File "bigData.py", line 10, in <module>
for chunk in reader:
File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 691, in __iter__
yield self.read(self.chunksize)
File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 715, in read
ret = self._engine.read(nrows)
File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 1164, in read
data = self._reader.read(nrows)
File "pandas/parser.pyx", line 758, in pandas.parser.TextReader.read (pandas/parser.c:7411)
File "pandas/parser.pyx", line 792, in pandas.parser.TextReader._read_low_memory (pandas/parser.c:7819)
File "pandas/parser.pyx", line 833, in pandas.parser.TextReader._read_rows (pandas/parser.c:8268)
File "pandas/parser.pyx", line 820, in pandas.parser.TextReader._tokenize_rows (pandas/parser.c:8142)
File "pandas/parser.pyx", line 1758, in pandas.parser.raise_parser_error (pandas/parser.c:20728)
CParserError: Error tokenizing data. C error: out of memory
*** glibc detected *** python: free(): invalid pointer: 0x00007f750d2a4c0e ***
====== Backtrace: ========
/lib/x86_64-linux-gnu/libc.so.6(+0x7db26)[0x7f7511529b26]
/usr/local/lib/python2.7/dist-packages/pandas/parser.so(+0x4d5a1)[0x7f750d29d5a1]
/usr/local/lib/python2.7/dist-packages/pandas/parser.so(parser_cleanup+0x15)[0x7f750d29de45]
/usr/local/lib/python2.7/dist-packages/pandas/parser.so(parser_free+0x9)[0x7f750d29e039]
/usr/local/lib/python2.7/dist-packages/pandas/parser.so(+0xb43e)[0x7f750d25b43e]
....
python(PyDict_SetItem+0x49)[0x577749]
python(_PyModule_Clear+0x149)[0x4cafb9]
python(PyImport_Cleanup+0x477)[0x4cb4f7]
python(Py_Finalize+0x18e)[0x549f0e]
python(Py_Main+0x3bc)[0x56b56c]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f75114cd76d]
python[0x41bb11]
======= Memory map: ========
00400000-00670000 r-xp 00000000 08:01 26612 /usr/bin/python2.7
0086f000-00870000 r--p 0026f000 08:01 26612 /usr/b.......
008d9000-008eb000 rw-p 00000000 00:00 0
01ddb000-036f7000 rw-p 00000000 00:00 0 [heap]
7f748c179000-7f74cc17a000 rw-p 00000000 00:00 0
7f7504000000-7f7504021000 rw-p 00000000 00:00 0
7f7504021000-7f7508000000 ---p 00000000 00:00 0
7f750bf83000-7f750c285000 rw-p 00000000 00:00 0
7f750c285000-7f750c586000 rw-p 00000000 00:00 0
7f750c586000-7f750c707000 rw-p 00000000 00:00 0
7f750c707000-7f750c711000 r-xp 00000000 08:01 533205 /usr/local/lib/python2.7/dist-packages/pandas/_testing.so
7f750c711000-7f750c911000 ---p 0000a000 08:01 533205 /usr/local/lib/python2.7/dist-packages/pandas/_testing.so
7f750c911000-7f750c912000 r--p 0000a000 08:01 533205 /usr/local/lib/python2.7/dist-packages/pandas/_testing.so
7f750c912000-7f750c913000 rw-p 0000b000 08:01 533205 /usr/local/lib/python2.7/dist-packages/pandas/_testing.so
7f750c913000-7f750c914000 rw-p 00000000 00:00 0
7f750c914000-7f750c918000 r-xp 00000000 08:01 2331 /lib/x86_64-linux-gnu/libuuid.so.1.3.0
7f750c918000-7f750cb17000 ---p 00004000 08:01 2331 /lib/x86_64-linux-gnu/libuuid.so.1.3.0
7f750cb17000-7f750cb18000 r--p 00003000 08:01 2331 /lib/x86_64-linux-gnu/libuuid.so.1.3.0
7f750cb18000-7f750cb19000 rw-p 00004000 08:01 2331 /lib/x86_64-linux-gnu/libuuid.so.1.3.0
7f750cb19000-7f750cb34000 r-xp 00000000 08:01 533071 /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so
7f750cb34000-7f750cd33000 ---p 0001b000 08:01 533071 /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so
7f750cd33000-7f750cd34000 r--p 0001a000 08:01 533071 /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so
7f750cd34000-7f750cd38000 rw-p 0001b000 08:01 533071 /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so
7f750cd38000-7f750d039000 rw-p 00000000 00:00 0
7f750d039000-7f750d04e000 r-xp 00000000 08:01 533070 /usr/local/lib/python2.7/dist-packages/pandas/json.so
7f750d04e000-7f750d24e000 ---p 00015000 08:01 533070 /usr/local/lib/python2.7/dist-packages/pandas/json.so
7f750d24e000-7f750d24f000 r--p 00015000 08:01 533070 /usr/local/lib/python2.7/dist-packages/pandas/json.so
7f750d24f000-7f750d250000 rw-p 00016000 08:01 533070 /usr/local/lib/python2.7/dist-packages/pandas/json.so
7f750d250000-7f750d2a9000 r-xp 00000000 08:01 533270 /usr/local/lib/python2.7/dist-packages/pandas/parser.so
7f750d2a9000-7f750d4a8000 ---p 00059000 08:01 533270 /usr/local/lib/python2.7/dist-packages/pandas/parser.so
7f750d4a8000-7f750d4a9000 r--p 00058000 08:01 533270 /usr/local/lib/python2.7/dist-packages/pandas/parser.so
7f750d4a9000-7f750d4af000 rw-p 00059000 08:01 533270 /usr/local/lib/python2.7/dist-packages/pandas/parser.so
7f750d4af000-7f750d591000 r-xp 00000000 08:01 49584 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7f750d591000-7f750d790000 ---p 000e2000 08:01 49584 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7f750d790000-7f750d798000 r--p 000e1000 08:01 49584 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7f750d798000-7f750d79a000 rw-p 000e9000 08:01 49584 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7f750d79a000-7f750d7af000 rw-p 00000000 00:00 0
7f750d7af000-7f750d7f1000 r-xp 00000000 08:01 530477 /usr/lib/pyshared/python2.7/matplotlib/_path.so
7f750d7f1000-7f750d9f1000 ---p 00042000 08:01 530477 /usr/lib/pyshared/python2.7/matplotlib/_path.so
7f750d9f1000-7f750d9f3000 r--p 00042000 08:01 530477 /usr/lib/pyshared/python2.7/matplotlib/_path.so
7f750d9f3000-7f750d9f4000 rw-p 00044000 08:01 530477 /usr/lib/pyshared/python2.7/matplotlib/_path.so
7f750d9f4000-7f750da2d000 r-xp 00000000 08:01 533269 /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so
7f750da2d000-7f750dc2c000 ---p 00039000 08:01 533269 /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so
7f750dc2c000-7f750dc2d000 r--p 00038000 08:01 533269 /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so
7f750dc2d000-7f750dc31000 rw-p 00039000 08:01 533269 /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so
7f750dc31000-7f750dc7d000 r-xp 00000000 08:01 533447 /usr/local/lib/python2.7/dist-packages/pandas/_period.so
7f750dc7d000-7f750de7c000 ---p 0004c000 08:01 533447 /usr/local/lib/python2.7/dist-packages/pandas/_period.so
7f750de7c000-7f750de7d000 r--p 0004b000 08:01 533447 /usr/local/lib/python2.7/dist-packages/pandas/_period.so
7f750de7d000-7f750de86000 rw-p 0004c000 08:01 533447 /usr/local/lib/python2.7/dist-packages/pandas/_period.so
7f750de86000-7f750de87000 rw-p 00000000 00:00 0
7f750de87000-7f750debc000 r-xp 00000000 08:01 533203 /usr/local/lib/python2.7/dist-packages/pandas/index.so
7f750debc000-7f750e0bb000 ---p 00035000 08:01 533203 /usr/local/lib/python2.7/dist-packages/pandas/index.so
7f750e0bb000-7f750e0bc000 r--p 00034000 08:01 533203 /usr/local/lib/python2.7/dist-packages/pandas/index.so
7f750e0bc000-7f750e0c0000 rw-p 00035000 08:01 533203 /usr/local/lib/python2.7/dist-packages/pandas/index.so
7f750e0c0000-7f750e295000 r-xp 00000000 08:01 533278 /usr/local/lib/python2.7/dist-packages/pandas/algos.so
7f750e295000-7f750e494000 ---p 001d5000 08:01 533278 /usr/local/lib/python2.7/dist-packages/pandas/algos.so
7f750e494000-7f750e495000 r--p 001d4000 08:01 533278 /usr/local/lib/python2.7/dist-packages/pandas/algos.so
7f750e495000-7f750e4a9000 rw-p 001d5000 08:01 533278 /usr/local/lib/python2.7/dist-packages/pandas/algos.so
7f750e4a9000-7f750e4ac000 rw-p 00000000 00:00 0
7f750e4ac000-7f750e4b2000 r-xp 00000000 08:01 48831 /usr/lib/python2.7/lib-dynload/_csv.so
7f750e4b2000-7f750e6b1000 ---p 00006000 08:01 48831 /usr/lib/python2.7/lib-dynload/_csv.so
7f750e6b1000-7f750e6b2000 r--p 00005000 08:01 48831 /usr/lib/python2.7/lib-dynload/_csv.so
7f750e6b2000-7f750e6b4000 rw-p 00006000 08:01 48831 /usr/lib/python2.7/lib-dynload/_csv.so
7f750e6b4000-7f750e782000 r-xp 00000000 08:01 533449 /usr/local/lib/python2.7/dist-packages/pandas/lib.so
7f750e782000-7f750e981000 ---p 000ce000 08:01 533449 /usr/local/lib/python2.7/dist-packages/pandas/lib.so
7f750e981000-7f750e982000 r--p 000cd000 08:01 533449 /usr/local/lib/python2.7/dist-packages/pandas/lib.so
7f750e982000-7f750e990000 rw-p 000ce000 08:01 533449 /usr/local/lib/python2.7/dist-packages/pandas/lib.so
7f750e990000-7f750e992000 rw-p 00000000 00:00 0
7f750e992000-7f750ea8f000 r-xp 00000000 08:01 533271 /usr/local/lib/python2.7/dist-packages/pandas/tslib.so
7f750ea8f000-7f750ec8e000 ---p 000fd000 08:01 533271 /usr/local/lib/python2.7/dist-packages/pandas/tslib.so
7f750ec8e000-7f750ec8f000 r--p 000fc000 08:01 533271 /usr/local/lib/python2.7/dist-packages/pandas/tslib.so
7f750ec8f000-7f750eca1000 rw-p 000fd000 08:01 533271 /usr/local/lib/python2.7/dist-packages/pandas/tslib.so
7f750eca1000-7f750eca4000 rw-p 00000000 00:00 0
7f750eca4000-7f750ecc5000 r-xp 00000000 08:01 48837 /usr/lib/python2.7/lib-dynload/_ctypes.so
7f750ecc5000-7f750eec4000 ---p 00021000 08:01 48837 /usr/lib/python2.7/lib-dynload/_ctypes.so
7f750eec4000-7f750eec5000 r--p 00020000 08:01 48837 /usr/lib/python2.7/lib-dynload/_ctypes.so
7f750eec5000-7f750eec9000 rw-p 00021000 08:01 48837 /usr/lib/python2.7/lib-dynload/_ctypes.so
7f750eec9000-7f750eeca000 rw-p 00000000 00:00 0
7f750eeca000-7f750ef24000 r-xp 00000000 08:01 532046 /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so
7f750ef24000-7f750f123000 ---p 0005a000 08:01 532046 /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so
7f750f123000-7f750f124000 r--p 00059000 08:01 532046 /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so
7f750f124000-7f750f15c000 rw-p 0005a000 08:01 532046 /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so
7f750f15c000-7f750f15d000 rw-p 00000000 00:00 0
7f750f15d000-7f750f166000 r-xp 00000000 08:01 532085 /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so
7f750f166000-7f750f365000 ---p 00009000 08:01 532085 /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so
7f750f365000-7f750f366000 r--p 00008000 08:01 532085 /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so
7f750f366000-7f750f367000 rw-p 00009000 08:01 532085 /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so
7f750f367000-7f750f368000 r-xp 00000000 08:01 48818 /usr/lib/python2.7/lib-dynload/future_builtins.so
7f750f368000-7f750f567000 ---p 00001000 08:01 48818 /usr/lib/python2.7/lib-dynload/future_builtins.so
7f750f567000-7f750f568000 r--p 00000000 08:01 48818 /usr/lib/python2.7/lib-dynload/future_builtins.so
7f750f568000-7f750f569000 rw-p 00001000 08:01 48818 /usr/lib/python2.7/lib-dynload/future_builtins.so
7f750f569000-7f750f588000 r-xp 00000000 08:01 48815 /usr/lib/python2.7/lib-dynload/_io.so
7f750f588000-7f750f787000 ---p 0001f000 08:01 48815 /usr/lib/python2.7/lib-dynload/_io.so
7f750f787000-7f750f788000 r--p 0001e000 08:01 48815 /usr/lib/python2.7/lib-dynload/_io.so
7f750f788000-7f750f791000 rw-p 0001f000 08:01 48815 /usr/lib/python2.7/lib-dynload/_io.so
7f750f791000-7f750f907000 r-xp 00000000 08:01 532132 /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so
7f750f907000-7f750fb06000 ---p 00176000 08:01 532132 /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so
7f750fb06000-7f750fb07000 r--p 00175000 08:01 532132 /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so
7f750fb07000-7f750fb08000 rw-p 00176000 08:01 532132 /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so
7f750fb08000-7f750fba4000 rw-p 00000000 00:00 0
7f750fba4000-7f750fd01000 r-xp 00000000 08:01 532128 /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so
7f750fd01000-7f750ff00000 ---p 0015d000 08:01 532128 /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so
7f750ff00000-7f750ff01000 r--p 0015c000 08:01 532128 /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so
7f750ff01000-7f750ff02000 rw-p 0015d000 08:01 532128 /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so
7f750ff02000-7f750ff9d000 rw-p 00000000 00:00 0
7f750ff9d000-7f750ffa3000 r-xp 00000000 08:01 532067 /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so
7f750ffa3000-7f75101a2000 ---p 00006000 08:01 532067 /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so
7f75101a2000-7f75101a3000 r--p 00005000 08:01 532067 /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so
7f75101a3000-7f75101a4000 rw-p 00006000 08:01 532067 /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so
7f7510265000-7f751028f000 r-xp 00000000 08:01 532108 /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so
7f751028f000-7f751048e000 ---p 0002a000 08:01 532108 /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so
7f751048e000-7f751048f000 r--p 00029000 08:01 532108 /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so
7f751048f000-7f7510491000 rw-p 0002a000 08:01 532108 /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so
7f7510491000-7f75104d2000 rw-p 00000000 00:00 0
7f75104d2000-7f75104d5000 r-xp 00000000 08:01 48833 /usr/lib/python2.7/lib-dynload/_heapq.so
7f75104d5000-7f75106d4000 ---p 00003000 08:01 48833 /usr/lib/python2.7/lib-dynload/_heapq.so
7f75106d4000-7f75106d5000 r--p 00002000 08:01 48833 /usr/lib/python2.7/lib-dynload/_heapq.so
7f75106d5000-7f75106d7000 rw-p 00003000 08:01 48833 /usr/lib/python2.7/lib-dynload/_heapq.so
7f75106d7000-7f751073e000 r-xp 00000000 08:01 532118 /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so
7f751073e000-7f751093d000 ---p 00067000 08:01 532118 /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so
7f751093d000-7f751093e000 r--p 00066000 08:01 532118 /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so
7f751093e000-7f7510942000 rw-p 00067000 08:01 532118 /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so
7f7510942000-7f7510944000 rw-p 00000000 00:00 0
7f7510944000-7f7510958000 r-xp 00000000 08:01 48804 /usr/lib/python2.7/lib-dynload/datetime.so
7f7510958000-7f7510b57000 ---p 00014000 08:01 48804 /usr/lib/python2.7/lib-dynload/datetime.so
7f7510b57000-7f7510b58000 r--p 00013000 08:01 48804 /usr/lib/python2.7/lib-dynload/datetime.so
7f7510b58000-7f7510b5c000 rw-p 00014000 08:01 48804 /usr/lib/python2.7/lib-dynload/datetime.so
7f7510b5c000-7f7510caf000 r-xp 00000000 08:01 532106 /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so
7f7510caf000-7f7510eae000 ---p 00153000 08:01 532106 /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so
7f7510eae000-7f7510eb0000 r--p 00152000 08:01 532106 /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so
7f7510eb0000-7f7510ebd000 rw-p 00154000 08:01 532106 /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so
7f7510ebd000-7f7510ecf000 rw-p 00000000 00:00 0
7f7510ecf000-7f7510f08000 r-xp 00000000 08:01 533450 /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so
7f7510f08000-7f7511107000 ---p 00039000 08:01 533450 /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so
7f7511107000-7f7511108000 r--p 00038000 08:01 533450 /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so
7f7511108000-7f751110c000 rw-p 00039000 08:01 533450 /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so
7f751110c000-7f751110d000 rw-p 00000000 00:00 0
7f751110d000-7f7511296000 r--p 00000000 08:01 58562 /usr/lib/locale/locale-archive
7f7511296000-7f75112ab000 r-xp 00000000 08:01 2312 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f75112ab000-7f75114aa000 ---p 00015000 08:01 2312 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f75114aa000-7f75114ab000 r--p 00014000 08:01 2312 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f75114ab000-7f75114ac000 rw-p 00015000 08:01 2312 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f75114ac000-7f7511660000 r-xp 00000000 08:01 2327 /lib/x86_64-linux-gnu/libc-2.15.so
7f7511660000-7f751185f000 ---p 001b4000 08:01 2327 /lib/x86_64-linux-gnu/libc-2.15.so
7f751185f000-7f7511863000 r--p 001b3000 08:01 2327 /lib/x86_64-linux-gnu/libc-2.15.so
7f7511863000-7f7511865000 rw-p 001b7000 08:01 2327 /lib/x86_64-linux-gnu/libc-2.15.so
7f7511865000-7f751186a000 rw-p 00000000 00:00 0
7f751186a000-7f7511965000 r-xp 00000000 08:01 2400 /lib/x86_64-linux-gnu/libm-2.15.so
7f7511965000-7f7511b64000 ---p 000fb000 08:01 2400 /lib/x86_64-linux-gnu/libm-2.15.so
7f7511b64000-7f7511b65000 r--p 000fa000 08:01 2400 /lib/x86_64-linux-gnu/libm-2.15.so
7f7511b65000-7f7511b66000 rw-p 000fb000 08:01 2400 /lib/x86_64-linux-gnu/libm-2.15.so
7f7511b66000-7f7511b7c000 r-xp 00000000 08:01 2288 /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f7511b7c000-7f7511d7b000 ---p 00016000 08:01 2288 /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f7511d7b000-7f7511d7c000 r--p 00015000 08:01 2288 /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f7511d7c000-7f7511d7d000 rw-p 00016000 08:01 2288 /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f7511d7d000-7f7511f2f000 r-xp 00000000 08:01 2279 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f7511f2f000-7f751212e000 ---p 001b2000 08:01 2279 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f751212e000-7f7512149000 r--p 001b1000 08:01 2279 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f7512149000-7f7512154000 rw-p 001cc000 08:01 2279 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f7512154000-7f7512158000 rw-p 00000000 00:00 0
7f7512158000-7f75121ac000 r-xp 00000000 08:01 2393 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f75121ac000-7f75123ac000 ---p 00054000 08:01 2393 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f75123ac000-7f75123af000 r--p 00054000 08:01 2393 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f75123af000-7f75123b6000 rw-p 00057000 08:01 2393 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f75123b6000-7f75123b8000 r-xp 00000000 08:01 2283 /lib/x86_64-linux-gnu/libutil-2.15.so
7f75123b8000-7f75125b7000 ---p 00002000 08:01 2283 /lib/x86_64-linux-gnu/libutil-2.15.so
7f75125b7000-7f75125b8000 r--p 00001000 08:01 2283 /lib/x86_64-linux-gnu/libutil-2.15.so
7f75125b8000-7f75125b9000 rw-p 00002000 08:01 2283 /lib/x86_64-linux-gnu/libutil-2.15.so
7f75125b9000-7f75125bb000 r-xp 00000000 08:01 2406
/lib/x86_64-linux-gnu/ld-2.15.so
7f7512a2d000-7f7512b31000 rw-p 00000000 00:00 0
7f7512b62000-7f7512bea000 rw-p 00000000 00:00 0
7f7512bf7000-7f7512bf9000 rw-p 00000000 00:00 0
7f7512bf9000-7f7512bfa000 rwxp 00000000 00:00 0
7f7512bfa000-7f7512bfc000 rw-p 00000000 00:00 0
7f7512bfc000-7f7512bfd000 r--p 00022000 08:01 2260 /lib/x86_64-linux-gnu/ld-2.15.so
7f7512bfd000-7f7512bff000 rw-p 00023000 08:01 2260 /lib/x86_64-linux-gnu/ld-2.15.so
7ffcf454c000-7ffcf4585000 rw-p 00000000 00:00 0 [stack]
7ffcf459b000-7ffcf459d000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Aborted (core dumped)
with below code, there is no memory problem, but what can the below code do , I mean doing group by and data aggregation
with open("data/petaJoined.csv", "r") as content:
for line in content:
#print line
pass
#do stuff with line`
content.close()
Anyone knows what is happening?
Actually I want to reach the result shown in Pandas read csv out of memory
Maybe there will be a solution?
Note I already use read csv by chunk, but still there is memory error
Then, I changed the chunk size to have my bigData.py file in another way
import pandas as pd
import numpy as np
import sys, traceback, os
import etl2 # my self processing flow
reload(etl2)
def iter_chunks(n,df):
while True:
try:
yield df.get_chunk(n)
except StopIteration:
break
cksize=5
try:
dfs = pd.DataFrame()
reader=pd.read_table( 'data/petaJoined.csv',
chunksize = cksize,
low_memory = False,
iterator = True
) # choose as appropriate
for chunk in iter_chunks(cksize,reader):
temp=etl2.tb_createTopRankTable(chunk)
dfs.append(temp)
df=tb_createTopRankTable(dfs)
#
# for chunk in reader:
# pass
# temp=tb_createTopRankTable(chunk)
# dfs.append(temp)
# df=tb_createTopRankTable(dfs)
except:
traceback.print_exc(file=sys.stdout)
Still, there will be segmentation error after running for sometime
def tb_createTopRankTable(df):
try:
key='name1'
key2='name2'
df2 = df.groupby([key,key2])['isError'].agg({ 'errorNum': 'sum','totalParcel': 'count' })
df2['errorRate'] = df2['errorNum'] / df2['totalParcel']
return df2
Based on your snippet, when reading line-by-line.
I assume that kb_2 is the error indicator,
groups={}
with open("data/petaJoined.csv", "r") as large_file:
for line in large_file:
arr=line.split('t')
#assuming this structure: ka,kb_1,kb_2,timeofEvent,timeInterval
k=arr[0]+','+arr[1]
if not (k in groups.keys())
groups[k]={'record_count':0, 'error_sum': 0}
groups[k]['record_count']=groups[k]['record_count']+1
groups[k]['error_sum']=groups[k]['error_sum']+float(arr[2])
for k,v in groups.items:
print ('{group}: {error_rate}'.format(group=k,error_rate=v['error_sum']/v['record_count']))
This code snippet stores all the groups in a dictionary, and calculates the error rate after reading the entire file.
It will encounter an out-of-memory exception, if there are too many combinations of groups.
Q: Anyone knows what is happening?
A: Yes. Sum of all data memory-overheads for in-RAM objects !< RAM
It is a natural part of any formal abstraction to add some additional overhead in case some additional features are to be implemented on a higher ( a more abstract ) layer. That means that the more abstract / the more feature-rich representation of any dataset was chosen, the more memory- & processing-overheads are to be expected.
ITEMasINT = 32345
ITEMasTUPLE = ( 32345, )
ITEMasLIST = [ 32345, ]
ITEMasARRAY = np.array( [ 32345, ] )
ITEMasDICT = { 0: 32345, }
######## .__sizeof__() -> intnsize of object in memory, in bytes'
ITEMasINT.__sizeof__() -> 12 #_____ 100% _ trivial INT
ITEMasTUPLE.__sizeof__() -> 16 # 133% _ en-tuple-d
ITEMasLIST.__sizeof__() -> 24 # 200% _ list-ed
ITEMasARRAY.__sizeof__() -> 40 # 333% _ numpy-wrapped
ITEMasDICT.__sizeof__() -> 124 # 1033% _ hash-associated asDict
If a personal experience is not enough, check the "costs" of re-wrapping the input ( already not small ) data into pandas overheads:
CParserError: Error tokenizing data. C error: out of memory
Segmentation fault (core dumped)
and
CParserError: Error tokenizing data. C error: out of memory
*** glibc detected *** python: free(): ...
...
..
.
Aborted (core dumped)
Q: Maybe there will be a solution?
A: Yes.
Simply follow the computational strategy and deploy memory-efficient & fast processing of the csv-input ( it's still a fileIO having some 8-15 ms access time and quite a low performance stream data-flow, even if you use SSD-devices with about 960MB/s peak-transfer rate, your blocking-fact is the memory-allocation limit ... so rather be patient on input-stream and do not crash into a principal memory-barrier for any in-RAM super-object ( which would have been introduced just to be finally asked ( if it did not crash during it's instantiation ... ) to compute a plain sum/nROWs ).
A line-by-line or block-arranged reads allow you to calculate results on-the-fly and using a register-based ( asDict and alike for an interim storage of results ) sliding-window computation strategy is both fast and memory-efficient . ( Uri has provided an example for such )
This principal approach is used to be used in both real-time constrained systems and for system-on-chip designs, that were used for processing large data-streams for more than the last half century, so nothing new uder the Sun.
In case your results 's size cannot fit in RAM, than it makes no sense to even start the processing of any input file, does it?
Processing BigData is neither about super-up-scaling of the COTS-dataObjects nor about finding a best or a most sexy "one-liner" ...
BigData requires a lot of understanding of the way how to process both fast and smart so as to avoid extreme costs of even small overheads, that are forgiving to do principal mistakes on just a few GB-s of small-bigData but will kill anyone's budget & efforts once trying the same on a larger playground.
链接地址: http://www.djcxy.com/p/26344.html