How to properly measure process memory usage on Linux?
I want to measure memory usage by processes on Linux, notably Ubuntu 15.04, but I'm not sure how to do this correctly. I want the measurements to correlate with the free command, so that the total amount of memory found to be in use corresponds to what free reports (sans buffers/cache).
So far, I've written this Python script, but it isn't in agreement with free , as it reports a lower total memory usage:
#!/usr/bin/env python
from collections import OrderedDict
import os.path
import re
def parse_mem_file(filename):
data = OrderedDict()
with open(filename, 'rb') as f:
for line in f:
splittage = line.split(':')
data[splittage[0]] = splittage[1].strip()
return data
def get_process_mem_usage():
re_pid = re.compile(r'^d+$')
re_mem = re.compile(r'^(d+) .+$')
pid2usage = {}
for pid in [d for d in os.listdir('/proc') if re_pid.match(d)]:
fpath = os.path.join('/proc', pid, 'status')
try:
data = parse_mem_file(fpath)
except IOError:
continue
try:
pid2usage[pid] = int(re_mem.match(data['VmHWM']).group(1)) / 1024.
except KeyError:
continue
return OrderedDict(
sorted(pid2usage.iteritems(), key=lambda x: x[1], reverse=True))
pid2usage = get_process_mem_usage()
total_usage = sum(pid2usage.values())
print('Total memory usage: {:.2f}'.format(total_usage))
for pid, usage in pid2usage.iteritems():
print('{}: {:.2f} MB'.format(pid, usage))
How can I adjust this program to report memory usage that is in accordance with what the free program reports?
For context, the problem is that the memory usage on my server is increasing over time until a lot of swap memory is in use and it slows down a great deal as a result. I should like to know the cause of this.
free on Linux reads /proc/meminfo file (see proc(5) ). You can ensure it by invoking strace free , there is not very long output.
PS Updated due to your comment.
free shows the information about physical memory. It calculates amount of used physical memory as MemTotal - MemFree . So we are interested in physical memory used by processes in the system.
First of all, you are using VmHWM field of /proc/<pid>/status file. From proc(5) :
VmHWM: Peak resident set size ("high water mark").
That is, the maximum amount of physical memory that the given process has used during its lifetime. It's not the current memory usage of the process. If you want to get the current physical memory usage, look at VmRSS :
VmRSS: Resident set size.
The second thing. The file /proc/<pid>/status designed to be human-readable. For a programmer it's easier to parse /proc/<pid>/statm and get the value in the second column, which is the same as VmRSS in /proc/<pid>/status .
Now the most important part. As we consider physical memory, we should remember that it is a system-wide thing. Amount of physical memory cannot be calculated as sum of resident set size of all processes in the system for a number of reasons. Firstly, it's not only user-space processes who use physical memory in the system, but also kernel -- all the device drivers, kernel threads and who knows what else. Secondly, processes can share memory between each other.
All in all, proc(5) is a very good read.
PPS Regarding:
For context, the problem is that the memory usage on my server is increasing over time until a lot of swap memory is in use and it slows down a great deal as a result. I should like to know the cause of this.
If you want to diagnose which processes consume all the memory, you can invoke top and sort processes by share of physical memory used (type "M").
上一篇: 测量代码单元的内存使用情况