Linux kernel module OOM when there is memory in cache
Embedded system, no swap, kernel v2.6.36, memory compaction enabled.
Under heavy usage, all the RAM is tied up in cache. Cache was using about 70M of memory. When a user space process allocates memory, no problem, cache gives it up.
But there's a 3rd party device driver that seems to try to allocate a physical 5th order page, and fails with OOM. A quick look at buddyinfo confirms this... no 5th order page available. But as soon as I drop cache, plenty becomes available and the device driver no longer OOM.
So it seems to me that virtual memory allocation will trigger cache drop, but physical memory allocation will not? This doesn't make sense, because then kernel modules are likely to OOM when memory is tied up in cache, and this behavior seems to be more detrimental than slow disk access from no caching.
Is there a tuning parameter to address this?
Thanks!
So here's what's going on. I still don't know why high cache use is causing kernel modules to OOM. The problem is in 3rd party code that we don't have access to, so who knows what they're doing.
I guess one can argue if this is by design, where non-critical disk cache could take all available free memory, and cause kernel modules to OOM, then IMHO, maybe disk cache should leave something for the kernel.
I've decided to instead, limit the cache, so there is always some "truly free" memory left for kernel use, and not depend on "sort of free" memory tied up in cache.
There is a kernel patch I found that will add /proc/sys/vm/pagecache_ratio so you can set how much memory the disk cache could take. But that was never incorporated into the kernel for whatever reason (I thought that was a good idea, especially if disk cache could cause kernel OOM). But I didn't want to mess with kernel patches for maintainability and future-proofing reasons. If someone is just doing a one-shot deal, and doesn't mind patches, here's the link: http://lwn.net/Articles/218890/
My solution is that I've recompiled the kernel and enabled cgroups, and I'm using that to limit the memory usage for a group of processes that are responsible for lots of disk access (hence running up the cache). After tweaking the configuration, it seems to be working fine. I'll leave my setup running the stress test over the weekend and see if OOM still happens.
Thanks Jesus.
Edit: I guess I found my own answer. There are VM tuning parameters in /proc/sys/vm/. Tune-able settings relevant to this issue are: min_free_kbytes, lowmem_reserve_ratio, and extfrag_threshold
链接地址: http://www.djcxy.com/p/70322.html