how memory allocation works at extremes?

 

I am quite confused how memory allocation (malloc/calloc) works in linux/c. Suppose, I have a machine with 16GB RAM and I run a program as root. It is 64-bit machine, so all 16GB can be addressed as a single segment. Can I allocate all of that (less the amount for OS of course) with a single malloc call? with many malloc calls?

How it all relates to "heap memory" and "virtual memory"? If i allocate a small memory chunk, and it happens be in heap memory, then I resize (enlarge) this chunk, what happens when I approach stack area?

Do I have to fiddle with setrlimit RLIMIT_AS if I want (almost) all RAM to be assigned to my single process, even thought it's running as root?

A 64-bit process can allocate all of the memory. It doesn't even need to be root, unless the system has defined a ulimit setting for non-root users. Try ulimit -v to see if a limit is set.

Under Linux default settings a process can ask for nearly any amount of memory and it will be granted. The memory will be actually assigned as it is used, and it will come from physical RAM or from disk swap as needed.

A memory allocation resize is normally done in the C library by allocating the new, larger size and copying the old data into the new allocation. It is not usually done by expanding the existing allocation. Memory allocations are chosen to not conflict with other allocations such as program stack.

On a virtual memory OS (such as Linux) malloc() does not allocate memory. It allocates address space For instance, compile and run the following fragment, and (in another terminal) run top : 

#include <stdlib.h>
#include <unistd.h>

int main(void) {
  char *cp;
  cp = malloc( 16ULL * 1024 *1024 *1024);
  if (cp) pause();
  return 0;
}

At my computer, TOP displays: 

PID    USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
29026 plasser    20   0 16.0g  324  248 S    0  0.0   0:00.00 a.out

Which means: a.out has 16GB Virtual size, and uses only 324 (bytes? KB?) resident memory (probably just the binary)

What happenened?

  • the malloc() call has asked the OS to extend the address space , adding 16 GB
  • The OS has fulfilled this request, and set up pagetables for it et cetera.
  • these page tables have no physical memory attached to them (yet), maybe except for the tables themselves
  • once the program starts referencing this address space, pages will be attached (The OS wil fault them in)
  • (the pages will probably be COW'ed from /dev/zero, but that is only a detail)
  • but, at the moment the program does reference the addresses, memory will have to be be allocated by the OS and become attached to the process. (and it will show up in the RES field, Trust me )
  • at some moment, the attached memory could be detached too, once the OS thinks it has not been used for a long time. It will be added to a pool of free memory or/and used for other purposes. Its contents may be pushed to backing store (disk) in this case

    • `

    malloc() can allocate memory either by extending the heap or by mmap() ing a sufficiently large memory block. Which of them it uses depends on the size you request and on several mallopt() options.

    If it uses the "real" heap section, which usually lies quite at the start of the virtual memory area, it can grow up to the point where it hits the next allocated memory block which lies quite away from the start.

    If it uses mmap() , it depends on if there is contiguous address space available.

    /proc/<PID>/maps is a good query point for finding out the address space situation of a process.

    I just tested with my 64 bit server with Python and ctypes: 

    import ctypes
import ctypes.util
m=ctypes.CDLL(ctypes.util.find_library("c")).malloc
m.argtypes=(ctypes.c_uint64,)
m.restype = ctypes.c_uint64

    Now I have m , which serves as a call to the libc's malloc() function.

    Now I can do 

    >>> hex(m(2700000000))
'0x7f1345e3b010L'
>>> hex(m(2700000000))
'0x7f12a4f4f010L'
>>> hex(m(2700000000))
'0x7f1204063010L'
>>> hex(m(2700000000))
'0x7f1163177010L'
>>> hex(m(2700000000))
'0x7f10c228b010L'
>>> hex(m(2700000000))
'0x7f102139f010L'
>>> hex(m(2700000000))
'0x7f0f804b3010L'

    but due to whatever reasons I can't do 

    >>> hex(m(2900000000))
'0x0L'

    as it returns 0 resp. a NULL pointer.

    Doing now 

    print open("/proc/self/maps").read()

    shows me the lines of said file; somewhere in 

    7f0ed8000000-7f0ed8021000 rw-p 00000000 00:00 0 
7f0ed8021000-7f0edc000000 ---p 00000000 00:00 0 
7f0edf5c7000-7f13e6d27000 rw-p 00000000 00:00 0

    I got my allocations.

    However, if I do instead 

    >>> hex(m(1))
'0x1f07320L'
>>> hex(m(1))
'0x1f0c0e0L'
>>> hex(m(1))
'0x1f02d50L'
>>> hex(m(1))
'0x1f02d70L'
>>> hex(m(1))
'0x1ef94f0L'
>>> hex(m(1))
'0x1eb7b20L'
>>> hex(m(1))
'0x1efb700L'
>>> hex(m(270))
'0x1f162a0L'
>>> hex(m(270))
'0x1f163c0L'
>>> hex(m(270))
'0x1f164e0L'
>>> hex(m(270))
'0x1f16600L'

    the memory addresses come from 

    >>> print open("/proc/self/maps").read()
[...]
01d2e000-01f2b000 rw-p 00000000 00:00 0                                  [heap]
[...]

    the designated heap.

    Note that while I can get this memory assigned, using it all causes harm because my process might get killed by the famous OOM killer.

    链接地址: http://www.djcxy.com/p/82716.html

    上一篇: 为什么memset函数使虚拟内存如此之大

    下一篇: 内存分配如何在极端情况下工作？