Remove the clouds of suspicion caused by the problem "Maximum malloc memory available on Linux"

Today, when I read related books, I saw the question "Maximum number of heap requests in the process". We know that allocating memory using malloc is allocated in heap Heap. If a machine has 8GB of physical memory and 5GB of free memory, can we use malloc() to request 5GB of memory? This is true in theory, because these memory is not used by other processes. But the actual test results can be confusing.

The test environment for this article is as follows:

1 qi@qi:~$ uname -a
2 Linux qi 5.4.0-89-generic #100~18.04.1-Ubuntu SMP Wed Sep 29 10:59:42 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux 


  First, several issues to consider

  1. Did we request physical memory using malloc()?
  2. Can malloc() only request 8GB of physical memory?
  3. Can all memory sizes requested by malloc() be used for memset()?

The above three issues are just what we are going to discuss. Now assuming all three statements are correct, we can test the memory size malloc() can request with the following program:

 1 #include <stdio.h>
 2 #include <stdlib.h>
 3 #include <string.h>
 5 unsigned long long int maximum = 0;
 7 int main (int argc, char* argv[])
 8 {
 9     unsigned int block_size[] = {1024*1024, 1024, 1};
10     int i, count;
12     for( int i=0; i<3; i++) {
13         for(count=1;; count++) {
14             void *block = malloc(maximum + block_size[i]*count);
15             if( block ) {
16                 //memset(block, 0, maximum + block_size[i]*count);
17                 free(block);
18                 maximum = maximum + block_size[i]*count;
19             } else {
20                 break;
21             }
22         }
23     }
25     printf("maximum malloc size is %llu bytes \n", maximum);
27     return 0;
28 }

  Run the above program and get the output as:

1 root@qi:/home/qi/test_park/elf_load# ./main 
2 maximum malloc size is 24587279333 bytes

  You can see that the above test program has requested a maximum of 22.9GB of memory, but how much is the actual memory on my machine? The following:

1 qi@qi:~/test_park/elf_load$ free
2               total        used        free      shared  buff/cache   available
3 Mem:        8011016     2373760     3517884      719508     2119372     4654640
4 Swap:      15999996           0    15999996

Obviously, the maximum physical memory on the machine is less than 8GB. If you know Swap swap space, it may be said that the sum of the sum of the Mem and Wap items seems to be exactly 22.9GB, but another problem is that not all of these memory or Swap space are idle, including the system kernel and the system interface, etc. will also occupy a part of the physical memory. So we see that the available memory for the Mem item is only about 4.5 GB, so the result is that malloc () is requesting far more memory than we actually have physical memory. Since the actual output of the malloc() function does not match our expectations, is that where we used it incorrectly? Use "man malloc" to see its official explanation:

2        By default, Linux follows an optimistic memory allocation strategy.  This means that when malloc() returns non-NULL there is no guarantee that the mem‐
3        ory really is available.  In case it turns out that the system is out of memory, one or more processes will be killed by  the  OOM  killer.   For  more
4        information,  see  the  description  of /proc/sys/vm/overcommit_memory and /proc/sys/vm/oom_adj in proc(5), and the Linux kernel source file Documenta‐
5        tion/vm/overcommit-accounting.

  Otherwise, Note explains that even if malloc() returns a non-NULL pointer, there is no guarantee that all the memory areas it points to will be used by the process. So why is that so? As you'll see later, one of the most important settings to start with is the file'/proc/sys/vm/overcommit_memory', which is described using'man proc':

 1        /proc/sys/vm/overcommit_memory
 2               This file contains the kernel virtual memory accounting mode.  Values are:
 4                      0: heuristic overcommit (this is the default)
 5                      1: always overcommit, never check
 6                      2: always check, never overcommit
 8               In mode 0, calls of mmap(2) with MAP_NORESERVE are not  checked,  and  the  default
 9               check is very weak, leading to the risk of getting a process "OOM-killed".
11               In mode 1, the kernel pretends there is always enough memory, until memory actually
12               runs out.  One use case for this mode is  scientific  computing  applications  that
13               employ  large  sparse  arrays.   In Linux kernel versions before 2.6.0, any nonzero
14               value implies mode 1.
16               In mode 2 (available since Linux 2.6), the total virtual address space that can  be
17               allocated (CommitLimit in /proc/meminfo) is calculated as
19                   CommitLimit = (total_RAM - total_huge_TLB) *
20                                 overcommit_ratio / 100 + total_swap

You can see that if the file content is 0, MMAP (internal calls to malloc) will not be checked and there is a risk that no memory will be used. If the file content is 1, malloc() can request very large amounts of memory. Tested on my machine, it can reach 90T. If the file content is 2, then all memory that can be requested is CommitLimit. You can see the size specifically through a formula or "cat/proc/meminfo | grep Limit". That explains why malloc () has more than 22GB of memory. If not, check'/proc/sys/vm/overcommit_memory', which is 0:

1 root@qi:/home/qi/test_park/elf_load# cat /proc/sys/vm/overcommit_memory 
2 0

Part of the second question answered above is that malloc() can apply for more than the physical memory of the machine under certain settings. Why is that part of it because the memory that can be applied for is not only related to the above settings, but also to the swap space of the machine if you don't know or have heard about it (In fact, when you install a Linux system on your machine, you should encounter a swap setting when partitioning a disk) Swap space, just know that it is mounted on a physical hard disk to store less frequently used memory, is a low-speed expansion of physical memory, when physical memory is not used up, previously inaccessible content in physical memory will be moved here to make room for other processes. swap space can also be requested by malloc().

As a result, Question 2 has been answered in its entirety. At this point you might say that Question 1 should also have an answer, because malloc() not only requested 8GB of physical memory, but also 15GB of swap hard disk space for expanded memory, or even about 90TB of non-existent memory, so did the first question solve?

True, but not all, because malloc() applied for memory at this time, but did not apply completely. This involves an east-east called Lazy Allocation, similar to the write-time replication mechanism of fork. When you use malloc(), the system does not really allocate from physical memory, but waits until the process has to operate to provide allocation. This explains why we have just started requesting 22.9GB of memory without segment errors. Only when you access this memory area will it be allocated, so we can boldly add memset to the program, uncomment the line of memset of the code posted above, and run it again. If you don't want to wait too long, you can do it like me:

1 root@qi:/home/qi/test_park/elf_load# echo 0 >  /proc/sys/vm/overcommit_memory 
2 root@qi:/home/qi/test_park/elf_load# swapoff -a
3 root@qi:/home/qi/test_park/elf_load# echo 2 >  /proc/sys/vm/overcommit_memory

The above command disables the swap space, which reduces the amount of memory available. When the swap space is closed, if /proc/sys/vm/overcommit_memory content is 0, then you can malloc() memory size should be around 8GB, but not every byte can be memset, it can be tested, you will find that memset 6~7GB of memory space after the program error exits abnormally, because the amount of memory available at this time is also so large, in this case, it is not safe to use malloc() arbitrarily to request memory. If/proc/sys/vm/overcommit_memory content is 2, then the memory that can be applied for at this time should look like "CommitLimit". Testing on my machine is that only about 1.5GB can be applied, in which case the illegal memory area will not be accessed anyway, but one drawback is that you cannot use all the free memory, you can only modify the corresponding settings.

So how do you know the actual available memory size? One solution is to look at available memory in'/proc/meminfo'and apply again by multiplying the safety factor.

All three of these problems have been solved, and we are one step closer to professional linuxer.


Tags: Linux

Posted on Tue, 23 Nov 2021 13:36:27 -0500 by Vincent III