memory management issues

["Bertani, John A" <john dot bertani at hp dot com>]

We are having a problem with memory allocations and deallocations that are smaller than 32 Mb. It appears that these small allocations are leaking even though Valgrind shows no leaks and even though we have done extensive desk-checking and logging to verify that we are performing the allocations and deallocations correctly.

Some background:

The application (written in C++) processes and stores large data files in memory where 'large' is defined as anwhere from 1 Mb up to 250 Mb. We call these large files 'frames' and the application is called framebroker.  The framebroker has a dedicated server running 64 bit RHEL 5.2 with 32 Gb of memory. The framebroker receives these files from another system over a socket connection, stores the files in a std::map object, and then delivers them, upon request, to another system, also via a socket connection. Once the receiving system is done processing the file, the file is deleted from the std::map.

Essentially, these large files flow through the system. Since the receiving application may process the files slower than the sending application, the framebroker may fill up its memory. We manage this by making the sender wait until space has been freed up in the framebroker.

To give us a rough estimate of how much memory we still have in the framebroker, we use the freeram field of the sysinfo structure. We limit the framebroker's memory usage to 30 Gb using setrlimit64. We also ensure that at least 1 Gb is available for other framebroker activities before we accept a new file from the sender by using C++ 'new' to try a 1 GB allocation. If it throws an exception because it doesn't have 1 Gb to allocate, we ignore the sender. (We are also assuming that attempting a 1 Gb allocation will force memory management to clean up deallocations to honor the request if it can.)

What we see:

The first time the application is run, we see memory usage climb from 0% up to 80% and then drop back down to 0%. The second time we try this, with the same test case, the memory only recedes to the 33% mark - roughly 11 Gb has been lost. At this point the freeram value returned from sysinfo says that we only have ~22 Gb of free space. That space is never recovered until we restart the framebroker.

Experimentation has shown us that when the file received from the sender is 32 Mb of greater, the memory is always returned. We believe this is because our MMAP_THRESHOLD value is set to 32 Mb.

We developed a simple test application to try and determine the problem. This test app first does 20 allocations of a given size (say 10,000,000 bytes), storing them in an array of char *. Then it deallocates each one. It will perform the allocate-all/deallocate-all loop numerous times. The first time through, all memory is returned regardless of the size of the allocation. However, on the 2nd through nth loop, the memory is not returned to the application until the last deallocation is performed. In other words, if there were 20 allocations of 10,000,000 bytes each, on the 1st through 19th iteration, no memory is returned. However, on the 20th deallocation the 200,000,000 bytes are returned. 

If we break out of the deallocation loop right before the last iteration, none of the memory is returned. In other words, for the 20-iteration loop, we will do 19 'delete [] buf[i]' commands but not the 20th, and roughly 200,000,000 bytes will be lost to the application. 

If the storage structure for the char * pointers is a map, then the memory is not returned, even after the last pointer's space has been deleted.

If the allocation size is greater than 32 Mb, all deallocations happen immediately and in every loop.


I've included the test application and the makefile. Any insight or suggestions will be appreciated.


Server configuration:

RHEL 5.2, 64 bit
32 Gb of RAM
glibc v. 2.5-24
libgcc v. 4.1.2-42.el5

Attachment: Makefile
Description: Makefile

Attachment: main.cpp
Description: main.cpp