Malloc uses 10x memory required

Question

I have a network application that highlights 65k predictive fragments as part of the IO subsystem. Memory usage is tracked atomically inside the system, so I know how much memory I use. This number can also be checked with malloc_stats ()

Result malloc_stats()

Arena 0:
system bytes     =    1617920
in use bytes     =    1007840
Arena 1:
system bytes     = 2391826432
in use bytes     =  247265696
Arena 2:
system bytes     = 2696175616
in use bytes     =  279997648
Arena 3:
system bytes     =    6180864
in use bytes     =    6113920
Arena 4:
system bytes     =   16199680
in use bytes     =     699552
Arena 5:
system bytes     =   22151168
in use bytes     =     899440
Arena 6:
system bytes     =    8765440
in use bytes     =     910736
Arena 7:
system bytes     =   16445440
in use bytes     =   11785872
Total (incl. mmap):
system bytes     =  935473152
in use bytes     =  619758592
max mmap regions =         32
max mmap bytes   =   72957952

Items for notes:

• total in use bytesis a completely correct number according to my internal counter. However, the application has a 5.2 GB (top / htop) RES. The distribution is almost always 65 thousand; I don't understand the sheer amount of fragmentation / waste that I see even more when mmap comes into play.
• total system bytesdoes not match the amount system bytesin each Arena.
• I'm on Ubuntu 16.04 using glibc 2.23-0ubuntu3
• 1 2 RES, .
• 1 2 10 .
• 65k ( )

malloc ?

, malloc . ( ) . , .

- mallinfo - - . . .

Arena 2:
system bytes     = 2548473856
in use bytes     =    3088112
Arena 3:
system bytes     = 3288600576
in use bytes     =    6706544
Arena 4:
system bytes     =   16183296
in use bytes     =     914672
Arena 5:
system bytes     =   24027136
in use bytes     =     911760
Arena 6:
system bytes     =   15110144
in use bytes     =     643168
Arena 7:
system bytes     =   16621568
in use bytes     =   11968016
Total (incl. mmap):
system bytes     = 1688858624
in use bytes     =   98154448
max mmap regions =         32
max mmap bytes   =   73338880
arena (total amount of memory allocated other than mmap)                 = 1617780736
ordblks (number of ordinary non-fastbin free blocks)                     =       1854
smblks (number of fastbin free blocks)                                   =         21
hblks (number of blocks currently allocated using mmap)                  =         31
hblkhd (number of bytes in blocks currently allocated using mmap)        =   71077888
usmblks (highwater mark for allocated space)                             =          0
fsmblks (total number of bytes in fastbin free blocks)                   =       1280
uordblks (total number of bytes used by in-use allocations)              =   27076560
fordblks (total number of bytes in free blocks)                          = 1590704176
keepcost (total amount of releaseable free space at the top of the heap) =     439216

: total system bytes, malloc, , arena. (1,6 6,1 ). , (A) malloc , (B), malloc , .

UPDATE. Ubuntu , , . , , malloc .

Answer 1

, , . , .

---

malloc sbrk, mmap . , 4096 . malloc. .

, , . , , , RAM []. [--] "" R/O.

​​, , ​​ , , . . / .

, . .

---

RSS ( )

RSS , , .

128 , (, 4 ), RSS 4 . RSS /, .

, - , RSS , , . , B "" A, RSS A B.

" " - , ​​ , "". , .

3 , 10 , , /scattershot .

, RSS [ ], , . RSS , . - " ", - , "" .

---

malloc

, malloc . "" ( "" ). , malloc(32), () A, , malloc(1024 * 1024), (, B.).

"" 32 B, , malloc(1M)

, , "" 2.

, malloc , . , mmap . 4096, A 4096/32 128 .

---

munmap

free(ptr) [ptr ], . free , / .

, (.. , free . free [] munmap .

munmap [ ], , . , "" .

---

malloc_trim madvise

free, , , munmap. , RSS (, A) .

, , , /, .

A RSS. (, B) , A [ A RSS], B [ RSS ].

, B, , A RSS . , 100 , , 1 , RSS 100 .

, "" B ​​ A, " " RSS.

, , syscall madvise MADV_WONTNEED. , , [] , RSS.

, . , .

, , ​​ RAM . RSS . .

madvise malloc_trim RSS .

Answer 2

free .

, , . , free . malloc_trim.

malloc_trim(0) , RSS. , , . , RSS.

, , .

---

keepcost, . man mallinfo:

.         . . Malloc_stats (3) malloc_info (3) , .