What does WinDbg mean from external fragmentation?

Question

I have a problem with bad_alloc. It throws during std::vector.push_back when it tries to move and allocate 2Mb

Heap state

  Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast (k) (k) (k) (k) length blocks cont. heap ----------------------------------------------------------------------------- 04e00000 00000002 67704 41088 67704 40735 135 135 0 25 LFH External fragmentation 99 % (135 free blocks) Virtual address fragmentation 39 % (135 uncommited ranges) ... 0c8f0000 00001002 82216 81384 82216 8247 172196 13 5 3 LFH ... 0: Heap 04e00000 Flags 00000002 - HEAP_GROWABLE Reserved memory in segments 67704 (k) Commited memory in segments 41088 (k) Virtual bytes (correction for large UCR) 58644 (k) Free space 40735 (k) (135 blocks) External fragmentation 99% (135 free blocks) Virtual address fragmentation 29% (135 uncommited ranges) Virtual blocks 0 - total 0 KBytes Lock contention 37 Segments 1 Low fragmentation heap 04e044f0 Lock contention 0 Metadata usage 0 bytes Statistics: Segments created 0 Segments deleted 0 Segments reused 0 Block cache: Buckets info: Size Blocks Seg Empty Aff Distribution ------------------------------------------------ ------------------------------------------------ Default heap Front heap Unused bytes Range (bytes) Busy Free Busy Free Total Average ------------------------------------------------------------------ 0 - 1024 217 0 0 0 487 2 1024 - 2048 1 0 0 0 1 1 3072 - 4096 0 1 0 0 0 0 15360 - 16384 1 0 0 0 1 1 203776 - 204800 0 2 0 0 0 0 220160 - 221184 0 2 0 0 0 0 224256 - 225280 0 1 0 0 0 0 236544 - 237568 0 2 0 0 0 0 240640 - 241664 0 1 0 0 0 0 257024 - 258048 0 1 0 0 0 0 265216 - 266240 0 4 0 0 0 0 273408 - 274432 0 1 0 0 0 0 277504 - 278528 0 2 0 0 0 0 281600 - 282624 0 2 0 0 0 0 285696 - 286720 0 6 0 0 0 0 289792 - 290816 0 4 0 0 0 0 293888 - 294912 0 12 0 0 0 0 297984 - 299008 0 10 0 0 0 0 302080 - 303104 0 13 0 0 0 0 306176 - 307200 0 14 0 0 0 0 310272 - 311296 0 19 0 0 0 0 314368 - 315392 0 15 0 0 0 0 318464 - 319488 0 3 0 0 0 0 326656 - 327680 0 1 0 0 0 0 330752 - 331776 0 3 0 0 0 0 333824 - 334848 1 0 0 0 8 8 351232 - 352256 0 1 0 0 0 0 363520 - 364544 0 2 0 0 0 0 367616 - 368640 0 1 0 0 0 0 396288 - 397312 0 3 0 0 0 0 416768 - 417792 0 1 0 0 0 0 420864 - 421888 0 2 0 0 0 0 424960 - 425984 0 1 0 0 0 0 433152 - 434176 0 1 0 0 0 0 437248 - 438272 0 1 0 0 0 0 466944 - 467968 0 1 0 0 0 0 470016 - 471040 0 1 0 0 0 0 482304 - 483328 0 1 0 0 0 0 ------------------------------------------------------------------ Total 220 135 0 0 497 2 0: Heap 0c8f0000 Flags 00001002 - HEAP_GROWABLE Reserved memory in segments 82216 (k) Commited memory in segments 81384 (k) Virtual bytes (correction for large UCR) 81620 (k) Free space 8247 (k) (172196 blocks) External fragmentation 10% (172196 free blocks) Virtual address fragmentation 0% (13 uncommited ranges) Virtual blocks 5 - total 0 KBytes Lock contention 3 Segments 1 Low fragmentation heap 43230048 Lock contention 0 Metadata usage 4096 bytes Statistics: Segments created 120 Segments deleted 2 Segments reused 0 Block cache: 5: 4096 bytes ( 54, 0) 6: 8192 bytes ( 26, 0) 7: 16384 bytes ( 13, 0) 8: 32768 bytes ( 8, 0) 9: 65536 bytes ( 9, 0) 10: 131072 bytes ( 5, 0) 11: 262144 bytes ( 4, 1) Buckets info: Size Blocks Seg Empty Aff Distribution ------------------------------------------------ 64 25483 53 0 0 (53-25483) 72 2991 30 0 0 (30-2991) 80 750 15 0 0 (15-750) 88 46 1 1 0 (1-46) 96 42 1 1 0 (1-42) 104 39 1 1 0 (1-39) 112 36 1 1 0 (1-36) 120 33 1 1 0 (1-33) 128 63 1 1 0 (1-63) 136 60 1 1 0 (1-60) 144 56 1 1 0 (1-56) 152 53 1 1 0 (1-53) 160 51 1 1 0 (1-51) 168 48 1 1 0 (1-48) 200 40 1 1 0 (1-40) 208 39 1 1 0 (1-39) 224 36 1 1 0 (1-36) 280 29 1 1 0 (1-29) 312 26 1 1 0 (1-26) 7432 17 1 1 0 (1-17) 9736 26 1 1 0 (1-26) 10760 24 1 1 0 (1-24) 11784 22 1 1 0 (1-22) ------------------------------------------------ Default heap Front heap Unused bytes Range (bytes) Busy Free Busy Free Total Average ------------------------------------------------------------------ 0 - 1024 619437 172194 28854 1067 11637848 17 1024 - 2048 10 1 0 0 113 11 3072 - 4096 2 0 0 0 32 16 4096 - 5120 1 0 0 0 16 16 5120 - 6144 1 0 0 0 20 20 7168 - 8192 1 0 0 17 20 20 9216 - 10240 0 0 0 26 0 0 10240 - 11264 0 0 0 24 0 0 11264 - 12288 0 0 0 22 0 0 12288 - 13312 2 0 0 0 32 16 15360 - 16384 1 0 0 0 1 1 16384 - 17408 2 0 0 0 32 16 20480 - 21504 1 0 0 0 8 8 21504 - 22528 1 0 0 0 20 20 40960 - 41984 1 0 0 0 8 8 43008 - 44032 1 0 0 0 16 16 48128 - 49152 1 0 0 0 16 16 54272 - 55296 1 0 0 0 8 8 58368 - 59392 1 0 0 0 17 17 72704 - 73728 1 0 0 0 20 20 83968 - 84992 0 1 0 0 0 0 96256 - 97280 1 0 0 0 16 16 108544 - 109568 1 0 0 0 16 16 116736 - 117760 1 0 0 0 21 21 139264 - 140288 1 0 0 0 16 16 145408 - 146432 1 0 0 0 16 16 159744 - 160768 1 0 0 0 22 22 163840 - 164864 1 0 0 0 20 20 193536 - 194560 1 0 0 0 20 20 262144 - 263168 3 0 0 0 40 13 327680 - 328704 1 0 0 0 16 16 333824 - 334848 1 0 0 0 8 8 436224 - 437248 1 0 0 0 16 16 ------------------------------------------------------------------ Total 619479 172196 28854 1156 11638454 17 

My question is what does 99% external fragmentation mean and why is “heap fragmentation” and not “virtual address space fragmentation”?

What is the difference between External Fragmentation (99%) and Virtual Address Fragmentation (29%)?

Can creating a new heap help?

PS: I have a lot of managed code in the same process. Private working set is 1.8 GB.

Answer 1

Basically, you have a large number of very small objects that cause fragmentation. In addition, you mix your own and managed code in the same process, and you make large memory allocations. These approaches require memory management. It would be better to create a dedicated heap for large memory allocations when starting the process to avoid such a problem. The heap should only be used for large memory allocations of its own code.

Also, it would be nice to analyze why managed code requires so much memory.