The eclipse memory analyzer sees a small portion (363.2 MB) of the entire heap heap (8 GB)

Question

I tried to investigate java.lang.OutOfMemoryError: GC limit exceeded , which occurs when a large load of our web application deployed to tomcat. The heap size was set to 8 GB ( -Xms2048m -Xmx8192m )

At some point, our application stops responding due to GC overhead. In magazines, I saw that Full GC happens several times in a row. So I took a bunch of heaps with the following command ( jmap -F -dump:format=b,file=/root/dump2.hprof 4963 ). The file containing the dump was 9 GB in size. After the reset was made (the application was frozen for about 45 minutes), several full GCs were made until OutOfMemoryError was thrown.

Here is an example of a GC activity log

 [Full GC [PSYoungGen: 932096K->875513K(1864128K)] [ParOldGen: 5592447K->5592447K(5592448K)] 6524543K->6467961K(7456576K) [PSPermGen: 112285K->112285K(262144K)], 12.3954040 secs] [Times: user=47.60 sys=0.43, real=12.39 secs] [Full GC [PSYoungGen: 932096K->890562K(1864128K)] [ParOldGen: 5592447K->5592447K(5592448K)] 6524543K->6483009K(7456576K) [PSPermGen: 112285K->112285K(262144K)], 12.6131900 secs] [Times: user=48.45 sys=0.49, real=12.61 secs] [Full GC [PSYoungGen: 932096K->895268K(1864128K)] [ParOldGen: 5592447K->5592447K(5592448K)] 6524543K->6487715K(7456576K) [PSPermGen: 112285K->112285K(262144K)], 12.9488670 secs] [Times: user=49.61 sys=0.46, real=12.95 secs] Heap PSYoungGen total 1864128K, used 896698K [0x0000000755560000, 0x0000000800000000, 0x0000000800000000) eden space 932096K, 96% used [0x0000000755560000,0x000000078c10e8a8,0x000000078e3a0000) from space 932032K, 0% used [0x000000078e3a0000,0x000000078e3a0000,0x00000007c71d0000) to space 932032K, 0% used [0x00000007c71d0000,0x00000007c71d0000,0x0000000800000000) ParOldGen total 5592448K, used 5592447K [0x0000000600000000, 0x0000000755560000, 0x0000000755560000) object space 5592448K, 99% used [0x0000000600000000,0x000000075555ff30,0x0000000755560000) PSPermGen total 262144K, used 112285K [0x00000005e0000000, 0x00000005f0000000, 0x0000000600000000) object space 262144K, 42% used [0x00000005e0000000,0x00000005e6da7530,0x00000005f0000000) heap dump is taken (ca 45minutes freeze) [Full GC [PSYoungGen: 932096K->903362K(1864128K)] [ParOldGen: 5592447K->5592447K(5592448K)] 6524543K->6495810K(7456576K) [PSPermGen: 112285K->112285K(262144K)], 2883.9864390 secs] [Times: user=49.41 sys=0.47, real=2884.17 secs] [Full GC [PSYoungGen: 932096K->897728K(1864128K)] [ParOldGen: 5592447K->5592444K(5592448K)] 6524543K->6490173K(7456576K) [PSPermGen: 112288K->112288K(262144K)], 13.3092680 secs] [Times: user=50.75 sys=0.40, real=13.31 secs] 

To analyze a bunch of heaps, I opened it in an eclipse memory analyzer (MAT). Unfortunately, the MAT shows that the heap size was 363.2 MB (on the Browse tab or the Heap Details tab), whereas according to GC logs the heap was filled up to 6467961K (6.4G). Unreachable objects The histogram shows a total of 75 511 736 (75 MB). The histogram also confirmed that the total small heap was 380,837,136 (363.2 MB)

My question is, why does MAT not display all objects from the heap dump if GC cannot recover memory?

 env details: Eclipse Memory Analyzer Version 1.2.1 heap dump taken on java version "1.7.0_13" Java(TM) SE Runtime Environment (build 1.7.0_13-b20) Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode) 

Here are screenshots of the imported heap heap in MAT:

• overview
• unreachable

Answer 1

MAT does not display unreachable objects by default.

You can enable this option by choosing Settings → Memory Analyzer → Save Inaccessible Objects. Download the heap again as soon as the option is enabled.

It will display the full heap after enabling the option. Even I was in the same situation and could not get a lot of information on the Internet, and my manager showed me an option. Hope this helps.

Answer 2

This is the usual MAT behavior when dumping a heap of such a large heap. I often collect 8 GB heap heap heaps and usually get a MAT profile showing ~ 1GB of live objects.

A delay of 45 minutes is also expected. My explanation is that during the collection of the heap heap , several FullGC cycles occur, which reduce the size of the actual heap dump received . But I did not find an official explanation or reference documentation why there is such a big difference.

Further, see this link - MAT does not show a full bunch :

Symptom: When monitoring memory usage online, the heap size used is much larger than the MAT reports.

During index creation, the memory analyzer deletes inaccessible objects because various garbage collection algorithms tend to leave some garbage (if the object is too small, moving and reassigning addresses is expensive). This, however, should be no more than 3-4 percent. If you want to know which objects are deleted, enable debug output as described here: MemoryAnalyzer / FAQ # Enable_Debug_Output

Answer 3

In fact, I do not recommend using the jmap utility when you fix the java.lang.OutOfMemoryError problem : the GC limit has exceeded . I saw problems with this approach, while the JVM destroys (excess # core collections).

Instead, try the following approach and see if you get the best MAT analysis and analysis results:

• Add the following JVM parameter HotSpot -XX: + HeapDumpOnOutOfMemoryError
• Repeat the problem again and wait for the OOM event. Heap dump generation should be much faster, which will increase the value and reliability of the data.
• The JVM then generates the JVM Heap Dump (HPROF format) after the OOM event.
• Download the heap JVM heap again in MAT and see if you get the best results, for example. more heap.

Regards, RN

Answer 4

Recently, we found an error in MAT, where she saw only a subset of the heap: https://bugs.eclipse.org/bugs/show_bug.cgi?id=404679#c3

The problem was that the JVM wrote one HEAP_DUMP entry that was larger than 4 GB, and therefore the length of the header wrapped.

What version of MAT are you using? Try a newer build.