All geek questions in one place

Docker Container - JVM Memory Spike - Chunk Memory Space Arena

I observe a large discrete surge in JVM memory during performance tests on my Java web application running in ECS / EC2 / Docker / Centos7 / Tomcat / OpenJDK8.

The performance test is quite simple and consists of constant simultaneous requests to the AWS application load balancer sitting in front of a pair of Docker containers running on EC2 hosts managed by the Elastic Container Service. Typically, the concurrency level is 30 concurrent connections / threads of client load tests. Within minutes, one of the Docker containers usually suffers.

A memory spike appears to be in memory without a heap. In particular, a burst of memory appears to be related to memory space Arena Chunk. Comparing the memory capacity of a JVM that has not experienced a surge with the one that has, memory spaces are allocated Threadand Arena Chunk.

The following is a comparison of the internal memory of a VM using the utility jcmd.

Note the absurdly high number for memory Arena Chunkand the relatively large number for memory Thread.

The concurrency test tier can create immediate demand for threads in the Tomcat request thread pool. However, a surge is not always found in the first wave of requests.

Have you seen something like this? Do you know what causes the surge?

Docker statistics

Memory storage:

Mon Oct  9 00:31:45 UTC 2017
89440337e936        27.36%              530 MiB / 2.93 GiB      17.67%              15.6 MB / 24.1 MB   122 MB / 2.13 MB    0
Mon Oct  9 00:31:48 UTC 2017
89440337e936        114.13%             2.059 GiB / 2.93 GiB    70.29%              16.3 MB / 25.1 MB   122 MB / 2.13 MB    0

Normal container:

Mon Oct  9 00:53:41 UTC 2017
725c23df2562        0.08%               533.4 MiB / 2.93 GiB   17.78%              5 MB / 8.15 MB      122 MB / 29.3 MB    0
Mon Oct  9 00:53:44 UTC 2017
725c23df2562        0.07%               533.4 MiB / 2.93 GiB   17.78%              5 MB / 8.15 MB      122 MB / 29.3 MB    0

VM internal memory

Spike JVM Memory:

# jcmd 393 VM.native_memory summary
393:

Native Memory Tracking:

Total: reserved=1974870KB, committed=713022KB
-                 Java Heap (reserved=524288KB, committed=524288KB)
                            (mmap: reserved=524288KB, committed=524288KB) 

-                     Class (reserved=1096982KB, committed=53466KB)
                            (classes #8938)
                            (malloc=1302KB #14768) 
                            (mmap: reserved=1095680KB, committed=52164KB) 

-                    Thread (reserved=8423906KB, committed=8423906KB)
                            (thread #35)
                            (stack: reserved=34952KB, committed=34952KB)
                            (malloc=114KB #175) 
                            (arena=8388840KB #68)

-                      Code (reserved=255923KB, committed=37591KB)
                            (malloc=6323KB #8486) 
                            (mmap: reserved=249600KB, committed=31268KB) 

-                        GC (reserved=6321KB, committed=6321KB)
                            (malloc=4601KB #311) 
                            (mmap: reserved=1720KB, committed=1720KB) 

-                  Compiler (reserved=223KB, committed=223KB)
                            (malloc=93KB #276) 
                            (arena=131KB #3)

-                  Internal (reserved=2178KB, committed=2178KB)
                            (malloc=2146KB #11517) 
                            (mmap: reserved=32KB, committed=32KB) 

-                    Symbol (reserved=13183KB, committed=13183KB)
                            (malloc=9244KB #85774) 
                            (arena=3940KB #1)

-    Native Memory Tracking (reserved=1908KB, committed=1908KB)
                            (malloc=8KB #95) 
                            (tracking overhead=1900KB)

-               Arena Chunk (reserved=18014398501093554KB, committed=18014398501093554KB)
                            (malloc=18014398501093554KB) 

-                   Unknown (reserved=38388KB, committed=38388KB)
                            (mmap: reserved=38388KB, committed=38388KB)

JVM:

# jcmd 391 VM.native_memory summary
391:

Native Memory Tracking:

Total: reserved=1974001KB, committed=710797KB
-                 Java Heap (reserved=524288KB, committed=524288KB)
                            (mmap: reserved=524288KB, committed=524288KB) 

-                     Class (reserved=1096918KB, committed=53738KB)
                            (classes #9005)
                            (malloc=1238KB #13654) 
                            (mmap: reserved=1095680KB, committed=52500KB) 

-                    Thread (reserved=35234KB, committed=35234KB)
                            (thread #35)
                            (stack: reserved=34952KB, committed=34952KB)
                            (malloc=114KB #175) 
                            (arena=168KB #68)

-                      Code (reserved=255261KB, committed=35237KB)
                            (malloc=5661KB #8190) 
                            (mmap: reserved=249600KB, committed=29576KB) 

-                        GC (reserved=6321KB, committed=6321KB)
                            (malloc=4601KB #319) 
                            (mmap: reserved=1720KB, committed=1720KB) 

-                  Compiler (reserved=226KB, committed=226KB)
                            (malloc=96KB #317) 
                            (arena=131KB #3)

-                  Internal (reserved=2136KB, committed=2136KB)
                            (malloc=2104KB #11715) 
                            (mmap: reserved=32KB, committed=32KB) 

-                    Symbol (reserved=13160KB, committed=13160KB)
                            (malloc=9221KB #85798) 
                            (arena=3940KB #1)

-    Native Memory Tracking (reserved=1890KB, committed=1890KB)
                            (malloc=8KB #95) 
                            (tracking overhead=1882KB)

-               Arena Chunk (reserved=178KB, committed=178KB)
                            (malloc=178KB) 

-                   Unknown (reserved=38388KB, committed=38388KB)
                            (mmap: reserved=38388KB, committed=38388KB)
+4
1

A glibc/malloc, , MALLOC_PER_THREAD=0. , debian/openjdk docker base image, centos, .

+1

Source: https://habr.com/ru/post/1687129/

More articles:


All Articles