All geek questions in one place

Linux OOM Killer and Java Process

I often run into a problem in our Tomcat process production environment that is being killed by Linux OOM.

Checking / var / log / messages, he says that java is not corrupted, and java called the OOM killer.

-Xms20480m -Xmx20480m on a 32GB box.

I see below crash -

Is OOM the cause of this failure? or accident occurred due to OOM? How can I debug this problem?

# # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007f4c3230aad7, pid=16248, tid=139964439296320 # # JRE version: Java(TM) SE Runtime Environment (7.0_45-b18) (build 1.7.0_45-b18) # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.45-b08 mixed mode linux-amd64 compressed oops) # Problematic frame: # V [libjvm.so+0x674ad7] JVM_Clone+0x97 # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # If you would like to submit a bug report, please visit: # http://bugreport.sun.com/bugreport/crash.jsp # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. # --------------- THREAD --------------- Current thread (0x0000000001313800): JavaThread "http-bio-14080-exec-17" daemon [_thread_in_native, id=18943, stack(0x00007f4c029f7000,0x00007f4c02af8000)] siginfo:si_signo=SIGSEGV: si_errno=0, si_code=128 (), si_addr=0x0000000000000000 Registers: RAX=0x0000000000010000, RBX=0x80000001000004db, RCX=0x0000000302bd6fb0, RDX=0x00007f4c32acda50 RSP=0x00007f4c02af2c88, RBP=0x00007f4c02af2d70, RSI=0x00007f4c02af2d80, RDI=0x00000000013139e8 R8 =0x0000000000000001, R9 =0x00000002f8191228, R10=0x00007f4c2e231658, R11=0x00007f4c2e231638 R12=0x0000000302bd6fb0, R13=0x00000007e0002420, R14=0x0000000000000000, R15=0x0000000001313800 RIP=0x00007f4c3230aad7, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000000 TRAPNO=0x000000000000000d Top of Stack: (sp=0x00007f4c02af2c88) 0x00007f4c02af2c88: 00007f4c3230aa63 00000000129196d1 0x00007f4c02af2c98: 00000002f817b620 00000000013139e8 0x00007f4c02af2ca8: 00007f4c02af2d18 00000002f8191080 0x00007f4c02af2cb8: 0000000000000009 0000000001313800 0x00007f4c02af2cc8: 0000000000000000 0000000001313800 0x00007f4c02af2cd8: 00000000138f5509 0000000000000000 0x00007f4c02af2ce8: 00007f4c2e382ae8 0000000000000000 0x00007f4c02af2cf8: 00007f4c02af2cf8 00000002f817b620 0x00007f4c02af2d08: 00000002f8190fd8 00000007e01ab278 0x00007f4c02af2d18: 00000002f8190ec0 00000007e0002420 0x00007f4c02af2d28: 0000000000000000 00000007e0002420 0x00007f4c02af2d38: 0000000000000000 0000000000000000 0x00007f4c02af2d48: 00000007e0002420 0000000000000000 0x00007f4c02af2d58: 00007f4c02af2de0 0000000000000000 0x00007f4c02af2d68: 0000000001313800 00007f4c02af2dc0 0x00007f4c02af2d78: 00007f4c2e2316c6 0000000302bd6fb0 0x00007f4c02af2d88: 0000000000000000 00007f4c02af2e08 0x00007f4c02af2d98: 00007f4c2e1bc8e1 00007f4c02af2db8 0x00007f4c02af2da8: 00007f4c2e1bc8e1 00000002f8190fd8 0x00007f4c02af2db8: 00000002f817b620 00007f4c02af2e28 0x00007f4c02af2dc8: 00007f4c2e1bc233 00000007e1ec4e9f 0x00007f4c02af2dd8: 00007f4c2e1bc233 0000000302bd6fb0 0x00007f4c02af2de8: 00007f4c02af2de8 00000007e1b7a5db 0x00007f4c02af2df8: 00007f4c02af2e30 00000007e37575a8 0x00007f4c02af2e08: 0000000000000000 00000007e1b7a5e8 0x00007f4c02af2e18: 00007f4c02af2de0 00007f4c02af2e40 0x00007f4c02af2e28: 00007f4c02af2ed0 00007f4c2edaaf34 0x00007f4c02af2e38: 00007f4c2edaaf34 00007f4c0000000a 0x00007f4c02af2e48: 00000007e416ec20 0000000000000000 0x00007f4c02af2e58: 00000007e416e2b8 00007f4c02af2ed0 0x00007f4c02af2e68: 00007f4c2e1bc233 00007f4c02af2ed0 0x00007f4c02af2e78: 00007f4c2e1bc233 000000000000000a Instructions: (pc=0x00007f4c3230aad7) 0x00007f4c3230aab7: 85 60 02 00 00 06 00 00 00 4c 89 6d b0 4c 8b 23 0x00007f4c3230aac7: 4d 85 e4 0f 84 68 02 00 00 49 8b 9d 20 01 00 00 0x00007f4c3230aad7: 48 83 7b 10 f7 0f 87 6e 02 00 00 48 8b 43 10 48 0x00007f4c3230aae7: 8d 50 08 48 3b 53 18 0f 87 9c 02 00 00 48 89 53 Register to memory mapping: RAX=0x0000000000010000 is an unknown value RBX=0x80000001000004db is an unknown value RCX=0x0000000302bd6fb0 is an oop [Lcom.mycompany.MyClass$Type; - klass: 'com/mycompany/MyClass$Type'[] - length: 16 RDX=0x00007f4c32acda50: <offset 0xe37a50> in /usr/java/jre64-1.7.0_45/jre/lib/amd64/server/libjvm.so at 0x00007f4c31c96000 RSP=0x00007f4c02af2c88 is pointing into the stack for thread: 0x0000000001313800 RBP=0x00007f4c02af2d70 is pointing into the stack for thread: 0x0000000001313800 RSI=0x00007f4c02af2d80 is pointing into the stack for thread: 0x0000000001313800 RDI=0x00000000013139e8 is an unknown value R8 =0x0000000000000001 is an unknown value R9 = [error occurred during error reporting (printing register info), id 0xb] Stack: [0x00007f4c029f7000,0x00007f4c02af8000], sp=0x00007f4c02af2c88, free space=1007k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x674ad7] JVM_Clone+0x97 J java.lang.Object.clone()Ljava/lang/Object; j com.mycompany.MyClass$Type.values()[Lcom/mycompany/MyClass$Type;+3 Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) J java.lang.Object.clone()Ljava/lang/Object; j com.mycompany.MyClass$Type.values()[Lcom/mycompany/MyClass$Type;+3

/ var / log / messages -

 myhostname kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 myhostname kernel: java cpuset=/ mems_allowed=0 myhostname kernel: Pid: 32307, comm: java Not tainted 2.6.39-400.209.1.el5uek #1 myhostname kernel: Call Trace: myhostname kernel: [<ffffffff811136b4>] dump_header+0x94/0xe0 myhostname kernel: [<ffffffff811137fd>] oom_kill_process+0x6d/0x160 myhostname kernel: [<ffffffff811139ec>] out_of_memory+0xfc/0x210 myhostname kernel: [<ffffffff811187ec>] __alloc_pages_slowpath+0x64c/0x660 myhostname kernel: [<ffffffff811189b4>] __alloc_pages_nodemask+0x1b4/0x200 myhostname kernel: [<ffffffff8111b140>] ? __do_page_cache_readahead+0xe0/0x170 myhostname kernel: [<ffffffff81150893>] alloc_pages_current+0xb3/0x120 myhostname kernel: [<ffffffff811100da>] __page_cache_alloc+0x9a/0xb0 myhostname kernel: [<ffffffff8111097f>] page_cache_read+0x4f/0xb0 myhostname kernel: [<ffffffff81111a54>] filemap_fault+0x174/0x270 myhostname kernel: [<ffffffff81137a2c>] __do_fault+0x5c/0x550 myhostname kernel: [<ffffffff81137fc6>] do_linear_fault+0x36/0x40 myhostname kernel: [<ffffffff81510b6e>] ? call_function_interrupt+0xe/0x20 myhostname kernel: [<ffffffff81138044>] handle_pte_fault+0x74/0x190 myhostname kernel: [<ffffffff815106ae>] ? apic_timer_interrupt+0xe/0x20 myhostname kernel: [<ffffffff8113828f>] handle_mm_fault+0x12f/0x1b0 myhostname kernel: [<ffffffff8150b1cd>] do_page_fault+0x17d/0x4b0 myhostname kernel: [<ffffffff8117b821>] ? user_path_at+0x11/0x20 myhostname kernel: [<ffffffff81170516>] ? vfs_fstatat+0x56/0x90 myhostname kernel: [<ffffffff8117067b>] ? vfs_stat+0x1b/0x20 myhostname kernel: [<ffffffff81507cd5>] page_fault+0x25/0x30 myhostname kernel: Mem-Info: myhostname kernel: Node 0 DMA per-cpu: myhostname kernel: CPU 0: hi: 0, btch: 1 usd: 0 myhostname kernel: CPU 1: hi: 0, btch: 1 usd: 0 myhostname kernel: CPU 2: hi: 0, btch: 1 usd: 0 myhostname kernel: CPU 3: hi: 0, btch: 1 usd: 0 myhostname kernel: CPU 4: hi: 0, btch: 1 usd: 0 myhostname kernel: CPU 5: hi: 0, btch: 1 usd: 0 myhostname kernel: Node 0 DMA32 per-cpu: myhostname kernel: CPU 0: hi: 186, btch: 31 usd: 0 myhostname kernel: CPU 1: hi: 186, btch: 31 usd: 0 myhostname kernel: CPU 2: hi: 186, btch: 31 usd: 0 myhostname kernel: CPU 3: hi: 186, btch: 31 usd: 11 myhostname kernel: CPU 4: hi: 186, btch: 31 usd: 30 myhostname kernel: CPU 5: hi: 186, btch: 31 usd: 0 myhostname kernel: Node 0 Normal per-cpu: myhostname kernel: CPU 0: hi: 186, btch: 31 usd: 0 myhostname kernel: CPU 1: hi: 186, btch: 31 usd: 0 myhostname kernel: CPU 2: hi: 186, btch: 31 usd: 0 myhostname kernel: CPU 3: hi: 186, btch: 31 usd: 0 myhostname kernel: CPU 4: hi: 186, btch: 31 usd: 0 myhostname kernel: CPU 5: hi: 186, btch: 31 usd: 0 myhostname kernel: active_anon:4372468 inactive_anon:275213 isolated_anon:0 myhostname kernel: active_file:17 inactive_file:21 isolated_file:0 myhostname kernel: unevictable:6002 dirty:24 writeback:4 unstable:0 myhostname kernel: free:38369 slab_reclaimable:3708 slab_unreclaimable:9265 myhostname kernel: mapped:1253 shmem:67 pagetables:10880 bounce:0 myhostname kernel: Node 0 DMA free:15880kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15688kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes myhostname kernel: lowmem_reserve[]: 0 3000 32290 32290 myhostname kernel: Node 0 DMA32 free:119288kB min:2136kB low:2668kB high:3204kB active_anon:30224kB inactive_anon:9152kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072096kB mlocked:0kB dirty:16kB writeback:4kB mapped:36kB shmem:0kB slab_reclaimable:40kB slab_unreclaimable:232kB kernel_stack:24kB pagetables:48kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:67 all_unreclaimable? yes myhostname kernel: lowmem_reserve[]: 0 0 29290 29290 myhostname kernel: Node 0 Normal free:17564kB min:20856kB low:26068kB high:31284kB active_anon:17459648kB inactive_anon:1091700kB active_file:120kB inactive_file:88kB unevictable:24008kB isolated(anon):0kB isolated(file):0kB present:29992960kB mlocked:24008kB dirty:80kB writeback:12kB mapped:4976kB shmem:268kB slab_reclaimable:14792kB slab_unreclaimable:36828kB kernel_stack:2768kB pagetables:43472kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:14972 all_unreclaimable? yes myhostname kernel: lowmem_reserve[]: 0 0 0 0 myhostname kernel: Node 0 DMA: 0*4kB 1*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15880kB myhostname kernel: Node 0 DMA32: 458*4kB 494*8kB 284*16kB 55*32kB 7*64kB 4*128kB 5*256kB 7*512kB 7*1024kB 6*2048kB 20*4096kB = 119288kB myhostname kernel: Node 0 Normal: 3333*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 17428kB myhostname kernel: 115783 total pagecache pages myhostname kernel: 114466 pages in swap cache myhostname kernel: Swap cache stats: add 2183060, delete 2068594, find 1860109/1985849 myhostname kernel: Free swap = 0kB myhostname kernel: Total swap = 2097148kB myhostname kernel: 8388592 pages RAM myhostname kernel: 134806 pages reserved myhostname kernel: 6845 pages shared myhostname kernel: 8210059 pages non-shared
+6
source share
2 answers

It is more likely that the OOM problem is crashing. One common reason for using more memory than the -Xmx argument is to use internal memory. This may be due to the fact that you are using the JNI library that allocates many objects, or if you are memory mapping files, etc.

You should try adding some log statements to your Java code to print memory. Java believes that it uses Runtime.totalMemory, etc. Then compare them with the values ​​you see above and see if there is anything else in the memory.

0
source

This is the linux restriction thread number.

Linux by default is limited to 1024 and maximum 65535.

you can use the command ulimit -c unlimited unlimited.

0
source

Source: https://habr.com/ru/post/1204819/

More articles:


All Articles