Out of memory at Tomcat shutdown

Question

A brief description of my problem: I am launching Tomcat using my deployed Wicket application. When I want to close tomcat, I get the following error message:

Error occurred during initialization of VM java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:640) at java.lang.ref.Reference.<clinit>(Reference.java:145) 

I run the following setup:

• Ubuntu Linux: 10.04 (lucid) with kernel 2.6.18-028stab094.3
• Java version: "1.6.0_26" HotSpot (TM) Java Server 64-bit server
• Tomcat Version: 7.0.23
• jvm_args: -Xms512m -Xmx512m -XX: MaxPermSize = 205m (they are added via CATALINA_OPTS, nothing more)
• Wicket 1.5.1
• Tomcat configured with two virtual hosts on sub-domains with ModProxy
• My application is deployed as ROOT.war in the application directory (no difference in deploying one or both applications)
• '' 'No application is deployed to disable OOM when disconnected' '' if I do not communicate with jvm args
• The size of the war is about 500 thousand, all libraries are deployed in tomcat / common / lib (the directory that I added to common.loader in conf / catalina.properties).
• ulimit -u → unlimited

When I test the Tomcat manager application, it says the following about JVM memory:

 Free memory: 470.70 MB Total memory: 490.68 MB Max memory: 490.68 MB (http connector) Max threads: 200 Current thread count: 6 Current thread busy: 1 

'top' or 'free -m' is similar:

 Mem: 2097152k total, 1326772k used, 770380k free, 0k buffers 20029 myuser 18 0 805m 240m 11m S 0 11.7 0:19.24 java 

I tried running jmap to get a heap dump, it also does not work with OutOfMemoryError. In fact, as long as one or both of my applications are deployed, any other java process will exit with the same OOM error (see above).

The problem occurs during application deployment. So there is something seriously wrong with this. However, the application is pretty fast. But I also saw OOM in the application, so I do not trust calm.

Does my application use its own filter class? Could it be?

For completeness (hopefully), here is a list of libraries in my shared / lib:

 activation-1.1.jar antlr-2.7.6.jar antlr-runtime-3.3.jar asm-3.1.jar asm-commons-3.1.jar asm-tree-3.1.jar c3p0-0.9.1.1.jar commons-collections-3.1.jar commons-email-1.2.jar dependencies-provided.tgz dom4j-1.6.1.jar ejb3-persistence-1.0.2.GA.jar geronimo-annotation_1.0_spec-1.1.1.jar geronimo-jaspic_1.0_spec-1.0.jar geronimo-jta_1.1_spec-1.1.1.jar hibernate-annotations-3.4.0.GA.jar hibernate-commons-annotations-3.1.0.GA.jar hibernate-core-3.3.0.SP1.jar hibernate-entitymanager-3.4.0.GA.jar hibernate-search-3.1.0.GA.jar javassist-3.4.GA.jar joda-time-1.6.2.jar jta-1.1.jar log4j-1.2.16.jar lombok-0.9.3.jar lucene-core-2.4.0.jar mail-1.4.1.jar mysql-connector-java-5.1.14.jar persistence-api-1.0.jar quartz-2.1.1.jar servlet-api-2.5.jar slf4j-api-1.6.1.jar slf4j-log4j12-1.6.1.jar stringtemplate-4.0.2.jar wicket-auth-roles-1.5.1.jar wicket-core-1.5.1.jar wicket-datetime-1.5.1.jar wicket-extensions-1.5.1.jar wicket-request-1.5.1.jar wicket-util-1.5.1.jar xml-apis-1.0.b2.jar 

I appreciate any hint or even suggestion that gives me additional ideas for what to try.

Update. I checked a few more and found that this behavior only occurs when deploying one or both of my applications. The behavior does not occur on the "empty" tomcat (it was a mistake on my part to mess with jvm args)

Update2: I am currently experimenting trying to reproduce this behavior in a virtual box, I want to debug this using the profiler. I'm still not convinced that starting my installation on 2 GB of RAM will be impossible.

Update3 (10/01/12): I'm trying to run jenkins instead of my own application. Same behavior, so this is definitely a server configuration problem. Jenkins fails when maven is called, so I don’t even have to try to crack the shutdown suggested below because I need a second Java process during Jenkins startup. I was suggested that since this is a virtual server, ulimits can be imposed from the outside, and I could not see them. I think I will ask a new question. thanks all.

Update4 (02/05/12): see below answer containing a hint. I’ll explain a few more examples: now I’m 95% sure that errors occur because I reach the limit of the stream. However, since this is a virtual server, the method described below will not work to check this value, because it is not displayed using ulimit, it confused me, and only today I found out that this is the value "numproc", which I can see in Parallels Power Panel, with which I can log in to my virtual server. There were resource alerts for numproc, but I have not seen them either before or now. The value has a hard limit of 96, which I cannot change, of course. The current numproc value corresponds to the number of processes that I see with "top" after switching "H" to see the threads. It was very difficult for me to find this, because this numproc value is hidden deep inside the panel. Sadly 96 is a pretty low number if you want to run tomcat with apache and mysql. I am also very sad that I can’t even find this value in the small print of my hosting contract, and this is very relevant for my application, which I dare say. Therefore, I assume that I will need a server update.

Thanks to everyone for your helpful answers, in the end, everyone helped me figure out a bit what the problem is.

Answer 1

You seem to have too many threads.

Use this command:

 ulimit -u 

What is the result? It should be something like:

 max user processes (-u) 100 

If this is correct, you can edit this file:

 /etc/security/limits.conf 

and the following changes:

 #<domain> <type> <item> <value> user soft nproc 10000 user hard nproc 10000 

Answer 2

The tomcat termination procedure is to send a command / word through the tcp port to the current tomcat VM copy. This port is configured in the server.xml file (if I remember correctly, dialing my phone right now). So far, so good.

Unfortunately, the shutdown script does this by running 2. VM using the same java options as for tomcat. There is not enough memory for this system.

As a permission, you can write your own stop script using telnet or something like that.

If necessary, I could help later.

Hope this helps. Viele grüsse bert

Answer 3

You will probably survive some time. All you have to do is kill the tomcat process when you need to restart it. This is not a good approach, but the main problem is that your application is working correctly.

It seems to me that in the end, you may need to order a hosting plan with more available RAM.

Answer 4

I had a similar problem with installing tomcat only last week. I managed to fix this by giving tomcat a smaller heap. Something like that:

 export CATALINA_OPTS="-Xms256m -Xmx512m" 

Before starting Tomcat can help. In the meantime, you have to kill it in the old fashioned way with kill -9 ;)

EDIT: you can also look here , it looks like tomcat automatically creates a bunch of "spare" threads, but you can limit them as well as the maximum number of threads in the configuration. Hope this helps.