Determine the cause of hundreds of AJP threads in Tomcat

Question

We have two Tomcat 6.0.20 servers leaving Apache, and the communication between them is done using AJP. Tomcat, in turn, consumes web services in the JBoss cluster.

This morning, one of the Tomcat machines used 100% of the processor on 6 of the 8 cores on our machine. We grabbed a bunch of heaps using JConsole and then tried plugging in JVisualVM to get a profile to see what the whole processor was accepting, but that caused Tomcat to crash. At least we had a bunch of heaps!

I uploaded a bunch of heaps in the Eclipse MAT, where I found that we have 565 instances of java.lang.Thread. Some of them are obviously completely legitimate, but the vast majority are called "ajp-6009-XXX", where XXX is the number.

I am very good at the Eclipse MAT, but could not find an explanation. If anyone has any hints as to why Tomcat can do this, or any hints as to why using the Eclipse MAT would be appreciated!

Answer 1

This is not a direct answer, I think, but perhaps as a mitigating approach to production, you can limit the damage by limiting maxThreads for AJP in your configuration, for http://tomcat.apache.org/tomcat-6.0-doc/config /ajp.html ?

- 200, , , , 565 . , , , , , -. , ? - Apache , , ?

Answer 2

, , , 8 100% ( Tomcat).

java.util.HashMap get(), for:

   public V get(Object key) {
       if (key == null)
           return getForNullKey();
       int hash = hash(key.hashCode());
       for (Entry<K,V> e = table[indexFor(hash, table.length)];
            e != null;
            e = e.next) {
           Object k;
           if (e.hash == hash && ((k = e.key) == key || key.equals(k)))
               return e.value;
       }
       return null;
   }

, - , . , , .

, , / ; , , . - , , , ( ).

, HashMap ( HashSet) . , , ConcurrentHashMap ReentrantReadWriteLock . , Collections.synchronizedMap(), .

, .

. :

http://lightbody.net/blog/2005/07/hashmapget_can_cause_an_infini.html http://mailinator.blogspot.com/2009/06/beautiful-race-condition.html