My web application (delpoyed on weblogic) went down when I check jstack info, I found that most of the threads are BLOCKED on org.apache.log4j.spi.RootLogger. The thread that owns this lock is also LOCKED and gets stuck for at least 20 minutes until I kill the process, but it does not wait on any other lock, how does this happen?
here is the jstack info:
BLOCKED lock lock <783b8910> but not expecting other locks:
"[STUCK] ExecuteThread: '58' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=7 tid=02e45400 nid=86 lwp_id=7959740 waiting for monitor entry [61cff000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.log4j.Category.callAppenders(Category.java:201)
- locked <783b8910> (a org.apache.log4j.spi.RootLogger)
at org.apache.log4j.Category.forcedLog(Category.java:388)
at org.apache.log4j.Category.error(Category.java:302)
Other threads waiting to block <783b8910>:
"[STUCK] ExecuteThread: '36' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=7 tid=02e0ea00 nid=64 lwp_id=7959717 waiting for monitor entry [62d7f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.log4j.Category.callAppenders(Category.java:201)
- waiting to lock <783b8910> (a org.apache.log4j.spi.RootLogger)
at org.apache.log4j.Category.forcedLog(Category.java:388)
at org.apache.log4j.Category.error(Category.java:319)
and
"[STUCK] ExecuteThread: '34' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=7 tid=02e0a200 nid=62 lwp_id=7959715 waiting for monitor entry [62eff000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.log4j.Category.callAppenders(Category.java:201)
- waiting to lock <783b8910> (a org.apache.log4j.spi.RootLogger)
at org.apache.log4j.Category.forcedLog(Category.java:388)
at org.apache.log4j.Category.info(Category.java:663)
etc., a total of 64 threads are waiting for blocking <783b8910>
it happens about 1-3 times a month, I can’t find a solution because I don’t know why the thread that owns the lock is BLOCKED until it waits for other locks.
log4j 1.2.13