C3p0 Deadlock detection - thread is running too often

Question

We use the Hibernate and c3p0 pooling library. Until now, this combination worked perfectly until recently, when we decided to increase maxPoolSize to 1000 and conduct a lot of stress testing of our application. The peak load of our application leads to the fact that the database responds very slowly, and, as a result, the c3p0 deadlock detector spits out the APPARENT DEADLOCK warning again and again.

Based on the c3p0 documentation, we changed maxAdministrativeTaskTime to 10 minutes, assuming Deadlock is detected once every 30 minutes (the code indicates that the deadlock detection rate is three times higher than maxAdministrativeTaskTime).

However, when analyzing c3p0 logs, the deadlock detection thread runs more than 30 minutes. The relevant part of the magazines is attached. Surprisingly, the frequency is uneven.

Line 573745: [Timer-2] 2013-06-26 04:47:52,492 WARN [null] com .mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ 662cee3b -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks! Line 573746: [Timer-2] 2013-06-26 04:47:52,512 WARN [null] com .mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ 662cee3b -- APPARENT DEADLOCK!!! Complete Status: Line 574292: [Timer-2] 2013-06-26 04:49:12,493 WARN [null] com .mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ 662cee3b -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks! Line 574293: [Timer-2] 2013-06-26 04:49:12,513 WARN [null] com .mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ 662cee3b -- APPARENT DEADLOCK!!! Complete Status: Line 575004: [Timer-2] 2013-06-26 04:50:32,494 WARN [null] com .mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ 662cee3b -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks! Line 575005: [Timer-2] 2013-06-26 04:50:32,511 WARN [null] com .mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ 662cee3b -- APPARENT DEADLOCK!!! Complete Status: Line 576062: [Timer-2] 2013-06-26 04:51:52,495 WARN [null] com .mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ 662cee3b -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks! Line 576063: [Timer-2] 2013-06-26 04:51:52,536 WARN [null] com .mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ 662cee3b -- APPARENT DEADLOCK!!! Complete Status: Line 576720: [Timer-2] 2013-06-26 04:53:12,496 WARN [null] com .mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ 662cee3b -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks! Line 576721: [Timer-2] 2013-06-26 04:53:12,516 WARN [null] com .mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ 662cee3b -- APPARENT DEADLOCK!!! Complete Status: Line 594087: [Timer-2] 2013-06-26 04:55:52,550 WARN [null] com .mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ 662cee3b -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks! Line 594088: [Timer-2] 2013-06-26 04:55:52,569 WARN [null] com .mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ 662cee3b -- APPARENT DEADLOCK!!! Complete Status: Line 594753: [Timer-2] 2013-06-26 04:57:12,550 WARN [null] com .mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ 662cee3b -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks! Line 594754: [Timer-2] 2013-06-26 04:57:12,572 WARN [null] com .mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ 662cee3b -- APPARENT DEADLOCK!!! Complete Status: Line 595624: [Timer-2] 2013-06-26 04:58:32,552 WARN [null] com .mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ 662cee3b -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks! Line 595625: [Timer-2] 2013-06-26 04:58:32,570 WARN [null] com .mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ 662cee3b -- APPARENT DEADLOCK!!! Complete Status: Line 596416: [Timer-2] 2013-06-26 04:59:52,552 WARN [null] com .mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ 662cee3b -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks! Line 596417: [Timer-2] 2013-06-26 04:59:52,572 WARN [null] com .mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ 662cee3b -- APPARENT DEADLOCK!!! Complete Status: Line 611011: [Timer-2] 2013-06-26 05:02:22,556 WARN [null] com .mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ 662cee3b -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks! Line 611012: [Timer-2] 2013-06-26 05:02:22,577 WARN [null] com .mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ 662cee3b -- APPARENT DEADLOCK!!! Complete Status: 

Can someone explain this anomaly?

Answer 1

maxAdministrativeTaskTime does not affect the thread pool lock detector; it simply limits the length that one task can perform before the pool tries to execute its interrupt() . one (hackish !, ugly !, not recommended!) to try to avoid APPARENT DEADLOCKs would be to set this in a short amount of time so that slow tasks are interrupted and not hung so long that the pool decides that it is stuck. See discussion here .

thanks to the questionnaire (via email), I double-checked my own% $ ^ * &! code, and the lock detection interval depends on maxAdministrativeTaskTime . as the question suggests, it should be 3 times; maxAdministrativeTaskTime . therefore, the mystery deepens.

it’s worth understanding what the deadlock detector does. c3p0 supports thread pool, numHelperThreads large. deadlock detector periodically notes all tasks. It runs threads in the pool, then sleeps for a while, and then checks again. if none of the tasks is completed, that is, the active tasks remain the same as during the last check, then a deadlock is declared. the interval at which c3p0 waits for the completion of a task is 10 seconds, and so far it is not configured. It can be configured.

you can post some examples of APPARENT DEADLOCK that you see. c3p0 prints a lot of information to help you understand what might be hanging.

Here are two suggestions for reducing APPARENT DEADLOCK under very heavy load:

• Are the deadlock blocks you observe related to Connection acqusistion? If so, and if you have not already done so, upgrade to c3p0-0.9.2.1 or the latest preliminary preview version 0.9.5.

• You tried to increase numHelperThreads , how did you increase your load? if some tasks run very slowly, c3p0 will not deadlock if all threads are blocked by slow tasks. if there are enough Threads so that the nimbler tasks are still running, you will not see deadlocks.

Hope this helps!

Answer 2

I can’t explain this, but 1000 connections are pretty many, does your c3p0 configuration reflect this large number of connections? My experience is that the default settings are good for fewer connections. I suggest reading the c3po documentation , especially the part in the “Other Data Source Configuration” section.