I have a webapp (Tomcat / Hibernate / DBCP 1.4) that runs MySQL queries, and this is great for a specific load, say 50 queries per second. When I route the same moderate load through HAProxy (still using only one database), I get a failure, possibly one for every 500 requests. My messages in the application:
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet successfully received from the server was 196,898 milliseconds ago. The last packet sent successfully to the server was 0 milliseconds ago. at sun.reflect.GeneratedConstructorAccessor210.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1117) at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3567) ... Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3017) ...
Meanwhile, the HAProxy log shows a lot of entries, such as:
27] mysql mysql/db03 0/0/34605 2364382 cD 3/3/3/3/0 0/0 Oct 15 15:43:12 localhost haproxy[3141]: 127.0.0.1:35500 [15/Oct/2012:15:42:50.0
"cD" apparently indicates the client timeout status. So, while my webapp says that HAProxy is refusing to accept new connections, HAProxy says that my webapp is not accepting data back.
I do not include my HAProxy configuration because I tried many different parameter values, essentially with the same result. In particular, I set maxconn for both high and low values, both in the global and in the server partitions, and what always happens in statistics is that the maximum sessions are increased by no more than 7. JDBC pool size also high.
Can I use the JDBC pool and the HAProxy pool at all? Before people face such problems?
I have an idea on how to solve this problem, which should send a “validation request” before each request. But there are certain overheads there, and I still would like to know why my webapp works successfully when it goes directly to MySQL, but receives remote connections when going through HAProxy.
How can I debug further and get more information than just "cD"? I tried to launch HAProxy in debug mode, but it seems that I did not find anything else.