How to debug using Rails connection pool?

Question

I have problems with Sidekiq employees.

ActiveRecord::ConnectionTimeoutError: could not obtain a database connection within 5.000 seconds (waited 5.000 seconds) 

I follow the recommendations for using ActiveRecord::ConnectionTimeoutError and a large enough connection pool.

I want to find out if I have exhausted the connection pool. I register size and connections.length from ActiveRecord::Base.connection_pool , but they remain at a constant size = 100 connections .length = 5. This suggests that this is not a resource leak problem.

My MySQL server is configured to allow up to 400 concurrent connections.

My work ended as follows:

 class MyJob < ActiveJob::Base queue_as :default rescue_from StandardError do |exception| # clear connections on exception. Not sure if this is a good idea or not. ActiveRecord::Base.clear_active_connections! end def perform() logger.info "size" logger.info ActiveRecord::Base.connection_pool.instance_eval { @size } logger.info "connections" logger.info ActiveRecord::Base.connection_pool.instance_eval { @connections }.length # Ensure connections come from connection pool. ActiveRecord::Base.connection_pool.with_connection do |conn| # do stuff end end end 

Is this the right way to diagnose the cause of this, be it resource starvation or leakage? Are there other methods that I can use to understand why this is happening?

Answer 1

This ActiveRecord::ConnectionTimeoutError can, in my opinion, occur only in one scenario - when there are so many threads that want to use database connections that the pool is exhausted and even waits for a free connection, it does not help (as described in the source code ).

In your case, this is strange. You use only 25 worker threads, but 100 connections are established in the pool, so there are many reserves. I still suspect you should have threads. Perhaps you make some threads in your workplaces? Perhaps you are using a gem that creates streams in your work?

In any case, if you can throw an exception, I would suggest catching it and getting a list of all the threads at the time it occurred, something like this:

 begin # job stuff... rescue ActiveRecord::ConnectionTimeoutError puts "listing #{Thread.list.count} threads:" Thread.list.each_with_index do |t,i| puts "---- thread #{i}: #{t.inspect}" puts t.backtrace.take(5) end end 

I expect there will be 100 or more threads, and you should see exactly where they got stuck from the backtrace.