Heroku, Puma, Redis, Sidekiq and connection restrictions, the maximum number of ERR clients

I use hero, puma, redis to go with sidekiq. when I try to do some background work that time throwsYour Redis connection pool is too small for Sidekiq to work. Your pool has 20 connections but really needs to have at least 22

Below are the configuration files, please help find a solution

enter image description here

I use

Redis To Go Instance: Mini

PROCFILE

web: bundle exec puma -C config/puma.rb  
worker: bundle exec sidekiq -C config/sidekiq.yml 

puma.rb

workers Integer(ENV['WEB_CONCURRENCY'] || 2)  
threads_count = Integer(ENV['MAX_THREADS'] || 1)  
threads threads_count, threads_count

preload_app!

rackup      DefaultRackup  
port        ENV['PORT']     || 3000  
environment ENV['RACK_ENV'] || 'development'

# Because we are using preload_app, an instance of our app is created by master process (calling our initializers) and then memory space
# is forked. So we should close DB connection in the master process to avoid connection leaks.
# https://github.com/puma/puma/issues/303
# http://stackoverflow.com/questions/17903689/puma-cluster-configuration-on-heroku
# http://www.rubydoc.info/gems/puma/2.14.0/Puma%2FDSL%3Abefore_fork
# Dont have to worry about Sidekiq connection to Redis because connections are only created when needed. As long as we are not
# queuing workers when rails is booting, there will be no redis connections to disconnect, so it should be fine.
before_fork do  
  puts "Puma master process about to fork. Closing existing Active record connections."
  ActiveRecord::Base.connection.disconnect!
end

on_worker_boot do  
  # Worker specific setup for Rails 4.1+
  # See: https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-boot
  ActiveRecord::Base.establish_connection
end 

In sidekiq.yml

development:  
  :concurrency: 5
production:  
  :concurrency: 20
:queues:
  - default

sidekiq.rb

Sidekiq.configure_client do |config|
  config.redis = { url: ENV['REDISTOGO_URL'], size: 2 }
end

Sidekiq.configure_server do |config|
  config.redis = { url: ENV['REDISTOGO_URL'], size: 20 }

  Rails.application.config.after_initialize do
    Rails.logger.info("DB Connection Pool size for Sidekiq Server before disconnect is: #{ActiveRecord::Base.connection.pool.instance_variable_get('@size')}")
    ActiveRecord::Base.connection_pool.disconnect!

    ActiveSupport.on_load(:active_record) do
      config = Rails.application.config.database_configuration[Rails.env]
      config['reaping_frequency'] = ENV['DATABASE_REAP_FREQ'] || 10 # seconds
      # config['pool'] = ENV['WORKER_DB_POOL_SIZE'] || `
      config['pool'] = 16
      ActiveRecord::Base.establish_connection(config)

      Rails.logger.info("DB Connection Pool size for Sidekiq Server is now: #{ActiveRecord::Base.connection.pool.instance_variable_get('@size')}")
    end
  end
end

After deploying this configuration, it throws below errors,

Your Redis connection pool is too small for Sidekiq to work. Your pool has 20 connections, but should actually be at least 22

I later updated the data below in the sidkiq.rb file,

Sidekiq.configure_client do |config|
  config.redis = { url: ENV['REDISTOGO_URL'], size: 2 }
end

Sidekiq.configure_server do |config|
  config.redis = { url: ENV['REDISTOGO_URL'], size: 22 }

  Rails.application.config.after_initialize do
    Rails.logger.info("DB Connection Pool size for Sidekiq Server before disconnect is: #{ActiveRecord::Base.connection.pool.instance_variable_get('@size')}")
    ActiveRecord::Base.connection_pool.disconnect!

    ActiveSupport.on_load(:active_record) do
      config = Rails.application.config.database_configuration[Rails.env]
      config['reaping_frequency'] = ENV['DATABASE_REAP_FREQ'] || 10 # seconds
      # config['pool'] = ENV['WORKER_DB_POOL_SIZE'] || `
      config['pool'] = 16
      ActiveRecord::Base.establish_connection(config)

      Rails.logger.info("DB Connection Pool size for Sidekiq Server is now: #{ActiveRecord::Base.connection.pool.instance_variable_get('@size')}")
    end
  end
end

This time he throws another problem that

The maximum number of ERR customers has reached

I only get this problem in production, please help me

+4

Source: https://habr.com/ru/post/1688256/


All Articles