How to prevent "Mysql2::Error: This connection is in use by" with Sidekiq

Question

After running Sidekiq for a couple of hours, I see a bunch of jobs fail with Mysql2::Error: This connection is in use by: #<Celluloid::Thread:0x0000000d1b56e0 sleep>. Seems the Sidekiq threads are somehow conflicting over the MySQL connection pool.

concurrency is set to the default 25 in sidekiq.yml and the pool is 28 in database.yml. There are no long-lived queries and the exceptions happen in standard finder calls, nothing fancy.

How can I prevent this error to ensure jobs run smoothly?

Answer 1

Finally fixed this by isolating all the workers. It came down to Typhoeus library apparently not being thread-safe. Replaced with Net/HTTP and it works again.

Some more details at https://github.com/mperham/sidekiq/issues/1400#issuecomment-45838886

Answer 2

Your problem is caused because sidekiq is getting all connections to your DB and at the same time the rails app is also requesting the connections.

You have 25 sidekiq workers but how much rails servers do you have?

e.g. if you have unicorn running 4 child workers you'll need 29 slots (at least)

Answer 3

In our case, we had a lot of these errors in a specific worker type. We identified that we were using Timeout.timeout() calls in one of the jobs running in these workers. We removed those calls and these errors went away after that. For some reference as to why Timeout.timeout() calls are dangerous, please look here and here.