deadlock detected with capybara-webkit

Question 1

This is happening because you have two threads, the test and the request, modifying the database asynchronously. It looks like you've rightly been experimenting with different configurations in your spec_helper. I've spend a fair amount of time struggling with this same issue, and have come up with this:

# adapted from https://gist.github.com/moonfly/4950750

require 'database_cleaner'

RSpec.configure do |config|

  config.use_transactional_fixtures = false

  config.before( :suite ) do
    DatabaseCleaner.clean_with :truncation
    DatabaseCleaner.strategy = :transaction
  end

  config.around( :each ) do |spec|
    if spec.metadata[:js]
      # JS => doesn't share connections => can't use transactions
      spec.run
      DatabaseCleaner.clean_with :deletion
    else
      # No JS/Devise => run with Rack::Test => transactions are ok
      DatabaseCleaner.start
      spec.run
      DatabaseCleaner.clean

      # see https://github.com/bmabey/database_cleaner/issues/99
      begin
        ActiveRecord::Base.connection.send :rollback_transaction_records, true
      rescue
      end
    end
  end

end

I keep it all in support/database_cleaner_configuration.rb, which would work for you, as well, or you can simply replace what you've got in your spec_helper. If this doesn't work, let me know - I have a few other ideas, but they're kookier, and not worth getting into if this works, which it probably will...

Note:

It might be worth mentioning that Rails 5.1+ solves the database problem. Eileen Uchitelle on the Rails team made the changes necessary to run ensure test threads and the Rails server can run in the same process by sharing the database connection.

Question 2

You have the deadlock when there are two threads which run the tests. For example open a tab in your terminal and run tests with the rspec command. Immediately open second tab in the terminal and run the tests there too with the same command.

So, I guess you just have run all tests in one tab and trying to run particular test with the rspec spec/acceptances/provider_services_spec.rb:31 command in other tab.

Question 3

The most probable reason is there must be some query going on through ajax / database cleaner is truncating the database prematurely while sql query is running. One of the method is to make webkit driver wait till the ajax request is finished. Add this method in the test helpers and call it after clicking the button / submitting the ajax form.

    def wait_for_ajax_to_finish
    looping = true
    loop do
      sleep 1
      begin
        count = page.evaluate_script('window.running_ajax_calls').to_i
        looping = false if count == 0
      rescue => e
        if e.message.include? 'HTMLunitCorejsJavascript::Undefined'
          raise "For 'wait for the AJAX call to finish' to work, please include culerity.js after including jQuery."
        else
          raise e
        end
      end
    end
  end

Question 4

I was having different tests fail randomly (like 1 test out of 500, every other time I ran the suite).

This may not be the prettiest solution, but I solved the problem by just rescuing and retrying the database cleaner after 2 seconds, like so:

config.after(:each) do |example|
  begin
    DatabaseCleaner.clean
  rescue Exception => e
    # Recover from thread locks and retry the database clean after a slight delay
    sleep 2
    DatabaseCleaner.clean
  end
end