drupalLogin in functional JavaScript tests passes locally but fails on Drupal.org?

Question

During functional JavaScript testing one can create and login a user by using the following code:

$this->testUser = $this->drupalCreateUser(['permission']);
$this->drupalLogin($this->testUser);

This works fine when testing locally with PhantomJS. However after uploading this code to drupal.org it keeps failing randomly (sometimes 1 test, sometimes 2) with the following error:

User dOGIV5Iu successfully logged in.

Failed asserting that false is true.

This appears to be coming from the assertion in BrowserTestBase::drupalLogin, which should normally check that the user is logged in correctly.

$this->drupalGet('user/login');
$this->submitForm([
  'name' => $account->getUsername(),
  'pass' => $account->passRaw,
], t('Log in'));

// @see BrowserTestBase::drupalUserIsLoggedIn()
$account->sessionId = $this->getSession()->getCookie($this->getSessionName());
$this->assertTrue($this->drupalUserIsLoggedIn($account), new FormattableMarkup('User %name successfully logged in.', ['%name' => $account->getAccountName()]));

How come that a test function does work locally, but not on Drupal.org? Even stranger is that this function is used throughout multiple core tests without issues. And why is this assertion failing randomly?

Answer 1

After reading through the Jenkins results for the nth time, it finally occurred to me that during Drupal core (and other contrib) tests all JavaScript tests always run separately from the other tests. All my tests however were in the same list.

This happened because all test classes started originally as functional tests, but as I went ahead some of them were changed into functional javascript tests. Class extending was changed into JavascriptTestBase and the tests were working perfectly fine. At least, when running individually. Running multiple tests at the same time made them somehow interfere with one another.

The problem was that I had never changed the namespace from Functional into FunctionalJavascript. The Drupal testrunner makes sure that all FunctionalJavascript tests are ran one by one, but performs other tests in parallel.

After changing the namespace, the tests were listed separate from the rest, ran after each other and finally showed up green on Drupal.org