Magento2 - Why would Magento Test Framework - Integration tests fail on a clean composer install

Question

Firstly as I'm using Magento Enterprise I don't think I can pull against a git repository where I know the tests are passing. I presume that these are being run in travis-ci.com rather than .org

Secondly I'm not sure I would want to anyway, as the github repro's have all the code in /app/code rather than /vendor.

So because I want to start to build out my Continuous Integration Pipeline, I follow the below steps:

1. Install EE v2.1.7 using composer
2. Commit to private git repository
3. Connect private repository to private travis-ci.com account
4. Edit .travis.yml to add encrypted credentials

5. Edit .travis.yml to give correct execute permissions on: ./dev/travis/before_install.sh and ./dev/travis/before_script.sh

   before_install:
     - chmod +x ./dev/travis/before_install.sh
     - ./dev/travis/before_install.sh
   install:
     - echo "{\"http-basic\":{\"repo.magento.com\":{\"username\":\"${MAGENTO_USERNAME}\",\"password\":\"${MAGENTO_PASSWORD}\"}}}" > auth.json
     - composer install --no-interaction --prefer-dist
   before_script:
     - chmod +x ./dev/travis/before_script.sh
     - ./dev/travis/before_script.sh
   

6. Commit and Push to git

7. Check build at travis-ci.com

Expect Result - build passes Actual Result - build fails

As I said above the community git repository has the code in /app/code are they only running tests from an /App/code setup? Is there some compatibility issue here from running them from the vendor folder.

I'm getting numerous. PHP Fatal error: Uncaught exception 'Exception' with message 'Asymmetric transaction rollback.' As pointed out here "Asymmetric transaction rollback error" in integration test this is a false error message as it means that an exception has been re-thrown in the commit callbacks - meaning that the true error message is lost.

One true stack trace is:

There was 1 error:
1) Magento\Customer\Model\GroupManagementTest::testGetDefaultGroupWithNonDefaultStoreId
Magento\Framework\Exception\NoSuchEntityException: No such entity with groupId = , storeId = 5

• Why would a clean install fail? - presumably the core-team's travis file has the correct set up to create environments for passing tests?
• And presumably they have actually run these tests?

I am really hoping this is environmental. I don't want to be digging around in the core looking for core bugs - I want to start with a nice clean build before I start adding stuff...

Any ideas?

Answer 1

Answering my own question:

After a fortnight, I now have a passing build...

I've identified and fixed four problems:

1. One failed test is fixed by applying the patch for this issue which has been applied to the develop branch but not to the 2.1 release https://github.com/magento/magento2/issues/7746

2. The majority of other issues were initally being obscured by the problem identified here. Once I eliminated the noise, I worked out that the actual failures can be eliminated if we exclude solar and ElasticSearch tests.

   This is because both install-config-mysql.travis.php.dist and install-config-mysql.php.dist only have connection details for MySql and not for Solr or ElasticSearch. (Really both should be excluded in the phpunit.xml.dist, A vendor can include them when required when using either of these engines and wanting to include the tests in their CI)

3. My next failing test is:

   There was 1 failure:
   
   Magento\Catalog\Cron\DeleteOutdatedPriceValuesTest::testExecute
   Failed asserting that two strings are equal.
   --- Expected
   +++ Actual
   @@ @@
   -'10.0000'
   +'9.9900'
   

   Reading the code, the test is assuming that a config setting is set as default, which doesn't appear to be true... I'm going to look into this further and will post back with a solution but presently I've just excluded the test.

4. Finally dev/travis/before_script.sh is connecting to the github repository for Community Edition in the section for phpcs static tests to look for changed code to apply tests to. I assume this is done to apply coding standards only to new code without having to refactor the entire codebase. BUT this is an enterprise build... so we shouldn't connect to CE. So I've changed the connection details to point at my private github repro.

   cd dev/tests/static
   
   echo "==> preparing changed files list"
   changed_files_ce="$TRAVIS_BUILD_DIR/dev/tests/static/testsuite/Magento/Test/_files/changed_files_ce.txt"
   php get_github_changes.php \
       --output-file="$changed_files_ce" \
       --base-path="$TRAVIS_BUILD_DIR" \
       --repo='https://github.com/magento/magento2.git' \
       --branch='2.1'
   cat "$changed_files_ce" | sed 's/^/  + including /'
   

The core team don't appear to be source controlling these edits to the configuration for EE - perhaps, like me, they are using patches to make the changes.

In itself this is fair enough, but the lack of documentation on how to set up Travis for CE and EE is very disappointing, it's caused me a lot of work. (I did put in an Enterprise support ticket, but no joy from them, and I had to personally fix these configuration errors).