What are good debugging strategies for MFTF (v2) tests which only faily sometimes?

Question

We have a MFTF test related to coupon codes that fail in 4 of 10 cases.

How can we debug this properly.

We already isolated the single test case and can reproduce it locally, it still takes 50 minutes to run 10 times.

Now we are looking for good strategies to debug that.

Answer 1

First idea:

Put a

<wait stepKey="almostForever" time="65535">

at the part where the test fails and try to reproduce the problem manually from there on

Interactive pause:

https://devdocs.magento.com/mftf/docs/interactive-pause.html#use-pause-on-test-failure

But this works only for MFTF v3 - we are currently on MFTF v2

Next idea: Manipulate the PHP code MFTF generates.

1. vendor/bin/mftf generate:tests
   

2. edit the PHP file in _generated folder like this:

   try {
   
       $I->see( ... );
   catch (\Exception $e) {
       sleep(10000000);
   }
   

3. vendor/bin/mftf run:group --skip-generated   # call 10 times
   

Further approaches:

• Comment out After-Suite to avoid cleaning up the database