Travis test fails but passes on local machine

Question

The following test loads all Markdown files from a path. It works locally, but randomly fails on Travis. Sometimes, it passes without any failures, sometimes it passes on some PHP versions. The class which is being tested is here

public function testLoadMultipleFiles()
{
    $index_content = "<h2>This is a Sub Page Index</h2>" . PHP_EOL . PHP_EOL
        . "<p>This is index.md in the 'sub' folder.</p>" . PHP_EOL . PHP_EOL
        . "<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit.</p>" . PHP_EOL . PHP_EOL
        . "<p>Donec ultricies tristique nulla et mattis.</p>" . PHP_EOL. PHP_EOL
        . "<p>Phasellus id massa eget nisl congue blandit sit amet id ligula.</p>" . PHP_EOL;

    $sub_page_content = "<h2>This is a Sub Page</h2>" . PHP_EOL . PHP_EOL
            . "<p>This is page.md in the 'sub' folder.</p>" . PHP_EOL . PHP_EOL
            . "<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit.</p>" . PHP_EOL . PHP_EOL
            . "<p>Donec ultricies tristique nulla et mattis.</p>" . PHP_EOL . PHP_EOL
            . "<p>Phasellus id massa eget nisl congue blandit sit amet id ligula.</p>". PHP_EOL;

    // Create a stub for the SomeClass class.
    $parser = $this->getMock('Michelf\MarkdownInterface', array('defaultTransform', 'transform'));

    $parser::staticExpects($this->at(0))
            ->method('defaultTransform')
            ->will($this->returnValue($index_content));

    $parser::staticExpects($this->at(1))
            ->method('defaultTransform')
            ->will($this->returnValue($sub_page_content));

    $loader = new MarkdownLoader($parser);

    $files['sub/index.md'] = array(
        'meta'    => array(
            'title'         => 'Sub Page Index'
        ),
        'content' => $index_content
    );

    $files['sub/page.md'] = array(
        'meta'    => array(
            'title'         => 'Sub Page'
        ),
        'content' => $sub_page_content
    );

    $result = $loader->load(ROOT_DIR . 'content/sub', array('md'));
    $this->assertEquals($files, $result);
}

Travis shows the following on a failed run:

There was 1 failure:
1) Zepto\FileLoader\MarkdownLoaderTest::testLoadMultipleFiles
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
     'sub/index.md' => Array (
         'meta' => Array (...)
-        'content' => '<h2>This is a Sub Page Index</h2>
+        'content' => '<h2>This is a Sub Page</h2>

-        <p>This is index.md in the 'sub' folder.</p>
+        <p>This is page.md in the 'sub' folder.</p>
@@ @@
         'meta' => Array (...)
-        'content' => '<h2>This is a Sub Page</h2>
+        'content' => '<h2>This is a Sub Page Index</h2>

-        <p>This is page.md in the 'sub' folder.</p>
+        <p>This is index.md in the 'sub' folder.</p>

         <p>Lorem ipsum dolor sit amet, consectetur adipiscing elit.</p>

         <p>Donec ultricies tristique nulla et mattis.</p>

         <p>Phasellus id massa eget nisl congue blandit sit amet id ligula.</p>
         '
     )
 )

Answer 1

Sorry for the long delay for awnsering.

Just a guess, but do you use Mac OS X locally?

Mac OS X returns lists in alphabetical order, but on linux the file system lists files in a random order.

In your tests error message, you can see that each file read was getting the opposite of what it expects, index or sub page, leading me to believe that the tests failing due to directory listing coming in a random error( The other times when it passes the file system returns the two files in the correct order). I don't know any phpunit so I can't help you with the specifics unfortunately.