How to collect data results in pytest using junitxml?

Question

Let's use the following code (conftest.py) :

import random
def test_val():
    value = random.random()
    assert value < 0.5

Running py.test --junitxml=result.xml conftest.py generates result.xml (when the test passes):

<?xml version="1.0" encoding="utf-8"?>
<testsuite errors="0" failures="0" name="" skips="0" tests="1" time="0.047">
<testcase classname="conftest" name="test_val" time="0.0"/>
</testsuite>

Now. What I'd like to be able to do is to store the value generated by test_val() in results.xml. Is there a way to do it ? I can't seem to find anything related in pytest doc.

Answer 1

The shipped junitxml plugin does not have hooks to add such data you can print it to stdout though, since that gets added to the junitxml data.

So as long as you print out logs you will at least be able to know the data.

Answer 2

You can solve your problem but not with junitxml.

You can use pytest-harvest for this. Simply install it, and you can directly use the pre-defined fixtures:

import random

def test_val(results_bag):
    value = random.random()

    # save it (before test !)
    results_bag.value = value

    assert value < 0.5

def test_synthesis(module_results_df):
    """
    Shows that the `module_results_df` fixture already contains what you need
    """
    # drop the 'pytest_obj' column
    module_results_df.drop('pytest_obj', axis=1, inplace=True)

    print("\n   `module_results_df` dataframe:\n")
    print(module_results_df)

Yields

>>> pytest -s -v

============================= test session starts =============================
collecting ... collected 2 items
tmp.py::test_val PASSED
tmp.py::test_synthesis 
   `module_results_df` dataframe:

          status  duration_ms     value
test_id                                
test_val  passed     0.999928  0.443547
PASSED

========================== 2 passed in 1.08 seconds ===========================

You could then decide to dump the dataframe as a csv or any other format in a dedicated file. Note that you do not have to write the above in a test, you can do it from any of the pytest hooks (either where you have access to the above fixtures, or to a pytest request.session object).

See documentation for details.

Finally, if you wish to also use parameters, fixtures, steps... you might wish to look at this datascience benchmark example

I'm the author by the way ;)

Answer 3

Years have passed and the best solution should be noted.

From the Pytest's record_property fixture docs:

Add extra properties to the calling test.

User properties become part of the test report and are available to the configured reporters, like JUnit XML.

The fixture is callable with name, value. The value is automatically XML-encoded.

Properties are captured for both passed and failed cases.

Suite:

def test_passes(record_property):
    record_property("key", "value1")
    assert 1 == 1


def test_fails(record_property):
    record_property("key", "value2")
    assert 1 == 2

Result when ran pytest with --junitxml=result.xml

Generated Junit test report:

<?xml version="1.0" encoding="utf-8"?>
<testsuites>
    <testsuite name="pytest" errors="0" failures="1" skipped="0" tests="2" time="0.085"
               timestamp="2021-04-12T14:25:09.900867" hostname="DESKTOP">
        <testcase classname="test_something" name="test_passes" time="0.001">
            <properties>
                <property name="key" value="value1"/>
            </properties>
        </testcase>
        <testcase classname="test_something" name="test_fails" time="0.001">
            <properties>
                <property name="key" value="value2"/>
            </properties>
            <failure message="assert 1 == 2">record_property = &lt;function record_property.&lt;locals&gt;.append_property
                at 0x000001A1A9EB40D0&gt;

                def test_fails(record_property):
                record_property("key", "value2")
                &gt; assert 1 == 2
                E assert 1 == 2

                test_something.py:8: AssertionError
            </failure>
        </testcase>
    </testsuite>
</testsuites>

If the version of Pytest you run is quite new, you'll see deprecation warning

test_something.py::test_fails
  test_something.py:6: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
    def test_fails(record_property):

This is due to the fact that schema of Xunit reports had changed. From Pytest changelog:

• record_property now emits a PytestWarning when used with junit_family=xunit2: the fixture generates property tags as children of testcase, which is not permitted according to the most recent schema <https://github.com/jenkinsci/xunit-plugin/blob/master/

To overcome warnings you can:

1. Run Pytest with override flag -o junit_family="xunit1" or place this property on pytest.ini

2. Use record_testsuite_property session-scoped fixture. Nevertheless this allows only to attach properties to test suite level.