Configuring pytest junitxml error reports

Question

I am trying to investigate test failures and include additional data in the junit xml test report. In particular, this is a set of functional tests for an external product, and I want to include product logs in error reports.

Using the method found here , I was able to print the logs to stdout before doing the multicast, which ultimately appears in the jenkin error report. But I'm sure there is a better way to achieve this.

I tried using hook pistest_runtest_logreport to add logs to the 'sections' attribute, which already contains the 'capture stdout' and 'capture stderr' streams. But recently added sections do not fall into the xml file. I tried this technique directly in hook pytest_runtest_makereport, with similar results.

The release notes for pytest 2.7 indicate that the use of multicast support is reduced by 2.8 and that @ pytest.mark.hookwrapper is a new way to do this, however I cannot get this to work at all - yield "returns None instead of the CallOutcome object (I tried it in the makreport hook.) And even if he returned something, I'm not sure that I can add material to it that will be displayed in the xml report.

Is there some kind of functionality that I'm missing that will allow me to do this in a flexible way? (flexible I mean: not tied to stdout or call logging like the capture-logs plugin)

Answer 1

EDIT: since for my report I need access to the functions of the test item (and test results), I was able to move the logic to pytest_runtest_makereport(item, __multicall__) . The trick is to perform multicast, which returns a report object:

 @pytest.mark.tryfirst def pytest_runtest_makereport(item, call, __multicall__): report = __multicall__.execute() # then I was able to manipulate report and get the same results as below 

---

Bruno's answer gave me the motivation I need for a more thorough analysis of this function :)

So how does it work:

 def pytest_runtest_logreport(report): if report.failed: report.longrepr.sections.append(("Header", "Message", "-")) report.sections.append(("Captured stdout", "This is added to stdout")) report.sections.append(("Captured stderr", "This is added to stderr")) report.sections.append(("Custom Section", "This can only be seen in the console - the xml won't have it.")) 

The longrepr attribute is only available in case of failures. A 3-tuple is required, the last value is the character used to decorate the decoration / surrounding heading. It will appear in the "Failure" section of the report:

 ----------------------------------- Header ------------------------------------ Message 

Custom sections will create additional sections of results that will be printed to the console. . But they will not use junitxml:

 ------------------------------- Custom Section -------------------------------- This can only be seen in the console - the xml won't have it. 

There are only 2 sections in the junitxml report: out and err. To add custom text to it, you must create sections called "Captured std", and only those that fall into the xml file. Any other name will create a custom section that will only be displayed on the console.

Here is the result of junitxml using the code above, with some reformatting for this post:

 <?xml version="1.0" encoding="utf-8" ?> <testsuite errors="0" failures="1" name="pytest" skips="0" tests="1" time="0.646"> <testcase classname="test_reporting" name="test_fail" time="0.000999927520752"> <failure message="test failure"> @ut def test_fail(): > assert 0, "It failed" E AssertionError: It failed E assert 0 test_reporting.py:346: AssertionError ----------------------------------- Header ------------------------------------ Message </failure> <system-out>This is added to stdout</system-out> <system-err>This is added to stderr</system-err> </testcase> </testsuite> 

Answer 2

To add information to the test report (XML, console, or otherwise), view intercept messages , specifically pytest_runtest_logreport .

Answer 3

Using pytest_runtest_makereport(item, call) with @pytest.hookimpl(hookwrapper=True) allows you to access the report before it is used to create XML, for example.

 @pytest.hookimpl(hookwrapper=True) def pytest_runtest_makereport(item, call): result = yield if result.get_result().failed: report.longrepr.addsection("Header","message") 

with pytest-3.9.3

Although the documentation for hookwrapper really warns

If the result of the underlying hook is a mutable object, they can change this result, but it's probably best avoided.