Page MenuHomePhabricator

T164 - Implement junit output support in test_runner.py
ClosedPublic

Authored by matra774 on Feb 13 2018, 12:49.

Details

Summary

Test results are saved in test/functional/junit_results.xml (if you have better location, please propose it).

Here is part of output:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite failures="0" id="0" name="bitcoin_abc_tests" skipped="1" tests="65" time="227" timestamp="2018-02-13T06:11:18.717714">
   <testcase classname="abandonconflict.py" name="abandonconflict.py" time="15">
      <system-out>2018-02-13 11:09:07.161000 TestFramework (INFO): Initializing test directory /tmp/bitcoin_test_runner_20180213_060723/abandonconflict_604
2018-02-13 11:09:19.598000 TestFramework (INFO): If balance has not declined after invalidateblock then out of mempool wallet tx which is no longer
2018-02-13 11:09:19.598000 TestFramework (INFO): conflicted has not resumed causing its inputs to be seen as spent.  See Issue #7315
2018-02-13 11:09:19.599000 TestFramework (INFO): 2489.99988480 -&gt; 2489.99988480 ?
2018-02-13 11:09:19.599000 TestFramework (INFO): Stopping nodes
2018-02-13 11:09:21.816000 TestFramework (INFO): Cleaning up
2018-02-13 11:09:21.819000 TestFramework (INFO): Tests successful
</system-out>
      <system-err/>
   </testcase>
   <testcase classname="abc-cmdline.py" name="abc-cmdline.py" time="8">

Unfortunately the test framework does not expose individual asserts/tests so we ouput one testcase per .py file (see abc-p2p-fullblocktest.py, which acutlaz runs 36 tests)

Refs T164

Test Plan

./test/functional/test_runner.py

Diff Detail

Repository
rABC Bitcoin ABC
Branch
T164_junit
Lint
Lint Passed
Unit
No Test Coverage
Build Status
Buildable 1888
Build 1936: Bitcoin ABC Buildbot (legacy)
Build 1935: arc lint + arc unit

Event Timeline

Owners added a reviewer: Restricted Owners Package.Feb 13 2018, 12:49
matra774 edited the summary of this revision. (Show Details)

You are my hero. I'll review this ASAP. A quick passover though:

lgtm

You are my hero. I'll review this ASAP. A quick passover though:

thanks :-) It this a duplicate of T128?

schancel requested changes to this revision.Feb 16 2018, 05:40

Looks great. Thanks again. Just a few minor changes, and it looks like you may need to run autopep8 on the file. Don't hesitate to ping me on telegram if you need anything.

test/functional/test_runner.py
346 ↗(On Diff #2863)

Please make this configurable on the command line with the default as junit_results.xml

562 ↗(On Diff #2863)

Please put this in as a doc comment for save_results_as_junit

593 ↗(On Diff #2863)

I don't think prettyxml is necessary for this. It's going to be parsed by CI anyways.

597 ↗(On Diff #2863)

Remove

This revision now requires changes to proceed.Feb 16 2018, 05:40

and it looks like you may need to run autopep8 on the file.

autopep8 was already run before the first commit. It looks like arc diff also runs arc lint. Unitl I've installed autopep8 or arc diff kept failing.

schancel added inline comments.
test/functional/test_runner.py
174

We should be using os.path.join here and with config.ini.

This revision is now accepted and ready to land.Feb 19 2018, 17:41
test/functional/test_runner.py
174

There are quite a lot of places where string concatenations are used instead of os.path.join in test_runner.py.
I'll fix all of them as part of T163.

This revision was automatically updated to reflect the committed changes.