T164 - Implement junit output support in test_runner.py
ClosedPublic
Actions

Authored by matra774 on Feb 13 2018, 12:49.

Details

Reviewers

Group Reviewers

Restricted Owners Package	(Owns No Changed Paths)
Restricted Project

Maniphest Tasks

T164: Implement junit output support in test_runner.py

Commits

rSTAGING66843371c2fd: T164 - Implement junit output support in test_runner.py
rABC66843371c2fd: T164 - Implement junit output support in test_runner.py

Summary

Test results are saved in test/functional/junit_results.xml (if you have better location, please propose it).

Here is part of output:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite failures="0" id="0" name="bitcoin_abc_tests" skipped="1" tests="65" time="227" timestamp="2018-02-13T06:11:18.717714">
   <testcase classname="abandonconflict.py" name="abandonconflict.py" time="15">
      <system-out>2018-02-13 11:09:07.161000 TestFramework (INFO): Initializing test directory /tmp/bitcoin_test_runner_20180213_060723/abandonconflict_604
2018-02-13 11:09:19.598000 TestFramework (INFO): If balance has not declined after invalidateblock then out of mempool wallet tx which is no longer
2018-02-13 11:09:19.598000 TestFramework (INFO): conflicted has not resumed causing its inputs to be seen as spent.  See Issue #7315
2018-02-13 11:09:19.599000 TestFramework (INFO): 2489.99988480 -&gt; 2489.99988480 ?
2018-02-13 11:09:19.599000 TestFramework (INFO): Stopping nodes
2018-02-13 11:09:21.816000 TestFramework (INFO): Cleaning up
2018-02-13 11:09:21.819000 TestFramework (INFO): Tests successful
</system-out>
      <system-err/>
   </testcase>
   <testcase classname="abc-cmdline.py" name="abc-cmdline.py" time="8">

Unfortunately the test framework does not expose individual asserts/tests so we ouput one testcase per .py file (see abc-p2p-fullblocktest.py, which acutlaz runs 36 tests)

Refs T164

Test Plan

./test/functional/test_runner.py

Diff Detail

Repository

rABC Bitcoin ABC

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

matra774 created this revision.Feb 13 2018, 12:49

Owners added a reviewer: Restricted Owners Package.Feb 13 2018, 12:49

Herald added a reviewer: Restricted Project. · View Herald TranscriptFeb 13 2018, 12:49

Build Bitcoin-ABC :: Staging :: Bitcoin-ABC Staging started.

Harbormaster completed remote builds in B1840: Diff 2863.Feb 13 2018, 13:11

Build Bitcoin-ABC :: Staging :: Bitcoin-ABC Staging passed.

matra774 edited the summary of this revision. (Show Details)Feb 13 2018, 13:54

matra774 edited the summary of this revision. (Show Details)

You are my hero. I'll review this ASAP. A quick passover though:

lgtm

In D1089#20130, @schancel wrote:

You are my hero. I'll review this ASAP. A quick passover though:

thanks :-) It this a duplicate of T128?

Looks great. Thanks again. Just a few minor changes, and it looks like you may need to run autopep8 on the file. Don't hesitate to ping me on telegram if you need anything.

test/functional/test_runner.py
346 ↗	(On Diff #2863)	Please make this configurable on the command line with the default as `junit_results.xml`
562 ↗	(On Diff #2863)	Please put this in as a doc comment for `save_results_as_junit`
593 ↗	(On Diff #2863)	I don't think prettyxml is necessary for this. It's going to be parsed by CI anyways.
597 ↗	(On Diff #2863)	Remove

This revision now requires changes to proceed.Feb 16 2018, 05:40

matra774 marked 4 inline comments as done.Feb 18 2018, 18:16

Addressed review

Harbormaster failed remote builds in B1888: Diff 2919!Feb 18 2018, 18:19

Build Bitcoin-ABC :: Staging :: Bitcoin-ABC Staging started.

Harbormaster completed remote builds in B1888: Diff 2919.Feb 18 2018, 20:07

Build Bitcoin-ABC :: Staging :: Bitcoin-ABC Staging passed.

In D1089#20170, @schancel wrote:

and it looks like you may need to run autopep8 on the file.

autopep8 was already run before the first commit. It looks like arc diff also runs arc lint. Unitl I've installed autopep8 or arc diff kept failing.

schancel accepted this revision.Feb 19 2018, 17:41

schancel added inline comments.

test/functional/test_runner.py
174 ↗	(On Diff #2919)	We should be using `os.path.join` here and with `config.ini`.

This revision is now accepted and ready to land.Feb 19 2018, 17:41

matra774 added inline comments.Feb 19 2018, 22:00

test/functional/test_runner.py
174 ↗	(On Diff #2919)	There are quite a lot of places where string concatenations are used instead of `os.path.join` in `test_runner.py`. I'll fix all of them as part of T163.