Page MenuHomePhabricator

Search the filesystem to discover tests. Use the timing info from build and src directory to chose which ones are extended tests and which are regular ones
ClosedPublic

Authored by matra774 on Feb 25 2018, 17:22.

Details

Summary

This is the second commit for T163. The hardcoded list of test has been removed, timings are now use to identify extended tests.

Depends on D1116

Test Plan

Preconditions: make sure, that you do not build from src tree.

--> Test1: In an absence of timing.json, all tests should be run
--> Make sure that timing.json does not exists in any directory
find -name timing.json
build/test/functional/test_runner.py
cat build/timing.json
--> Check if all test are present in timing.json including both 'txn_doublespend.py --mineblock' and 'txn_doublespend.py'

--> Test2: Run with timings.json present:
--> This time timings.json will be used and only test shorter than 40 seconds will be run

build/test/functional/test_runner.py
cat build/timing.json
--> First few test that should be executed are wallet.py,abc-p2p-fullblocktest.py, wallet-hd.py,fundrawtransaction.py
--> dbcrash.py should not be executed

--> Test3: Let's run again, specifying --extended. All test should be executed, regardless of their timings
build/test/functional/test_runner.py --extended
cat build/timing.json

--> Test4: timings from src directory should be used if not found in build
mv build/timing.json test/functional/timing.json
build/test/functional/test_runner.py
--> long running test (dbcrash.py...) should not be executed. They should not be present in build/timing.json
cat build/timing.json

--> Test5: Timings from build should override timings from src
--> Edit build/timing.json, so that it only contains single line:
--> [{ "time": 0, "name": "dbcrash.py" }]
--> run without --extended flag, the test dbcrash should run,
--> because it is classified as not extended in build and extended in src
build/test/functional/test_runner.py

Diff Detail

Repository
rABC Bitcoin ABC
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

deadalnix requested changes to this revision.Mar 2 2018, 09:53

Overall, it looks good. A few small changes and it can get in.

test/functional/test_runner.py
195 ↗(On Diff #2982)

I wrote two comment that I deleted. This is very confusing. The timing object should definitively take a file rather than a folder.

199 ↗(On Diff #2982)

I think this should be done by get_tests_to_run . I don't think there is a use case where we want one but not the other, so there is no point exposing both as functions.

This revision now requires changes to proceed.Mar 2 2018, 09:53

Fix according to review comments

test/functional/test_runner.py
456 ↗(On Diff #3027)

You probably want to time test with each parameter differently. If that's hard to do, we should go ahead with the patch that way and make a task. If it is easy, you should do this.

test/functional/test_runner.py
456 ↗(On Diff #3027)

The for loop starting in line 448 create multiple entries if a test is present in test_params.

Example: it currently creates two entries for txn_doublespend and they are executed independently as they were two separate tests. The timing information is also recorded twice.

{
 "name": "txn_doublespend.py",
 "time": 14
},
{
 "name": "txn_doublespend.py --mineblock",
 "time": 15
}

Is this OK or did you expect something else?

If you modify TEST_PARAMS so that it contains:

"txn_doublespend.py": [["--mineblock"],["--anotherInvocation", "--withTwoParameters"] ],

it will execute test three times and record three timing - for each of the following:
txn_doublespend.py, txn_doublespend.py --mineblock txn_doublespend.py --anotherInvocation --withTwoParameters.

It this OK?

deadalnix added inline comments.
test/functional/test_runner.py
456 ↗(On Diff #3027)

yes this is to be the right behavior.

This revision is now accepted and ready to land.Mar 5 2018, 14:59

Rebased on master to resolve merge conflicts (some tests were added to TEST_LIST, which is being deleted bz this DIFF)
I've also re-run all tests to get updated timing.json, which now includes the new tests.

This revision was automatically updated to reflect the committed changes.