HomePhabricator

Search the filesystem to discover tests. Use the timing info from build and src…

Description

Search the filesystem to discover tests. Use the timing info from build and src directory to chose which ones are extended tests and which are regular ones

Summary:
This is the second commit for T163. The hardcoded list of test has been removed, timings are now use to identify extended tests.

Depends on D1116

Test Plan:
Preconditions: make sure, that you do not build from src tree.

--> Test1: In an absence of timing.json, all tests should be run
--> Make sure that timing.json does not exists in any directory
find -name timing.json
build/test/functional/test_runner.py
cat build/timing.json
--> Check if all test are present in timing.json including both 'txn_doublespend.py --mineblock' and 'txn_doublespend.py'

--> Test2: Run with timings.json present:
--> This time timings.json will be used and only test shorter than 40 seconds will be run

build/test/functional/test_runner.py
cat build/timing.json
--> First few test that should be executed are wallet.py,abc-p2p-fullblocktest.py, wallet-hd.py,fundrawtransaction.py
--> dbcrash.py should not be executed

--> Test3: Let's run again, specifying --extended. All test should be executed, regardless of their timings
build/test/functional/test_runner.py --extended
cat build/timing.json

--> Test4: timings from src directory should be used if not found in build
mv build/timing.json test/functional/timing.json
build/test/functional/test_runner.py
--> long running test (dbcrash.py...) should not be executed. They should not be present in build/timing.json
cat build/timing.json

--> Test5: Timings from build should override timings from src
--> Edit build/timing.json, so that it only contains single line:
--> [{ "time": 0, "name": "dbcrash.py" }]
--> run without --extended flag, the test dbcrash should run,
--> because it is classified as not extended in build and extended in src
build/test/functional/test_runner.py

Reviewers: deadalnix, #bitcoin_abc

Reviewed By: deadalnix, #bitcoin_abc

Subscribers: teamcity

Maniphest Tasks: T163

Differential Revision: https://reviews.bitcoinabc.org/D1135