Page MenuHomePhabricator

Use functional tests timings from the source directory
ClosedPublic

Authored by Fabien on Jul 8 2019, 19:08.

Details

Summary

The current behavior is to read timing from the timing.json file
location in the build directory is it exists.
But this file is updated every time the functional test suite is run,
which makes it impossible to predict which test will run as part of the
--extended call and which one will run as part as a standard call.
This diffs change this behavior so that the timing.json file from the
source directory is always used as the timings source to determine which
test to run.

Thanks @markblundeberg for reporting the issue.

Test Plan
./test/functional/test_runner.py

Change the timing for a previously ran test to a duration > 40s in the
test/functional/timing.json file.

./test/functional/test_runner.py

The test should not run anymore.

Diff Detail

Repository
rABC Bitcoin ABC
Branch
timings_read_from_src
Lint
Lint Passed
Unit
No Test Coverage
Build Status
Buildable 6739
Build 11525: Bitcoin ABC Buildbot (legacy)
Build 11524: arc lint + arc unit

Event Timeline

The inspiration for this change was an issue I was seeing: if a test once took longer than 40s to run, then it would never run again until I ran make clean since the last memory of running that test would be >40s. So the number of tests that would actually run was smaller than TeamCity, and gradually reducing over time. Not exactly the desired situation!

This revision is now accepted and ready to land.Jul 8 2019, 20:11