test: add a --repeat argument to test_runner.py
ClosedPublic
Actions

Authored by PiRK on Tue, May 6, 10:58.

Details

Reviewers

Fabien

Group Reviewers

Restricted Project

Commits

rABC5c43c94353ca: test: add a --repeat argument to test_runner.py

Summary

This allows repeating the test suite or invidual tests a specified number of time, with the benefit of parallelism.

Test Plan

ninja check-functional

$ test/functional/test_runner.py --repeat 50 p2p_leak rpc_dumpt*
...
p2p_leak.py         | ✓ Passed  | 6 s
p2p_leak.py         | ✓ Passed  | 6 s
p2p_leak.py         | ✓ Passed  | 6 s
p2p_leak.py         | ✓ Passed  | 6 s
...
rpc_dumptxoutset.py | ✓ Passed  | 0 s
rpc_dumptxoutset.py | ✓ Passed  | 1 s
rpc_dumptxoutset.py | ✓ Passed  | 1 s
rpc_dumptxoutset.py | ✓ Passed  | 1 s

ALL                 | ✓ Passed  | 333 s (accumulated)
Runtime: 18 s

Diff Detail

Repository

rABC Bitcoin ABC

Branch

repeat_testr

Lint

Lint Passed

Unit

No Test Coverage

Build Status

Buildable 33222
Build 65927: Build Diff	build-clang-tidy · build-without-wallet · build-debug · build-clang · build-diff
Build 65926: arc lint + arc unit

Event Timeline

PiRK created this revision.Tue, May 6, 10:58

Herald added a reviewer: Restricted Project. · View Herald TranscriptTue, May 6, 10:58

PiRK requested review of this revision.Tue, May 6, 10:58

Harbormaster completed remote builds in B33222: Diff 53870.Tue, May 6, 11:08

Nice ! Should --repeat also imply --failfast ? What do you think ?

In D18054#411619, @Fabien wrote:

Nice ! Should --repeat also imply --failfast ? What do you think ?

Sometimes we want to get a feel of how often it fails in 100 runs, and then --failfast would be an issue. Probably better to let users specify it explicitely.

Fabien accepted this revision.Tue, May 6, 14:51

This revision is now accepted and ready to land.Tue, May 6, 14:51

Closed by commit rABC5c43c94353ca: test: add a --repeat argument to test_runner.py (authored by PiRK). · Explain WhyTue, May 6, 14:58

This revision was automatically updated to reflect the committed changes.

PiRK added a commit: rABC5c43c94353ca: test: add a --repeat argument to test_runner.py.