Page MenuHomePhabricator
Differential D17330

Run more functional tests in parallel by default
ClosedPublic
Actions

Authored by roqqit on Mon, Dec 9, 22:47.

Details

Reviewers
Fabien
Group Reviewers
Restricted Owners Package(Owns No Changed Paths)
Restricted Project
Commits
rABC8917a6f8af6c: Run more functional tests in parallel by default
Summary

Defaults should always provide the best dev experience except in specialized cases.

Bumping test_runner's default jobs from 1/3 of CPUs to all of them translates to a ~58% speed up in time to complete ninja check-functional

More detailed rationale:

https://reviews.bitcoinabc.org/D1473#30006 indicates 1/3 of CPUs was chosen because one job might run multiple nodes but there was no basis for this fraction.

If we look at this more empirically, we find that tests with num_nodes set to 1 make up ~68%, 2 is ~21%, and 3+ is ~11% of tests. This suggests we can be more aggressive with the number of default jobs and by not doing so we are wasting valuable dev time.

On my machine (16 cpus), ninja check-functional completed in this amount of time per fraction of cpus:
1/3 = 255s
1/2 = 180s
3/5 = 162s
2/3 = 163s
cpus-1 = 122s
cpus = 106s
cpus+1 = 98s

Given the above, we can actually be way more aggressive than some fraction of CPUs. I suspect this is because a lot of time is spent on tests waiting and for I/O operations to complete.

Test Plan
ninja check-functional

Diff Detail

Event Timeline

roqqit created this revision.Mon, Dec 9, 22:47
Owners added a reviewer: Restricted Owners Package.Mon, Dec 9, 22:47
Herald added a reviewer: Restricted Project. · View Herald TranscriptMon, Dec 9, 22:47
roqqit requested review of this revision.Mon, Dec 9, 22:47
Harbormaster completed remote builds in B31595: Diff 51510.Mon, Dec 9, 23:01
Fabien accepted this revision.Tue, Dec 10, 08:08
Fabien added a subscriber: Fabien.
Fabien added inline comments.
test/functional/test_runner.py
152

why not cpus + 1 then ?

This revision is now accepted and ready to land.Tue, Dec 10, 08:08
PiRK added a subscriber: PiRK.Tue, Dec 10, 08:45

Benchmark on my machine:

DEFAULT_JOBS =  (multiprocessing.cpu_count() // 3) + 1
ALL                                        | ✓ Passed  | 1603 s (accumulated)
Runtime: 179 s


DEFAULT_JOBS = multiprocessing.cpu_count()
ALL                                        | ✓ Passed  | 2187 s (accumulated)
Runtime: 95 s

DEFAULT_JOBS = 2 * multiprocessing.cpu_count()
ALL                                        | ✖ Failed  | 2583 s (accumulated)
Runtime: 68 s

With one port issue in the 2 * cpu_count attempt.

RuntimeError: Could not find available PortName.P2P port after 5 attempts.
roqqit added a comment.Tue, Dec 10, 16:47
In D17330#393720, @PiRK wrote:

Benchmark on my machine:

DEFAULT_JOBS =  (multiprocessing.cpu_count() // 3) + 1
ALL                                        | ✓ Passed  | 1603 s (accumulated)
Runtime: 179 s


DEFAULT_JOBS = multiprocessing.cpu_count()
ALL                                        | ✓ Passed  | 2187 s (accumulated)
Runtime: 95 s

DEFAULT_JOBS = 2 * multiprocessing.cpu_count()
ALL                                        | ✖ Failed  | 2583 s (accumulated)
Runtime: 68 s

With one port issue in the 2 * cpu_count attempt.

RuntimeError: Could not find available PortName.P2P port after 5 attempts.

Interesting! I had not considered going so much higher than the CPU count.

test/functional/test_runner.py
152

The 1-2% improvement didn't seem justified when it is reasonable to assume devs will have other stuff going on on the same machine.

roqqit marked an inline comment as not done.Tue, Dec 10, 17:02
Closed by commit rABC8917a6f8af6c: Run more functional tests in parallel by default (authored by roqqit). · Explain WhyTue, Dec 10, 18:08
This revision was automatically updated to reflect the committed changes.
roqqit added a commit: rABC8917a6f8af6c: Run more functional tests in parallel by default.

Revision Contents
Changeset List

PathSize
doc/
2 lines
test/
functional/
2 lines

Diff 51510

View Options

doc/functional-tests.md

Loading...
View Options

test/functional/test_runner.py

Loading...