Paths

Table of Contentst

Differential D4852

[DEPENDS] Use parallel compilation when building packages
ClosedPublic
Actions

Authored by Fabien on Jan 8 2020, 09:01.

Details

Reviewers

deadalnix

Group Reviewers

Restricted Project

Commits

rSTAGINGa6f39494ac51: [DEPENDS] Use parallel compilation when building packages
rABCa6f39494ac51: [DEPENDS] Use parallel compilation when building packages

Summary

This diffs try to guess the number of cores availble on the user machine
and forward it (+1) to the -jX option to make when building the
packages.
This JOBS variable can be overriden by the user and fallback to 1 if
it can't guess the number of cores.

For now only ccache makes use of this feature. This is built on all the platforms and is simple enough for demonstrating the feature.

Test Plan

cd depends

Edit packages/native_ccache.mk, add the line:

echo "BUILDING WITH -j$(JOBS)" && \

before the line:

$(MAKE) -j$(JOBS)

make V=1

Check in the output that make is invoked with the appropriated -jX
option(for the ccache package).

make V=1 JOBS=2

Check in the output that make is invoked with the appropriated -j2
option (for the ccache package).

make V=1 build-all

Check in the output that the JOBS is propagated to the ccache package of all the targets.

Run the Gitian builds and check the output is still deterministic (clear
the cached packages before building).

Diff Detail

Repository

rABC Bitcoin ABC

Branch

depends_parallel

Lint

Lint Passed

Unit

No Test Coverage

Build Status

Buildable 8780
Build 15541: Default Diff Build & Tests
Build 15540: arc lint + arc unit

Event Timeline

Fabien created this revision.Jan 8 2020, 09:01

Herald added a reviewer: Restricted Project. · View Herald TranscriptJan 8 2020, 09:01

Herald added a reviewer: deadalnix. · View Herald Transcript

Build Bitcoin-ABC / Diffs / Diff Testing started.

Build Bitcoin-ABC / Diffs / Diff Testing passed.

Harbormaster completed remote builds in B8779: Diff 15217.Jan 8 2020, 09:06

Can you make sure that JOB is respected properly when using build-all ?

depends/Makefile
20 ↗	(On Diff #15217)	Make it zero at the end and always add 1 so we get n+1 thread, which is usually considered what's good.

I'd make sure to get the mechanism working using one of the package before extending it to all packages. This is hard to review and any change to one package is going to block everything.

depends/packages/boost.mk
36 ↗	(On Diff #15217)	Why is it removed?
40 ↗	(On Diff #15217)	Why -j4 ?

This revision now requires changes to proceed.Jan 8 2020, 13:20

OK, I will make it on a single package to begin with.

depends/packages/boost.mk
36 ↗	(On Diff #15217)	`b2` builds with all the cores by default: https://boostorg.github.io/build/manual/develop/index.html#bbv2.overview.invocation.options
40 ↗	(On Diff #15217)	I have no idea where this limit comes from. I did not change the install commands since there is not much to gain here (I tested, the difference is not significant).

Fabien planned changes to this revision.Jan 8 2020, 13:31

Fabien edited the summary of this revision. (Show Details)Jan 8 2020, 13:47

Fabien edited the test plan for this revision. (Show Details)

Address feedback:

Use N+1 jobs, where N is the number of cores
Only use JOBS in a single package (ccache)
Add a build-all check to the test plan

Build Bitcoin-ABC / Diffs / Diff Testing started.

Note to reviewers:
The build-all check from the test plan is still running on my machine (so far so good).
I will put a notice in a later comment when it ends.

Build Bitcoin-ABC / Diffs / Diff Testing passed.

Harbormaster completed remote builds in B8780: Diff 15219.Jan 8 2020, 13:56

In D4852#117199, @Fabien wrote:

Note to reviewers:
The build-all check from the test plan is still running on my machine (so far so good).
I will put a notice in a later comment when it ends.

You don't need check for the build to terminate, you need to check for the right value of JOBS to be passed down.

deadalnix accepted this revision.Jan 8 2020, 13:57

This revision is now accepted and ready to land.Jan 8 2020, 13:57

Closed by commit rABCa6f39494ac51: [DEPENDS] Use parallel compilation when building packages (authored by Fabien). · Explain WhyJan 8 2020, 15:21

This revision was automatically updated to reflect the committed changes.

Fabien added a commit: rABCa6f39494ac51: [DEPENDS] Use parallel compilation when building packages.

Fabien mentioned this in D4867: [DEPENDS] Make the qt package build parallel.Jan 9 2020, 08:47

Fabien added a commit: rSTAGINGa6f39494ac51: [DEPENDS] Use parallel compilation when building packages.Apr 7 2020, 22:19