[CI] Only build the required static dependencies

Summary:
This will shorten the first build time (before caching) and prevent the
need for a large timeout on cross builds.

Test Plan: Run any CI cross build.

Reviewers: #bitcoin_abc, jasonbcox

Reviewed By: #bitcoin_abc, jasonbcox

Differential Revision: https://reviews.bitcoinabc.org/D7231