This reduces bloat, complexity, and makes tests less fragile to intermittent failures.
This is a backport of [[https://github.com/bitcoin/bitcoin/pull/28409 | core#28409]]
Note: I don't know if this already fixes any existing intermittent failure, but at least it will prevent introducing a new one when backporting [[https://github.com/bitcoin/bitcoin/pull/27675 | core#27675]]