From the PR discussion:
> Fair enough as a temporary workaround.
>
> The source of this bug is that not all time in net is using mocked time, giving those "rounding errors".
>
> The long-term fix should be to use mocked time consistently.
This is a backport of [[https://github.com/bitcoin/bitcoin/pull/23740 | core#23740]]