Fix spurious addrman test failure

Summary:
Depends on D1955
Backport of core PR 10429

Test Plan:

./src/test/test_bitcoin -t addrman_tests

Reviewers: #bitcoin_abc, deadalnix

Reviewed By: #bitcoin_abc, deadalnix

Subscribers: teamcity, deadalnix

Differential Revision: https://reviews.bitcoinabc.org/D1954