Address nits in ADDR caching

Summary:
This is a backport of core#19697 [3/4]
https://github.com/bitcoin/bitcoin/pull/19697/commits/83ad65f31b5c9441ae1618614082e584854a14e1

Depends on D10291

Test Plan: ninja all check-all

Reviewers: #bitcoin_abc, Fabien, baby636

Reviewed By: #bitcoin_abc, Fabien, baby636

Subscribers: baby636, Fabien

Differential Revision: https://reviews.bitcoinabc.org/D10292