[net_processing] Keep addrman reference in PeerManager
Summary:
Partial backport of core#20228:
https://github.com/bitcoin/bitcoin/pull/20228/commits/392a95d393a9af01b53e5e68197e81968efb84fc
Depends on D12246.
Test Plan:
ninja all check-all
Reviewers: #bitcoin_abc, PiRK
Reviewed By: #bitcoin_abc, PiRK
Differential Revision: https://reviews.bitcoinabc.org/D12247