Minor refactoring to remove implied m_addr_relay_peer.

Summary:
Co-authored-by: MarcoFalke <falke.marco@gmail.com>

This is a backport of Core PR17164 [3/3]
https://github.com/bitcoin/bitcoin/pull/17164/commits/b6d2183858975abc961207c125c15791e531edcc
Depends on D8239

Test Plan: ninja all check-all

Reviewers: O1 Bitcoin ABC, #bitcoin_abc, deadalnix

Reviewed By: O1 Bitcoin ABC, #bitcoin_abc, deadalnix

Differential Revision: https://reviews.bitcoinabc.org/D8240