Since knowledge of CAddrInfo is limited to callsites that import addrman_impl.h, only objects in addrman.cpp or the tests have access. Thus we can remove calling them friends and make the members public.
Partial backport of core#22950:
https://github.com/bitcoin/bitcoin/pull/22950/commits/7cf41bbb38db5008f9b69037b88138076d6a6cc5
Depends on D12331.