Backport of core#22697.
Only this commit is interesting:
https://github.com/bitcoin/bitcoin/pull/22697/commits/181a1207ba6bd179d181f3e2534ef8676565ce72
Removing the Clear() method would be interesting but it prevents use from being able to make the CConnman's CAddrMan deterministic and we need this for testing, so we skip that part.
Because we already backported PR22791 (see D12294), no ordering bug is introduced here.
Depends on D12300.