Backport of core#22697.

Only these commits are interesting:
https://github.com/bitcoin/bitcoin/pull/22697/commits/181a1207ba6bd179d181f3e2534ef8676565ce72
https://github.com/bitcoin/bitcoin/pull/22697/commits/e8e7392311edf44278d76743bebe902d4ac94662

Removing the Clear() method would be interesting but it prevents use from being able to make the CConnman's CAddrMan deterministic and we need this for testing, so we skip that part.

Because we already backported PR22791 (see D12294), no ordering bug is introduced here.

Depends on D12300.