Assert should only be used for program internal logic errors, not to sanitize external user input. The assert was introduced via the debug-only runtime option -checkaddrman in commit 803ef70, thus won't need a backport. Also, it doesn't really make sense to continue when the deserialized addrman doesn't pass the sanity check. For example, if nLastSuccess is negative, it would later result in integer overflows.
Backport of core#22734.
Depends on D12310.