The shift operators will call the write or read member function, which
already does the check. Also, call sites are free to directly call
::(Un)Serialize(s, obj) to skip this check, so removing it increases
consistency.
This is a partial backport of core#28060
https://github.com/bitcoin/bitcoin/pull/28060/commits/fafe2ca0ce842cd8f0d8135fa8c8bac9b2c72da6