refactor: Use ignore helper when unserializing an invalid pubkey

Summary:
This is a partial backport of core#23451
https://github.com/bitcoin/bitcoin/pull/23451/commits/fa18038f519db76befb9a7bd0b1540143bfeb12b

Depends on D15857

Test Plan: ninja all check-all

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Differential Revision: https://reviews.bitcoinabc.org/D15858