This is a partial backport of Core [[https://github.com/bitcoin/bitcoin/pull/14477 | PR14477]] : https://github.com/bitcoin/bitcoin/pull/14477/commits/4d78bd93b5bdf886e743022e80f4edb8a982cf0d
I made InferPubkey static because it is only called from there.