This is to prepare for xonly_pubkeys and keypairs.
This is a partial backport of secp256k1 [[https://github.com/bitcoin-core/secp256k1/pull/558 | PR558]] : https://github.com/bitcoin-core/secp256k1/pull/558/commits/47e6618e11813cfabe91f0909ca031f960cb7dd4