Check error code return by CheckPubKeyEncoding in tests

Summary:
This ensure that any change of behavior related to these error code will be detected.

Depends on D1566

Test Plan: make check

Reviewers: #bitcoin_abc, jasonbcox

Reviewed By: #bitcoin_abc, jasonbcox

Subscribers: teamcity

Differential Revision: https://reviews.bitcoinabc.org/D1568