Add test for CheckPubKeyEncoding

Summary:
As per title. More test is better.

Depends on D1565

Test Plan:

make check

Reviewers: #bitcoin_abc, jasonbcox

Reviewed By: #bitcoin_abc, jasonbcox

Subscribers: jasonbcox, teamcity

Differential Revision: https://reviews.bitcoinabc.org/D1566