Add tests for HexStr corner cases

Summary: Backport of core PR12570

Test Plan:

make check

Reviewers: #bitcoin_abc, deadalnix, jasonbcox

Reviewed By: #bitcoin_abc, jasonbcox

Subscribers: teamcity, schancel

Differential Revision: https://reviews.bitcoinabc.org/D2426