add new encoding checker for Schnorr sigs

Summary:
Part of effort to add new CHECKMULTISIG mode.

Depends on D3417

Test Plan: make check

Reviewers: deadalnix, Mengerian, #bitcoin_abc

Reviewed By: deadalnix, #bitcoin_abc

Maniphest Tasks: T528

Differential Revision: https://reviews.bitcoinabc.org/D3265