test: Add format-dependent comparison to bctest

This splits the output comparison for bitcoin-tx into two steps:

• First, check for data mismatch, parsing the data as json or hex depending on the extension of the output file

• Then, check if the literal string matches

For either of these cases give a different error.

This prevents wild goose chases when e.g. a trailing space doesn't match
exactly, and makes sure that both test output and examples are valid
data of the purported format.