[interpreter] Return INTEGER_OVERFLOW or BAD_INTEGER_ENCODING instead of UNKNOWN
Summary:
Currently, the Script interpreter returns UNKNOWN when it encountered an integer overflow or a non-minimally encoded integer.
This has been a nuisance for new Script devs for a while, because "unknown error" sounds like the interpreter is broken, but it's actually expected behavior.
Also, this makes script number tests more brittle.
Instead, we return INTEGER_OVERFLOW if there's been any kind of integer overflow, e.g. when parsing operands, or when executing an operation like OP_ADD, OP_BIN2NUM, etc.
Test Plan: ninja check && ninja check-functional
Reviewers: Fabien, #bitcoin_abc
Reviewed By: Fabien, #bitcoin_abc
Differential Revision: https://reviews.bitcoinabc.org/D18481