Change from overflow to testing integer minimum (eliminate compiler warning)

Summary:
This test case was badly written, overflowing integer only causes compiler
warning instead of being useful.
Changed to test integer minimum instead.

Test Plan: make check

Reviewers: #bitcoin_abc, deadalnix, sickpig, kyuupichan, awemany

Reviewed By: #bitcoin_abc, deadalnix

Differential Revision: https://reviews.bitcoinabc.org/D259