Add test case for boundary conditions for div and mod.

Summary: As per title.

Test Plan:

make check

Reviewers: #bitcoin_abc, schancel

Reviewed By: #bitcoin_abc, schancel

Subscribers: teamcity

Differential Revision: https://reviews.bitcoinabc.org/D1467