Fix off by one errors in the difference fomatter serialize_tests

Summary:
One of the overflow test added in D11515 is not doing what is expected, due to a couple errors:

• the MAX_SIZE value is a compact size difference value, so the indexes should increment by MAX_SIZE+1
• the loop size if off by one

Test Plan:

ninja check-bitcoin-serialize_tests

Reviewers: #bitcoin_abc, sdulfari

Reviewed By: #bitcoin_abc, sdulfari

Subscribers: sdulfari

Differential Revision: https://reviews.bitcoinabc.org/D11567