[avalanche] Remove blocks not worth pollling from the vote reccords rather than just ignore them

Summary: This require to make getInvsForNextPoll non const, but overall, the tradeoff is worth it.

Test Plan:

ninja all check-all

Reviewers: #bitcoin_abc, jasonbcox

Reviewed By: #bitcoin_abc, jasonbcox

Differential Revision: https://reviews.bitcoinabc.org/D6658