[psbt] add file size limit

Summary: Backport of Core PR17509 part [5/6] : https://github.com/bitcoin/bitcoin/pull/17509/commits/764bfe4cba35c24f7627cc425d9e7eba56e98964

Test Plan:

ninja all check

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Subscribers: Fabien

Differential Revision: https://reviews.bitcoinabc.org/D8070