refactor: Make MessageBoxFlags enum underlying type unsigned

Summary:
This is a backport of core#24191

Depends on D12816

Test Plan:
With UBSAN
ninja && ninja check check-functional

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Differential Revision: https://reviews.bitcoinabc.org/D12817