Seems odd to reserve memory for the struct (the heaviest member being m_tx_inventory_known_filter) when it is never used.
Note that we cannot add the same functional test as Core, because we didn't start backporting the Erlay feature (which is still far from being ready, see core#28646)
This is a backport of core#26396
Depends on D16796