[electrum] Performance: Speed up loading of very large wallets
Summary:
Use sets instead of lists to test membership. Don't copy all the wallet.transaction keys, instead just iterate over them, enqueueing deletions.
Abstract_Wallet.remove_transaction was doing some unnecessary copying and slow ops. Optimized a bit.
Backport of https://github.com/Electron-Cash/Electron-Cash/commit/115f0f20dec1f5429d8b0e70b41056bf3464781a and https://github.com/Electron-Cash/Electron-Cash/commit/2beea5e3e61d487b3e60c1259a8ac8cbf07316ad
Before this commit, with only the @profiler changes, when loading a very large wallet:
$ ./electrum-abc -v | 2.753| |00| [profiler] AbstractWallet.load_transactions 1.3602 | 22.364| |00| [profiler] AbstractWallet.prepare_for_verifier 19.5695 | 22.378| |00| [profiler] Synchronizer._initialize 0.0141 | 22.406| |04| [SPV/ifp_addr] started | 22.407| |04| [Synchronizer/ifp_addr] started | 22.468| |06| [TaskThread/ifp_addr/Wallet] started | 23.864| |04| [WalletStorage] saved /home/pierre/.electrum-abc/wallets/ifp_addr | 23.871| |04| [profiler] WalletStorage.write 0.7861 | 24.834| |04| [profiler] AbstractWallet.save_transactions 0.9629 | 25.556| |04| [WalletStorage] saved /home/pierre/.electrum-abc/wallets/ifp_addr | 25.562| |04| [profiler] WalletStorage.write 0.5759 | 26.034| |00| [profiler] AbstractWallet.get_history 2.4158
After:
$ ./electrum-abc -v | 2.627| |00| [profiler] AbstractWallet.load_transactions 1.3024 | 2.705| |00| [profiler] AbstractWallet.prepare_for_verifier 0.0372 | 2.716| |00| [profiler] Synchronizer._initialize 0.0112 | 2.786| |05| [SPV/ifp_addr] started | 2.786| |05| [Synchronizer/ifp_addr] started | 2.795| |04| [TaskThread/ifp_addr/Wallet] started | 3.657| |05| [WalletStorage] saved /home/pierre/.electrum-abc/wallets/ifp_addr | 3.664| |05| [profiler] WalletStorage.write 0.7289 | 4.729| |05| [profiler] AbstractWallet.save_transactions 1.0653 | 5.444| |05| [WalletStorage] saved /home/pierre/.electrum-abc/wallets/ifp_addr | 5.451| |05| [profiler] WalletStorage.write 0.5702 | 5.798| |00| [profiler] AbstractWallet.get_history 1.8234
Test Plan:
python test_runner.py pytest electrumabc/tests/regtest
Reviewers: #bitcoin_abc, bytesofman
Reviewed By: #bitcoin_abc, bytesofman
Differential Revision: https://reviews.bitcoinabc.org/D14785