Page MenuHomePhabricator

[electrum] Performance: Speed up loading of very large wallets
ClosedPublic

Authored by PiRK on Nov 14 2023, 17:04.

Details

Summary

Use sets instead of lists to test membership. Don't copy all the wallet.transaction keys, instead just iterate over them, enqueueing deletions.

Abstract_Wallet.remove_transaction was doing some unnecessary copying and slow ops. Optimized a bit.

Backport of https://github.com/Electron-Cash/Electron-Cash/commit/115f0f20dec1f5429d8b0e70b41056bf3464781a and https://github.com/Electron-Cash/Electron-Cash/commit/2beea5e3e61d487b3e60c1259a8ac8cbf07316ad

Before this commit, with only the @profiler changes, when loading a very large wallet:

$ ./electrum-abc -v
|  2.753| |00| [profiler] AbstractWallet.load_transactions 1.3602
| 22.364| |00| [profiler] AbstractWallet.prepare_for_verifier 19.5695
| 22.378| |00| [profiler] Synchronizer._initialize 0.0141
| 22.406| |04| [SPV/ifp_addr] started
| 22.407| |04| [Synchronizer/ifp_addr] started
| 22.468| |06| [TaskThread/ifp_addr/Wallet] started
| 23.864| |04| [WalletStorage] saved /home/pierre/.electrum-abc/wallets/ifp_addr
| 23.871| |04| [profiler] WalletStorage.write 0.7861
| 24.834| |04| [profiler] AbstractWallet.save_transactions 0.9629
| 25.556| |04| [WalletStorage] saved /home/pierre/.electrum-abc/wallets/ifp_addr
| 25.562| |04| [profiler] WalletStorage.write 0.5759
| 26.034| |00| [profiler] AbstractWallet.get_history 2.4158

After:

$ ./electrum-abc -v
|  2.627| |00| [profiler] AbstractWallet.load_transactions 1.3024
|  2.705| |00| [profiler] AbstractWallet.prepare_for_verifier 0.0372
|  2.716| |00| [profiler] Synchronizer._initialize 0.0112
|  2.786| |05| [SPV/ifp_addr] started
|  2.786| |05| [Synchronizer/ifp_addr] started
|  2.795| |04| [TaskThread/ifp_addr/Wallet] started
|  3.657| |05| [WalletStorage] saved /home/pierre/.electrum-abc/wallets/ifp_addr
|  3.664| |05| [profiler] WalletStorage.write 0.7289
|  4.729| |05| [profiler] AbstractWallet.save_transactions 1.0653
|  5.444| |05| [WalletStorage] saved /home/pierre/.electrum-abc/wallets/ifp_addr
|  5.451| |05| [profiler] WalletStorage.write 0.5702
|  5.798| |00| [profiler] AbstractWallet.get_history 1.8234
Test Plan
python test_runner.py
pytest electrumabc/tests/regtest

Diff Detail

Repository
rABC Bitcoin ABC
Branch
reorg
Lint
Lint Passed
Unit
No Test Coverage
Build Status
Buildable 25616
Build 50813: Build Diffelectrum-tests
Build 50812: arc lint + arc unit

Event Timeline

PiRK requested review of this revision.Nov 14 2023, 17:04
PiRK retitled this revision from Performance: Speed up loading of very large wallets to [electrum] Performance: Speed up loading of very large wallets.
electrum/electrumabc/wallet.py
1544

the indices are in increasing order, so we remove from the beginning of the list and need to correct following indices accordingly.

This revision is now accepted and ready to land.Nov 15 2023, 00:01