[electrum] Performance: Speed up loading of very large wallets
ClosedPublic
Actions

Authored by PiRK on Nov 14 2023, 17:04.

Details

Reviewers

bytesofman

Group Reviewers

Restricted Project

Commits

rABC0f9882cff056: [electrum] Performance: Speed up loading of very large wallets

Summary

Use sets instead of lists to test membership. Don't copy all the wallet.transaction keys, instead just iterate over them, enqueueing deletions.

Abstract_Wallet.remove_transaction was doing some unnecessary copying and slow ops. Optimized a bit.

Backport of https://github.com/Electron-Cash/Electron-Cash/commit/115f0f20dec1f5429d8b0e70b41056bf3464781a and https://github.com/Electron-Cash/Electron-Cash/commit/2beea5e3e61d487b3e60c1259a8ac8cbf07316ad

Before this commit, with only the @profiler changes, when loading a very large wallet:

$ ./electrum-abc -v
|  2.753| |00| [profiler] AbstractWallet.load_transactions 1.3602
| 22.364| |00| [profiler] AbstractWallet.prepare_for_verifier 19.5695
| 22.378| |00| [profiler] Synchronizer._initialize 0.0141
| 22.406| |04| [SPV/ifp_addr] started
| 22.407| |04| [Synchronizer/ifp_addr] started
| 22.468| |06| [TaskThread/ifp_addr/Wallet] started
| 23.864| |04| [WalletStorage] saved /home/pierre/.electrum-abc/wallets/ifp_addr
| 23.871| |04| [profiler] WalletStorage.write 0.7861
| 24.834| |04| [profiler] AbstractWallet.save_transactions 0.9629
| 25.556| |04| [WalletStorage] saved /home/pierre/.electrum-abc/wallets/ifp_addr
| 25.562| |04| [profiler] WalletStorage.write 0.5759
| 26.034| |00| [profiler] AbstractWallet.get_history 2.4158

After:

$ ./electrum-abc -v
|  2.627| |00| [profiler] AbstractWallet.load_transactions 1.3024
|  2.705| |00| [profiler] AbstractWallet.prepare_for_verifier 0.0372
|  2.716| |00| [profiler] Synchronizer._initialize 0.0112
|  2.786| |05| [SPV/ifp_addr] started
|  2.786| |05| [Synchronizer/ifp_addr] started
|  2.795| |04| [TaskThread/ifp_addr/Wallet] started
|  3.657| |05| [WalletStorage] saved /home/pierre/.electrum-abc/wallets/ifp_addr
|  3.664| |05| [profiler] WalletStorage.write 0.7289
|  4.729| |05| [profiler] AbstractWallet.save_transactions 1.0653
|  5.444| |05| [WalletStorage] saved /home/pierre/.electrum-abc/wallets/ifp_addr
|  5.451| |05| [profiler] WalletStorage.write 0.5702
|  5.798| |00| [profiler] AbstractWallet.get_history 1.8234

Test Plan

python test_runner.py
pytest electrumabc/tests/regtest

Diff Detail

Repository

rABC Bitcoin ABC

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

PiRK created this revision.Nov 14 2023, 17:04

Herald added a reviewer: Restricted Project. · View Herald TranscriptNov 14 2023, 17:04

PiRK requested review of this revision.Nov 14 2023, 17:04

PiRK edited the summary of this revision. (Show Details)Nov 14 2023, 17:05

PiRK retitled this revision from Performance: Speed up loading of very large wallets to [electrum] Performance: Speed up loading of very large wallets.

Harbormaster completed remote builds in B25616: Diff 43081.Nov 14 2023, 17:06

PiRK added inline comments.Nov 14 2023, 17:09

electrum/electrumabc/wallet.py
1544 ↗	(On Diff #43081)	the indices are in increasing order, so we remove from the beginning of the list and need to correct following indices accordingly.