HomePhabricator

[Chronik] Use `merge_cf` in `reverse_lookup.rs` to speed up indexing.

Description

[Chronik] Use merge_cf in reverse_lookup.rs to speed up indexing.

Summary:
TxWriter is surprizingly slow considering what it does is very simple, however, the bottleneck is LookupIndex, which in insert_pairs reads from the DB before updating, which is very slow.

Instead, use merge_cf like in D14430, which avoids that roundrip, and RocksDB is able to combine a read+write efficiently for us.

To resync the first 300000-ish blocks, with all other indices disabled (i.e. only TxWriter), we get an overall speedup of almost 2x. Note that this still includes reading the blocks from disk etc., which seems to have an overhead of 821.10s (see D15552, where one benchmark strips basically everything away), so factoring this out we get a speedup of roughly 4x.

The insert call itself is now extremely fast, because it basically just issues a bunch of merge_cf calls, and the real action is handled by RocksDB, so benchmarking the call isn't too useful anymore.

benchtotal time [s]time TxWriter::insert [s]
master2378.861134.73
merge_cf1212.0836.9

Depends on D15547

Test Plan: ninja check-crates && ninja check-functional

Reviewers: Fabien, #bitcoin_abc

Reviewed By: Fabien, #bitcoin_abc

Differential Revision: https://reviews.bitcoinabc.org/D15548

Details

Provenance
tobias_ruckAuthored on Feb 26 2024, 10:44
tobias_ruckPushed on Feb 26 2024, 15:22
Reviewer
Restricted Project
Differential Revision
D15548: [Chronik] Use `merge_cf` in `reverse_lookup.rs` to speed up indexing.
Parents
rABCcfbddd174fec: [Chronik] `ReverseLookup`: Remove redundant `lookup_cf_name`
Branches
Unknown
Tags
Unknown