HomePhabricator

[Chronik] Optimize `GroupHistoryWriter` using batching and RocksDB's merge…

Description

[Chronik] Optimize GroupHistoryWriter using batching and RocksDB's merge operator

Summary:
Employ the following strategies:

  1. Add a new column family, script_history_num_txs, which tracks the number of txs for each script. This tells us directly which page we need to append new txs to and how many without looking up anything else.
  2. Use a "merge operator" (available in RocksDB, see https://github.com/facebook/rocksdb/wiki/Merge-Operator), which simply concats bytes to pages (for insert) or trims them (for delete).
  3. Store pages not as serialized Vec<TxNum>, but as concatenated byte stream of serialized TxNum. This allows us to simply byte-append serialized tx nums, which simplifies the merge operator.
  4. Use multi_get when querying the num of existing txs for scripts, which allows RocksDB to fetch data more efficiently than when querying one-by-one.

Data that has been gathered + methodology for the optimization can be found here:
https://marble-timimus-187.notion.site/Bitcoin-ABC-Chronik-082e0ebbb02343198c79d2c58d2e9b45?pvs=4

Test Plan: ninja check-crates && ninja check-functional

Reviewers: Fabien, #bitcoin_abc

Reviewed By: Fabien, #bitcoin_abc

Differential Revision: https://reviews.bitcoinabc.org/D14393

Details

Provenance
tobias_ruckAuthored on Aug 22 2023, 15:55
tobias_ruckPushed on Aug 24 2023, 12:15
Reviewer
Restricted Project
Differential Revision
D14393: [Chronik] Optimize `GroupHistoryWriter` using batching and RocksDB's merge operator
Parents
rABC257c857d0321: refactor: remove c-string interfaces for DecodeBase58{Check}
Branches
Unknown
Tags
Unknown