Page MenuHomePhabricator

[Chronik] Optimize `prepare_indexed_txs` by using batch queries and removing redundancy
ClosedPublic

Authored by tobias_ruck on Feb 28 2024, 11:37.

Details

Summary

Currently, prepare_indexed_txs is doubly inefficient:

  1. If two inputs spend two outputs from the same tx, it'll query the tx num of this tx twice
  2. It queries tx nums one-by-one, instead of utilizing the optimized batch query gets that RocksDB provides

We fix this by adding batch query versions to LookupColumn and TxReader, and then utilize those in prepare_indexed_txs.

Note: We can't use sorted_inputs anywhere in this function when querying input_tx_nums, since those can come in a random order.

Benchmarks show there's a speedup of about 1.6 when syncing the first 300000ish blocks:

benchtotal time [s]time prepare [s]
no prepare1066.850
master4375.393308.54
this diff3148.512081.66
Test Plan

ninja check-crates && ninja check-functional

Diff Detail

Repository
rABC Bitcoin ABC
Branch
chronik-optimize-prepare_indexed_txs
Lint
Lint Passed
Unit
No Test Coverage
Build Status
Buildable 27506
Build 54575: Build Diffbuild-chronik-plugins · build-chronik · chronik-client-integration-tests
Build 54574: arc lint + arc unit

Event Timeline

remove wrong usage of sorted_inputs

Failed tests logs:

====== Get blocktxs, txs, and history for SLP NFT1 token txs: "before each" hook for "Gets an SLP NFT1 child genesis tx from the mempool".Get blocktxs, txs, and history for SLP NFT1 token txs "before each" hook for "Gets an SLP NFT1 child genesis tx from the mempool" ======
Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/work/modules/chronik-client/test/integration/token_slp_nft1.ts)
    at listOnTimeout (node:internal/timers:573:17)
    at processTimers (node:internal/timers:514:7)

Each failure log is accessible here:
Get blocktxs, txs, and history for SLP NFT1 token txs: "before each" hook for "Gets an SLP NFT1 child genesis tx from the mempool".Get blocktxs, txs, and history for SLP NFT1 token txs "before each" hook for "Gets an SLP NFT1 child genesis tx from the mempool"

tobias_ruck edited the summary of this revision. (Show Details)
tobias_ruck edited the summary of this revision. (Show Details)
This revision is now accepted and ready to land.Feb 29 2024, 10:47