Page MenuHomePhabricator

[Chronik] Use `merge_cf` in `reverse_lookup.rs` to speed up indexing.
ClosedPublic

Authored by tobias_ruck on Feb 26 2024, 10:45.

Details

Summary

TxWriter is surprizingly slow considering what it does is very simple, however, the bottleneck is LookupIndex, which in insert_pairs reads from the DB before updating, which is very slow.

Instead, use merge_cf like in D14430, which avoids that roundrip, and RocksDB is able to combine a read+write efficiently for us.

To resync the first 300000-ish blocks, with all other indices disabled (i.e. only TxWriter), we get an overall speedup of almost 2x. Note that this still includes reading the blocks from disk etc., which seems to have an overhead of 821.10s (see D15552, where one benchmark strips basically everything away), so factoring this out we get a speedup of roughly 4x.

The insert call itself is now extremely fast, because it basically just issues a bunch of merge_cf calls, and the real action is handled by RocksDB, so benchmarking the call isn't too useful anymore.

benchtotal time [s]time TxWriter::insert [s]
master2378.861134.73
merge_cf1212.0836.9

Depends on D15547

Test Plan

ninja check-crates && ninja check-functional

Diff Detail

Repository
rABC Bitcoin ABC
Branch
chronik-reverse_lookup-merge
Lint
Lint Passed
Unit
No Test Coverage
Build Status
Buildable 27413
Build 54389: Build Diffbuild-chronik-plugins · chronik-client-integration-tests · build-chronik
Build 54388: arc lint + arc unit

Event Timeline

The build failed due to an unexpected infrastructure outage. The administrators have been notified to investigate. Sorry for the inconvenience.

Tail of the build log:

  Downloaded stable-eyre v0.2.2
  Downloaded memchr v2.5.0
  Downloaded gimli v0.27.3
  Downloaded object v0.31.1
  Downloaded libc v0.2.147
  Downloaded http v1.0.0
  Downloaded backtrace v0.3.68
  Downloaded miniz_oxide v0.7.1
  Downloaded eyre v0.6.8
  Downloaded once_cell v1.18.0
  Downloaded cc v1.0.83
  Downloaded rustc-demangle v0.1.23
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on build directory
   Compiling bytes v1.5.0
   Compiling libc v0.2.147
   Compiling abc-rust-lint v0.1.0 (/work/chronik/abc-rust-lint)
   Compiling memchr v2.5.0
   Compiling object v0.31.1
   Compiling http v1.0.0
   Compiling cc v1.0.83
   Compiling backtrace v0.3.68
   Compiling stable-eyre v0.2.2
   Compiling abc-rust-error v0.1.0 (/work/chronik/abc-rust-error)
    Finished test [unoptimized + debuginfo] target(s) in 4m 30s
     Running unittests src/lib.rs (abc-ci-builds/build-chronik-plugins/cargo/build/debug/deps/abc_rust_error-fbb1aab080abd8f8)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/test_error.rs (abc-ci-builds/build-chronik-plugins/cargo/build/debug/deps/test_error-5ef5f197f15f6364)

running 1 test
test test_error ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/lib.rs (abc-ci-builds/build-chronik-plugins/cargo/build/debug/deps/abc_rust_lint-dba48c2d41985cd4)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests abc-rust-error

running 1 test
test chronik/abc-rust-error/src/http_status.rs - http_status::parse_error_status (line 10) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.19s

   Doc-tests abc-rust-lint

running 1 test
test chronik/abc-rust-lint/src/lib.rs - lint (line 13) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.12s

ninja: build stopped: cannot make progress due to previous errors.
Build build-chronik-plugins failed with exit code 1

actually use the options created

The build failed due to an unexpected infrastructure outage. The administrators have been notified to investigate. Sorry for the inconvenience.
This revision is now accepted and ready to land.Feb 26 2024, 14:33