Page MenuHomePhabricator

[Chronik] Add `TxReader` and `TxWriter`
ClosedPublic

Authored by tobias_ruck on Mar 23 2023, 10:46.

Details

Reviewers
Fabien
Group Reviewers
Restricted Project
Commits
rABCa76e6da794e6: [Chronik] Add `TxReader` and `TxWriter`
Summary

Allows us to store txs from the node. This currently doesn't add txs coming from the node, just adds the structs/methods to do so.

Instead of having txids be the keys for the column family containing the txs, we use a 64-bit serial number "TxNum" that increments with every transaction in block order. This allows us to e.g. very easily iterate over all the txs in a block just by knowing the first tx_num of the block. It also simplifies the address index (especially reduces space requirements), as we simply store a list of relatively small integers instead of txids.

64-bits allows us to store a maximum of 18446744073709551616 txs, which even at 1M tx/s would be enough for +500000 years.

We only store the txid, data_pos, undo_pos and time_first_seen in the DB, everything else we can read from the block/undo files. We use the fact that coinbase txs don't have undo data, and undo data for txs never is at position 0, so we set undo_pos = 0 for coinbase txs, and treat every entry with undo_pos == 0 as a coinbase tx.

Just as we do for blocks, for the reverse index txid -> tx_num, we use ReverseLookup (which actually makes much more sense for txs). We use a 64-bit cheap hash to make collisions difficult. For the future, it might be worthwhile to salt/seed the cheap hash with some number generated when indexing for the first time, just to prevent someone spamming txs with lots of collisions.

Incrementing CURRENT_INDEXER_VERSION is a technicality, as we add the column families to the DB, but don't add data from the node (yet), so using it later could result in an inconsistent state.

Depends on D13458.

Test Plan

ninja check-crates

Diff Detail

Repository
rABC Bitcoin ABC
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Tail of the build log:

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests chronik-util

running 2 tests
test src/log.rs - log::log (line 11) ... ignored
test src/log.rs - log::log_chronik (line 33) ... ignored

test result: ok. 0 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 0.00s

[6/6] cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.67.1-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.67.1-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.67.1-x86_64-unknown-linux-gnu/bin/cargo --locked test --package abc-rust-*
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on build directory
   Compiling memchr v2.5.0
   Compiling libc v0.2.139
   Compiling abc-rust-lint v0.1.0 (/work/chronik/abc-rust-lint)
   Compiling object v0.29.0
   Compiling backtrace v0.3.66
   Compiling stable-eyre v0.2.2
   Compiling abc-rust-error v0.1.0 (/work/chronik/abc-rust-error)
    Finished test [unoptimized + debuginfo] target(s) in 2m 25s
     Running unittests src/lib.rs (abc-ci-builds/build-chronik/cargo/build/debug/deps/abc_rust_error-eec6582b4b0e9875)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/test_error.rs (abc-ci-builds/build-chronik/cargo/build/debug/deps/test_error-ad65b1b6998d5ffe)

running 1 test
test test_error ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/lib.rs (abc-ci-builds/build-chronik/cargo/build/debug/deps/abc_rust_lint-7244a5015e74dfbb)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests abc-rust-error

running 1 test
test src/http_status.rs - http_status::parse_error_status (line 10) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.19s

   Doc-tests abc-rust-lint

running 1 test
test src/lib.rs - lint (line 13) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.12s

ninja: build stopped: cannot make progress due to previous errors.
Build build-chronik failed with exit code 1

Factor TxId into its own diff

blk is singular so we use tx instead of txs as well

Add a nice preamble to txs.rs explaining what we're doing.

This revision is now accepted and ready to land.Mar 24 2023, 15:36

Make TxWriter::insert a bit more readable

This revision was automatically updated to reflect the committed changes.