Page MenuHomePhabricator

[Chronik] Add `BlockWriter`, storing block data in RocksDB using postcard
ClosedPublic

Authored by tobias_ruck on Jul 7 2022, 12:48.

Details

Summary

Add chronik-db:

  1. Add Db struct, handle on the RocksDB database.
  2. Add BlockWriter.

The block database only stores a handful of fields of a block (those that are returned by querying a range of blocks). The column family for block data is called "blk".

There is some redundancy to what the node provides (e.g. BlockIndex), however, keeping it separate, the ownership of the data is very clear, e.g. when querying through HTTP, and independent of the C++ code's invariants.

The encoding of the database is the postcard format, which stores especially integers compactly (see here for a spec: https://postcard.jamesmunns.com/wire-format.html). Serialization is done using serde.

Note: The experimental Chronik implementation uses zerocopy, such that stored data is represented the same as in memory. This turned out to be unnecessary in practice, and increased disk usage. There are a bunch of other breaks from the experimental implementation (e.g. renaming the column family to "blk" from "blocks"), but users should re-index anyway.

Note: The rocksdb dependency depends on librocksdb-sys, which currently creates around 2GB of artefacts and increases the binary size by ~55MB. For opt-in Chronik support, this is sufficient, but in the future when Chronik ships with bitcoind, we might want to switch to sled or another key-value store or bridge to bitcoind's existing LevelDB dependency.

Depends on D11774.

Test Plan

ninja check-chronik

Diff Detail

Repository
rABC Bitcoin ABC
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Fabien requested changes to this revision.Jul 8 2022, 09:59

Please split this in 2 diffs, one for bitcoinsuite-error and one for the db

chronik/bitcoinsuite-error/src/lib.rs
30 ↗(On Diff #34295)

Can this be imported from some common file to avoid duplicating it in every lib.rs ?

This revision now requires changes to proceed.Jul 8 2022, 09:59
tobias_ruck edited the summary of this revision. (Show Details)

Separate bitcoinsuite-error into its own diff (D11774)

Tail of the build log:

-- Found Event: /usr/include (found suitable version "2.1.8-stable", minimum required is "2.0.22") found components: event 
-- Found Boost: /usr/include (found suitable version "1.67.0", minimum required is "1.59") found components: filesystem system 
-- Found Boost: /usr/include (found suitable version "1.67.0", minimum required is "1.59") found components: thread chrono system date_time atomic 
-- Found Event component pthreads: /usr/lib/x86_64-linux-gnu/libevent_pthreads.so
-- Found Event: /usr/include (found suitable version "2.1.8-stable", minimum required is "2.0.22") found components: pthreads 
-- Found MiniUPnPc component miniupnpc: /usr/lib/x86_64-linux-gnu/libminiupnpc.so
-- Found MiniUPnPc: /usr/include/miniupnpc (found suitable version "2.1", minimum required is "1.9")  
-- Found Boost: /usr/include (found suitable version "1.67.0", minimum required is "1.59") found components: unit_test_framework 
-- Performing Test BOOST_REQUIRES_TEST_DYN_LINK
-- Performing Test BOOST_REQUIRES_TEST_DYN_LINK - Success
-- Found BerkeleyDB component CXX: /usr/lib/x86_64-linux-gnu/libdb_cxx.so
-- Found BerkeleyDB: /usr/include (found suitable version "5.3.28", minimum required is "5.3") found components: CXX 
-- Found SQLite3: /usr/include (found suitable version "3.27.2", minimum required is "3.7.17") 
-- Found ZeroMQ component zmq: /usr/lib/x86_64-linux-gnu/libzmq.so
-- Found ZeroMQ: /usr/include (found suitable version "4.3.1", minimum required is "4.1.5")  
-- Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.so;-pthread (found version "3.6.1") 
-- Found OpenSSL: /usr/lib/x86_64-linux-gnu/libcrypto.so (found version "1.1.1n")  
-- Looking for EVP_MD_CTX_new
-- Looking for EVP_MD_CTX_new - found
-- Found QREncode component qrencode: /usr/lib/x86_64-linux-gnu/libqrencode.so
-- Found QREncode: /usr/include   
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.61.0 (fe5b13d68 2022-05-18)`
-- Rust Toolchain: 1.61.0-x86_64-unknown-linux-gnu
-- Rust Target: x86_64-unknown-linux-gnu
-- Found Rust: /root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/rustc (found version "1.61.0") 
-- Configuring native build in /work/abc-ci-builds/build-chronik/native
-- Configuring done
-- Generating done
-- Build files have been written to: /work/abc-ci-builds/build-chronik
[1/4] cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik/chronik/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/cargo test --package chronik-*
FAILED: chronik/CMakeFiles/check-chronik-test 
cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik/chronik/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/cargo test --package chronik-*
error: failed to parse lock file at: /work/Cargo.lock

Caused by:
  package `bitfield` is specified twice in the lockfile
[2/4] cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik/chronik/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/cargo test --package bitcoinsuite-*
FAILED: chronik/CMakeFiles/check-bitcoinsuite-test 
cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik/chronik/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/cargo test --package bitcoinsuite-*
error: failed to parse lock file at: /work/Cargo.lock

Caused by:
  package `bitfield` is specified twice in the lockfile
[3/4] cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik/chronik/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/cargo clippy --package bitcoinsuite-* -- -D warnings
FAILED: chronik/CMakeFiles/check-bitcoinsuite-clippy 
cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik/chronik/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/cargo clippy --package bitcoinsuite-* -- -D warnings
error: failed to parse lock file at: /work/Cargo.lock

Caused by:
  package `bitfield` is specified twice in the lockfile
[4/4] cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik/chronik/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/cargo clippy --package chronik-* -- -D warnings
FAILED: chronik/CMakeFiles/check-chronik-clippy 
cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik/chronik/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.61.0-x86_64-unknown-linux-gnu/bin/cargo clippy --package chronik-* -- -D warnings
error: failed to parse lock file at: /work/Cargo.lock

Caused by:
  package `bitfield` is specified twice in the lockfile
ninja: build stopped: cannot make progress due to previous errors.
Build build-chronik failed with exit code 1

Cargo.lock broken after wrong merge

Fabien requested changes to this revision.Jul 20 2022, 15:32
Fabien added inline comments.
Cargo.toml
7 ↗(On Diff #34436)

You need to rebase this diff on top of D11774 before you run arc diff, so that only the changes from this diff will appear

This revision now requires changes to proceed.Jul 20 2022, 15:32
This revision is now accepted and ready to land.Jul 21 2022, 20:33

Update diff to new abc-rust- naming prefix