HomePhabricator

[Chronik] Add `BlockWriter`, storing block data in RocksDB using postcard

Description

[Chronik] Add BlockWriter, storing block data in RocksDB using postcard

Summary:
Add chronik-db:

  1. Add Db struct, handle on the RocksDB database.
  2. Add BlockWriter.

The block database only stores a handful of fields of a block (those that are returned by querying a range of blocks). The column family for block data is called "blk".

There is some redundancy to what the node provides (e.g. BlockIndex), however, keeping it separate, the ownership of the data is very clear, e.g. when querying through HTTP, and independent of the C++ code's invariants.

The encoding of the database is the postcard format, which stores especially integers compactly (see here for a spec: https://postcard.jamesmunns.com/wire-format.html). Serialization is done using serde.

Note: The experimental Chronik implementation uses zerocopy, such that stored data is represented the same as in memory. This turned out to be unnecessary in practice, and increased disk usage. There are a bunch of other breaks from the experimental implementation (e.g. renaming the column family to "blk" from "blocks"), but users should re-index anyway.

Note: The rocksdb dependency depends on librocksdb-sys, which currently creates around 2GB of artefacts and increases the binary size by ~55MB. For opt-in Chronik support, this is sufficient, but in the future when Chronik ships with bitcoind, we might want to switch to sled or another key-value store or bridge to bitcoind's existing LevelDB dependency.

Depends on D11774.

Test Plan: ninja check-chronik

Reviewers: Fabien, #bitcoin_abc

Reviewed By: Fabien, #bitcoin_abc

Differential Revision: https://reviews.bitcoinabc.org/D11728

Details

Provenance
tobias_ruckAuthored on Jul 7 2022, 12:46
tobias_ruckPushed on Jul 22 2022, 17:08
Reviewer
Restricted Project
Differential Revision
D11728: [Chronik] Add `BlockWriter`, storing block data in RocksDB using postcard
Parents
rABC6c2d19642640: [Chronik] Add `abc-rust-error` crate for convenient error handling
Branches
Unknown
Tags
Unknown