Employ the following strategies:
- Add a new column family, script_history_num_txs, which tracks the number of txs for each script. This tells us directly which page we need to append new txs to and how many without looking up anything else.
- Use a "merge operator" (available in RocksDB, see https://github.com/facebook/rocksdb/wiki/Merge-Operator), which simply concats bytes to pages (for insert) or trims them (for delete).
- Store pages not as serialized Vec<TxNum>, but as concatenated byte stream of serialized TxNum. This allows us to simply byte-append serialized tx nums, which simplifies the merge operator.
- Use multi_get when querying the num of existing txs for scripts, which allows RocksDB to fetch data more efficiently than when querying one-by-one.
Data that has been gathered + methodology for the optimization can be found here:
https://marble-timimus-187.notion.site/Bitcoin-ABC-Chronik-082e0ebbb02343198c79d2c58d2e9b45?pvs=4