Details

Reviewers

deadalnix
jasonbcox
Fabien

Group Reviewers

Restricted Project

Maniphest Tasks

T617: Backport PR 13033 "Build txindex in parallel with validation"

Commits

rSTAGINGb12ddc527a09: [txindex] Activate new transaction index code that runs in background
rABCb12ddc527a09: [txindex] Activate new transaction index code that runs in background

Summary

Backports the remaining six commits from PR13033, thus activating the
new transaction index database:
9b2704777 [doc] Include txindex changes in the release notes.
ed77dd6b3 [test] Simple unit test for TxIndex.
6d772a3d4 [rpc] Public interfaces to GetTransaction block until synced.
a03f804f2 [index] Move disk IO logic from GetTransaction to TxIndex::FindTx.
e0a3b8003 [validation] Replace tx index code in validation code with TxIndex.
8181db88f [init] Initialize and start TxIndex in init code.

Also incorporates fixes from PR14071 and PR15410:
faf4a9b67 qa: Stop txindex thread before calling destructor
fab6b07c1 test: txindex: interrupt threadGroup before calling destructor

https://github.com/bitcoin/bitcoin/compare/8181db88f~1..9b2704777
https://github.com/bitcoin/bitcoin/pull/13033/files
https://github.com/bitcoin/bitcoin/pull/14071/files
https://github.com/bitcoin/bitcoin/pull/15410/files

Depends on D2904

Test Plan

make check
test_runner.py

manual testing done:

migration from existing txindex (on testnet: 2 minute block on startup)
fresh txindex=1 with new code (testnet: 30 minutes background job)
mainnet test (57 minutes for blocking migration on startup; 8.5 hours fresh indexing in background)
test ElectrumX compatibility

Diff Detail

Repository

rABC Bitcoin ABC

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

markblundeberg created this revision.May 5 2019, 01:21

Herald added a reviewer: Restricted Project. · View Herald TranscriptMay 5 2019, 01:21

Herald added a reviewer: deadalnix. · View Herald Transcript

markblundeberg added a task: T617: Backport PR 13033 "Build txindex in parallel with validation".May 5 2019, 01:22

markblundeberg added a parent revision: D2904: [index] Create new TxIndex class..

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing started.

RFC: I know the release-notes-pr13033.md doesn't make sense as a file name, but leaving it as-is for now until I find a better place for this text.

doc/files.md
12 ↗	(On Diff #8389)	note to self: don't forget to change the 0.19.x before landing

Some testnet tests:

Benchmark for migration (important as it blocks during node startup!)

2019-05-02T02:07:39Z Using obfuscation key for /home/bitcoin/testnet/data/testne
t3/indexes/txindex: 0000000000000000
2019-05-02T02:07:39Z Upgrading txindex database... [0%]
2019-05-02T02:07:53Z Upgrading txindex database... [10%]
2019-05-02T02:08:08Z Upgrading txindex database... [20%]
2019-05-02T02:08:20Z Upgrading txindex database... [30%]
2019-05-02T02:08:32Z Upgrading txindex database... [40%]
2019-05-02T02:08:46Z Upgrading txindex database... [50%]
2019-05-02T02:09:00Z Upgrading txindex database... [60%]
2019-05-02T02:09:17Z Upgrading txindex database... [70%]
2019-05-02T02:09:31Z Upgrading txindex database... [80%]
2019-05-02T02:09:45Z Upgrading txindex database... [90%]
2019-05-02T02:09:59Z Upgrading txindex database... [100%]
2019-05-02T02:10:01Z [DONE].

Final txindex directory db size: 985712

Benchmark for tx indexing in background (nice because node continues to operate perfectly)

2019-05-01T23:32:55Z txindex thread start
2019-05-02T00:04:37Z txindex thread exit

Final txindex directory db size: 926572 kB (noticeably smaller than above but I suppose it's due to having a more recent leveldb compression)

markblundeberg added a child revision: D2925: test: Stop txindex thread before calling destructor.May 5 2019, 01:37

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing passed.

Harbormaster completed remote builds in B5674: Diff 8389.May 5 2019, 01:37

markblundeberg edited the summary of this revision. (Show Details)May 5 2019, 17:08

markblundeberg added a child revision: D2933: db: Remove obsolete methods from CBlockTreeDB..May 5 2019, 20:11

markblundeberg added a child revision: D2934: db: Make reusable base class for index databases..May 5 2019, 20:17

markblundeberg added a child revision: D2935: index: Extract logic from TxIndex into reusable base class..May 5 2019, 20:22

incorporate fixes from D2925

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing started.

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing passed.

Harbormaster completed remote builds in B5690: Diff 8405.May 5 2019, 22:10

markblundeberg added a parent revision: D2903: [db] Migration for txindex data to new, separate database..May 6 2019, 00:06

markblundeberg planned changes to this revision.May 7 2019, 14:10

markblundeberg retitled this revision from [txindex] Activate new transaction index code that runs in background to [WIP] [txindex] Activate new transaction index code that runs in background.May 7 2019, 14:15

Notes from core's upgrade: https://bitcoin.org/en/release/v0.17.0

If your node has a txindex, the txindex db will be migrated the first time you run 0.17.0 or newer, which may take up to a few hours. Your node will not be functional until this migration completes.

markblundeberg edited the summary of this revision. (Show Details)May 19 2019, 19:15

rebased (still WIP)

markblundeberg planned changes to this revision.May 20 2019, 23:50

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing started.

Harbormaster completed remote builds in B5915: Diff 8765.May 21 2019, 00:05

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing passed.

Mainnet migration test (on a VPS with OK specs -- about 57 minutes)

...
2019-05-23T02:08:01Z Upgrading txindex database... [0%]
2019-05-23T02:13:57Z Upgrading txindex database... [10%]
2019-05-23T02:19:52Z Upgrading txindex database... [20%]
2019-05-23T02:25:57Z Upgrading txindex database... [30%]
2019-05-23T02:31:26Z Upgrading txindex database... [40%]
2019-05-23T02:37:09Z Upgrading txindex database... [50%]
2019-05-23T02:42:31Z 

< did kill -9 just to check robustness; then restart >

...
2019-05-23T02:42:47Z Upgrading txindex database... [0%]
2019-05-23T02:42:47Z Upgrading txindex database... [59%]
2019-05-23T02:43:11Z Upgrading txindex database... [60%]
2019-05-23T02:48:45Z Upgrading txindex database... [70%]
2019-05-23T02:53:52Z Upgrading txindex database... [80%]
2019-05-23T02:59:25Z Upgrading txindex database... [90%]
2019-05-23T03:04:55Z Upgrading txindex database... [100%]
...

Note that during migration users will see the following when they try an RPC call. Be prepared for confused people who didn't read release notes to ask questions about this!

error code: -28
error message:
Verifying blocks...

Mainnet indexing in background results: 8.5 hours

2019-05-23T03:08:04Z Using obfuscation key for /home/bitcoin/.bitcoin/indexes/tx
index: 0000000000000000
2019-05-23T03:08:04Z txindex thread start
...
2019-05-23T03:24:36Z Syncing txindex with block chain from height 200889
...
2019-05-23T04:28:09Z Syncing txindex with block chain from height 300655
...
2019-05-23T07:13:36Z Syncing txindex with block chain from height 400084
...
2019-05-23T10:53:08Z Syncing txindex with block chain from height 501172
...
2019-05-23T11:42:01Z Syncing txindex with block chain from height 582571
2019-05-23T11:42:32Z Syncing txindex with block chain from height 583169
2019-05-23T11:43:03Z Syncing txindex with block chain from height 583374
2019-05-23T11:43:26Z txindex is enabled at height 583818
2019-05-23T11:43:26Z txindex thread exit

markblundeberg edited the test plan for this revision. (Show Details)May 23 2019, 14:25