Page MenuHomePhabricator

[txindex] Activate new transaction index code that runs in background
ClosedPublic

Authored by markblundeberg on May 5 2019, 01:21.

Details

Summary

Backports the remaining six commits from PR13033, thus activating the
new transaction index database:
9b2704777 [doc] Include txindex changes in the release notes.
ed77dd6b3 [test] Simple unit test for TxIndex.
6d772a3d4 [rpc] Public interfaces to GetTransaction block until synced.
a03f804f2 [index] Move disk IO logic from GetTransaction to TxIndex::FindTx.
e0a3b8003 [validation] Replace tx index code in validation code with TxIndex.
8181db88f [init] Initialize and start TxIndex in init code.

Also incorporates fixes from PR14071 and PR15410:
faf4a9b67 qa: Stop txindex thread before calling destructor
fab6b07c1 test: txindex: interrupt threadGroup before calling destructor

https://github.com/bitcoin/bitcoin/compare/8181db88f~1..9b2704777
https://github.com/bitcoin/bitcoin/pull/13033/files
https://github.com/bitcoin/bitcoin/pull/14071/files
https://github.com/bitcoin/bitcoin/pull/15410/files

Depends on D2904

Test Plan

make check
test_runner.py

manual testing done:

  • migration from existing txindex (on testnet: 2 minute block on startup)
  • fresh txindex=1 with new code (testnet: 30 minutes background job)
  • mainnet test (57 minutes for blocking migration on startup; 8.5 hours fresh indexing in background)
  • test ElectrumX compatibility

Diff Detail

Repository
rABC Bitcoin ABC
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

markblundeberg created this revision.May 5 2019, 01:21
Herald added a reviewer: Restricted Project. · View Herald TranscriptMay 5 2019, 01:21

RFC: I know the release-notes-pr13033.md doesn't make sense as a file name, but leaving it as-is for now until I find a better place for this text.

doc/files.md
12 ↗(On Diff #8389)

note to self: don't forget to change the 0.19.x before landing

Some testnet tests:

Benchmark for migration (important as it blocks during node startup!)

2019-05-02T02:07:39Z Using obfuscation key for /home/bitcoin/testnet/data/testne
t3/indexes/txindex: 0000000000000000
2019-05-02T02:07:39Z Upgrading txindex database... [0%]
2019-05-02T02:07:53Z Upgrading txindex database... [10%]
2019-05-02T02:08:08Z Upgrading txindex database... [20%]
2019-05-02T02:08:20Z Upgrading txindex database... [30%]
2019-05-02T02:08:32Z Upgrading txindex database... [40%]
2019-05-02T02:08:46Z Upgrading txindex database... [50%]
2019-05-02T02:09:00Z Upgrading txindex database... [60%]
2019-05-02T02:09:17Z Upgrading txindex database... [70%]
2019-05-02T02:09:31Z Upgrading txindex database... [80%]
2019-05-02T02:09:45Z Upgrading txindex database... [90%]
2019-05-02T02:09:59Z Upgrading txindex database... [100%]
2019-05-02T02:10:01Z [DONE].

Final txindex directory db size: 985712

Benchmark for tx indexing in background (nice because node continues to operate perfectly)

2019-05-01T23:32:55Z txindex thread start
2019-05-02T00:04:37Z txindex thread exit

Final txindex directory db size: 926572 kB (noticeably smaller than above but I suppose it's due to having a more recent leveldb compression)

markblundeberg edited the summary of this revision. (Show Details)May 5 2019, 17:08
markblundeberg updated this revision to Diff 8405.May 5 2019, 21:55
markblundeberg edited the summary of this revision. (Show Details)

incorporate fixes from D2925

markblundeberg planned changes to this revision.May 7 2019, 14:10
markblundeberg retitled this revision from [txindex] Activate new transaction index code that runs in background to [WIP] [txindex] Activate new transaction index code that runs in background.May 7 2019, 14:15

Notes from core's upgrade: https://bitcoin.org/en/release/v0.17.0

If your node has a txindex, the txindex db will be migrated the first time you run 0.17.0 or newer, which may take up to a few hours. Your node will not be functional until this migration completes.

markblundeberg edited the summary of this revision. (Show Details)May 19 2019, 19:15
markblundeberg edited the test plan for this revision. (Show Details)

rebased (still WIP)

markblundeberg planned changes to this revision.May 20 2019, 23:50
markblundeberg added a comment.EditedMay 23 2019, 03:15

Mainnet migration test (on a VPS with OK specs -- about 57 minutes)

...
2019-05-23T02:08:01Z Upgrading txindex database... [0%]
2019-05-23T02:13:57Z Upgrading txindex database... [10%]
2019-05-23T02:19:52Z Upgrading txindex database... [20%]
2019-05-23T02:25:57Z Upgrading txindex database... [30%]
2019-05-23T02:31:26Z Upgrading txindex database... [40%]
2019-05-23T02:37:09Z Upgrading txindex database... [50%]
2019-05-23T02:42:31Z 

< did kill -9 just to check robustness; then restart >

...
2019-05-23T02:42:47Z Upgrading txindex database... [0%]
2019-05-23T02:42:47Z Upgrading txindex database... [59%]
2019-05-23T02:43:11Z Upgrading txindex database... [60%]
2019-05-23T02:48:45Z Upgrading txindex database... [70%]
2019-05-23T02:53:52Z Upgrading txindex database... [80%]
2019-05-23T02:59:25Z Upgrading txindex database... [90%]
2019-05-23T03:04:55Z Upgrading txindex database... [100%]
...

Note that during migration users will see the following when they try an RPC call. Be prepared for confused people who didn't read release notes to ask questions about this!

error code: -28
error message:
Verifying blocks...

Mainnet indexing in background results: 8.5 hours

2019-05-23T03:08:04Z Using obfuscation key for /home/bitcoin/.bitcoin/indexes/tx
index: 0000000000000000
2019-05-23T03:08:04Z txindex thread start
...
2019-05-23T03:24:36Z Syncing txindex with block chain from height 200889
...
2019-05-23T04:28:09Z Syncing txindex with block chain from height 300655
...
2019-05-23T07:13:36Z Syncing txindex with block chain from height 400084
...
2019-05-23T10:53:08Z Syncing txindex with block chain from height 501172
...
2019-05-23T11:42:01Z Syncing txindex with block chain from height 582571
2019-05-23T11:42:32Z Syncing txindex with block chain from height 583169
2019-05-23T11:43:03Z Syncing txindex with block chain from height 583374
2019-05-23T11:43:26Z txindex is enabled at height 583818
2019-05-23T11:43:26Z txindex thread exit
markblundeberg edited the test plan for this revision. (Show Details)May 23 2019, 14:25
markblundeberg updated this revision to Diff 8852.EditedMay 24 2019, 14:37
markblundeberg retitled this revision from [WIP] [txindex] Activate new transaction index code that runs in background to [txindex] Activate new transaction index code that runs in background.
markblundeberg edited the test plan for this revision. (Show Details)

Testing done :-) ... this is robust!
(& rebased)

markblundeberg edited the test plan for this revision. (Show Details)May 24 2019, 14:37
markblundeberg added a comment.EditedMay 24 2019, 14:43

(I'm happy to have this deferred to 0.19.8, if there is a desire for another 'good and stable' release after 0.19.6)

markblundeberg added inline comments.May 24 2019, 16:10
doc/files.md
12 ↗(On Diff #8852)

note to self again -- don't forget to change 0.19.x when release becomes clear

markblundeberg edited the summary of this revision. (Show Details)May 26 2019, 22:43
deadalnix accepted this revision.May 26 2019, 23:43
deadalnix added inline comments.
doc/files.md
12 ↗(On Diff #8852)

Put the proper version in there.

This revision is now accepted and ready to land.May 26 2019, 23:43

last rebase & set the version number