Page MenuHomePhabricator

index: fix reindex-chainstate with active indexes
Needs RevisionPublic

Authored by PiRK on Thu, Feb 27, 15:10.

Details

Reviewers
Fabien
Group Reviewers
Restricted Project
Summary

When started together with -reindex-chainstate, currently coinstatsindex gets corrupted and the blockfilterindex flatfiles duplicated.

This makes two improvements to the index init phase:

  1. Prevent index corruption in case a reorg happens when the index was switched off:

This is done by reading in the top block stored in the locator instead of looking for a fork point already in BaseIndex::Init().
Before, we'd just go back to the fork point by calling FindForkInGlobalIndex(), which would have corrupted the coinstatsindex because its saved muhash needs to be reverted step by step by un-applying all blocks in between, which wasn't done before. This is now being done a bit later in ThreadSync(), which has existing logic to call the custom Rewind() method when going back along the chain to the forking point (thanks ryanofsky for pointing this out to me!).

  1. Allow using the -reindex-chainstate option without needing to disabling indexes:

With BaseIndex::Init() not calling FindForkInGlobalIndex() anymore, we can allow reindex-chainstate with active indexes. reindex-chainstate deletes the chain and rebuilds it later in ThreadImport, so there is no chain available during BaseIndex::Init(), which would lead to problems (see #24789).
But now we'll only need the chain a bit later in BaseIndex::ThreadSync, which will wait for the reindex-chainstate in ThreadImport to finish and will continue syncing after that.

This is a backport of core#24789 and core#25193

Test Plan

ninja all check-all

Event Timeline

PiRK requested review of this revision.Thu, Feb 27, 15:10

Tail of the build log:

[392/577] Building CXX object src/CMakeFiles/bitcoin-cli.dir/bitcoin-cli.cpp.o
[393/577] Linking C executable src/secp256k1/ecmult-bench
[394/577] Linking CXX executable src/bitcoin-cli
[395/577] Building CXX object src/CMakeFiles/server.dir/rpc/blockchain.cpp.o
[396/577] Building CXX object src/CMakeFiles/bitcoind.dir/bitcoind.cpp.o
[397/577] Building CXX object src/test/CMakeFiles/testutil.dir/util/str.cpp.o
[398/577] Building CXX object src/CMakeFiles/server.dir/torcontrol.cpp.o
[399/577] Building CXX object src/test/CMakeFiles/testutil.dir/util/logging.cpp.o
[400/577] Building CXX object src/CMakeFiles/bitcoin-wallet.dir/bitcoin-wallet.cpp.o
[401/577] Building CXX object src/test/CMakeFiles/testutil.dir/util/blockindex.cpp.o
[402/577] Building CXX object src/test/CMakeFiles/testutil.dir/util/coins.cpp.o
[403/577] Building CXX object src/test/CMakeFiles/testutil.dir/util/random.cpp.o
[404/577] Building CXX object src/CMakeFiles/bitcoin-tx.dir/bitcoin-tx.cpp.o
[405/577] Linking CXX executable src/bitcoin-tx
[406/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/context.cpp.o
[407/577] Building CXX object src/test/CMakeFiles/testutil.dir/util/transaction_utils.cpp.o
[408/577] Building CXX object src/test/CMakeFiles/testutil.dir/util/blockfilter.cpp.o
[409/577] Building CXX object src/CMakeFiles/server.dir/wallet/init.cpp.o
[410/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/coincontrol.cpp.o
[411/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/coinselection.cpp.o
[412/577] Building CXX object src/test/CMakeFiles/testutil.dir/util/mining.cpp.o
[413/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/db.cpp.o
[414/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/crypter.cpp.o
[415/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/bdb.cpp.o
[416/577] Building CXX object src/test/CMakeFiles/testutil.dir/util/net.cpp.o
[417/577] Building CXX object src/test/CMakeFiles/testutil.dir/util/validation.cpp.o
[418/577] Building CXX object src/test/CMakeFiles/testutil.dir/util/txmempool.cpp.o
[419/577] Building CXX object src/test/CMakeFiles/testutil.dir/util/wallet.cpp.o
[420/577] Building CXX object src/test/CMakeFiles/testutil.dir/util/setup_common.cpp.o
[421/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/fees.cpp.o
[422/577] Building CXX object src/CMakeFiles/server.dir/validation.cpp.o
[423/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/load.cpp.o
[424/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/rpc/encrypt.cpp.o
[425/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/sqlite.cpp.o
[426/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/rpc/util.cpp.o
[427/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/rpc/signmessage.cpp.o
[428/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/receive.cpp.o
[429/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/transaction.cpp.o
[430/577] Building CXX object src/seeder/CMakeFiles/seeder-base.dir/dns.cpp.o
[431/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/salvage.cpp.o
[432/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/interfaces.cpp.o
[433/577] Building CXX object src/seeder/CMakeFiles/seeder-base.dir/options.cpp.o
[434/577] Building CXX object src/seeder/CMakeFiles/seeder-base.dir/db.cpp.o
[435/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/walletutil.cpp.o
[436/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/spend.cpp.o
[437/577] Building CXX object src/iguana/CMakeFiles/iguana.dir/iguana_formatter.cpp.o
[438/577] Building CXX object src/iguana/CMakeFiles/iguana.dir/iguana_interpreter.cpp.o
[439/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/scriptpubkeyman.cpp.o
[440/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/rpc/backup.cpp.o
[441/577] Building CXX object src/iguana/CMakeFiles/iguana.dir/iguana.cpp.o
[442/577] Building CXX object src/seeder/CMakeFiles/bitcoin-seeder.dir/main.cpp.o
[443/577] Linking CXX executable src/iguana/iguana
[444/577] Building CXX object src/seeder/CMakeFiles/seeder-base.dir/bitcoin.cpp.o
[445/577] Building CXX object src/wallet/CMakeFiles/wallet-tool.dir/wallettool.cpp.o
[446/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/walletdb.cpp.o
[447/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/rpcwallet.cpp.o
[448/577] Building CXX object src/wallet/CMakeFiles/wallet.dir/wallet.cpp.o
[449/577] Linking CXX static library src/wallet/libwallet.a
ninja: build stopped: cannot make progress due to previous errors.
Build build-clang-tidy failed with exit code 1
PiRK planned changes to this revision.Thu, Feb 27, 15:17

Failed tests logs:

====== Bitcoin ABC functional tests: feature_coinstatsindex.py ======

------- Stdout: -------
2025-02-27T15:15:31.876000Z TestFramework (INFO): Initializing test directory /work/abc-ci-builds/build-without-wallet/test/tmp/test_runner_₿₵_🏃_20250227_151412/feature_coinstatsindex_131
2025-02-27T15:15:33.661000Z TestFramework (INFO): Test that gettxoutsetinfo() output is consistent with or without coinstatsindex option
2025-02-27T15:15:33.786000Z TestFramework (INFO): Test that gettxoutsetinfo() can get fetch data on specific heights with index
2025-02-27T15:15:33.999000Z TestFramework (INFO): Test gettxoutsetinfo() with index and verbose flag
2025-02-27T15:15:34.270000Z TestFramework (INFO): Test that the index is robust across restarts
2025-02-27T15:15:35.079000Z TestFramework (INFO): Test that the index works with -reindex
2025-02-27T15:15:36.364000Z TestFramework (INFO): Test that the index works with -reindex-chainstate
2025-02-27T15:15:37.356000Z TestFramework (INFO): Test use_index option for nodes running the index
2025-02-27T15:15:37.498000Z TestFramework (INFO): Test that index can handle reorgs
2025-02-27T15:15:37.961000Z TestFramework (INFO): Test that the rpc raises if the legacy hash is passed with the index
2025-02-27T15:15:38.059000Z TestFramework (INFO): Test a reorg while the index is deactivated
2025-02-27T15:16:39.169000Z TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 149, in main
    self._run_test_internal()
  File "/work/test/functional/test_framework/test_framework.py", line 139, in _run_test_internal
    self.run_test()
  File "/work/test/functional/feature_coinstatsindex.py", line 41, in run_test
    self._test_init_index_after_reorg()
  File "/work/test/functional/feature_coinstatsindex.py", line 354, in _test_init_index_after_reorg
    self.generatetoaddress(index_node, 5, getnewdestination()[2])
  File "/work/test/functional/test_framework/test_framework.py", line 787, in generatetoaddress
    sync_fun() if sync_fun else self.sync_all()
                                ^^^^^^^^^^^^^^^
  File "/work/test/functional/test_framework/test_framework.py", line 869, in sync_all
    self.sync_blocks(nodes)
  File "/work/test/functional/test_framework/test_framework.py", line 813, in sync_blocks
    raise AssertionError(f"Block sync timed out after {timeout}s:{best_hashes}")
AssertionError: Block sync timed out after 60s:
  '0670c1bbfdb34016c0559d11f586318b5d19e5dca1c93312f38b114b4e59be72'
  '405f63785825a47ddffe321b82a6377880d1fa5449d51d1c724ef2269cf74617'
2025-02-27T15:16:39.224000Z TestFramework (INFO): Stopping nodes
2025-02-27T15:16:39.380000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/build-without-wallet/test/tmp/test_runner_₿₵_🏃_20250227_151412/feature_coinstatsindex_131
2025-02-27T15:16:39.381000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/build-without-wallet/test/tmp/test_runner_₿₵_🏃_20250227_151412/feature_coinstatsindex_131/test_framework.log
2025-02-27T15:16:39.381000Z TestFramework (ERROR): 
2025-02-27T15:16:39.381000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/build-without-wallet/test/tmp/test_runner_₿₵_🏃_20250227_151412/feature_coinstatsindex_131' to consolidate all logs
2025-02-27T15:16:39.381000Z TestFramework (ERROR): 
2025-02-27T15:16:39.382000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2025-02-27T15:16:39.382000Z TestFramework (ERROR): https://github.com/Bitcoin-ABC/bitcoin-abc/issues
2025-02-27T15:16:39.382000Z TestFramework (ERROR):

Each failure log is accessible here:
Bitcoin ABC functional tests: feature_coinstatsindex.py

Failed tests logs:

====== Bitcoin ABC functional tests: feature_coinstatsindex.py ======

------- Stdout: -------
2025-02-27T15:17:52.525000Z TestFramework (INFO): Initializing test directory /work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_🏃_20250227_151436/feature_coinstatsindex_131
2025-02-27T15:17:55.663000Z TestFramework (INFO): Test that gettxoutsetinfo() output is consistent with or without coinstatsindex option
2025-02-27T15:17:55.946000Z TestFramework (INFO): Test that gettxoutsetinfo() can get fetch data on specific heights with index
2025-02-27T15:17:56.480000Z TestFramework (INFO): Test gettxoutsetinfo() with index and verbose flag
2025-02-27T15:17:57.144000Z TestFramework (INFO): Test that the index is robust across restarts
2025-02-27T15:17:58.604000Z TestFramework (INFO): Test that the index works with -reindex
2025-02-27T15:18:00.395000Z TestFramework (INFO): Test that the index works with -reindex-chainstate
2025-02-27T15:18:02.080000Z TestFramework (INFO): Test use_index option for nodes running the index
2025-02-27T15:18:02.299000Z TestFramework (INFO): Test that index can handle reorgs
2025-02-27T15:18:03.051000Z TestFramework (INFO): Test that the rpc raises if the legacy hash is passed with the index
2025-02-27T15:18:03.337000Z TestFramework (INFO): Test a reorg while the index is deactivated
2025-02-27T15:19:04.675000Z TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 149, in main
    self._run_test_internal()
  File "/work/test/functional/test_framework/test_framework.py", line 139, in _run_test_internal
    self.run_test()
  File "/work/test/functional/feature_coinstatsindex.py", line 41, in run_test
    self._test_init_index_after_reorg()
  File "/work/test/functional/feature_coinstatsindex.py", line 354, in _test_init_index_after_reorg
    self.generatetoaddress(index_node, 5, getnewdestination()[2])
  File "/work/test/functional/test_framework/test_framework.py", line 787, in generatetoaddress
    sync_fun() if sync_fun else self.sync_all()
                                ^^^^^^^^^^^^^^^
  File "/work/test/functional/test_framework/test_framework.py", line 869, in sync_all
    self.sync_blocks(nodes)
  File "/work/test/functional/test_framework/test_framework.py", line 813, in sync_blocks
    raise AssertionError(f"Block sync timed out after {timeout}s:{best_hashes}")
AssertionError: Block sync timed out after 60s:
  '50d72641a3f95733b72ba1418f8eb28044c13f80392f7aff01b70b2c47056859'
  '574dd4870ae7c5f41fef4b8986e11cf806b4c75adb5fc7e3c98cf58e445944f9'
2025-02-27T15:19:04.730000Z TestFramework (INFO): Stopping nodes
2025-02-27T15:19:04.987000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_🏃_20250227_151436/feature_coinstatsindex_131
2025-02-27T15:19:04.987000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_🏃_20250227_151436/feature_coinstatsindex_131/test_framework.log
2025-02-27T15:19:04.987000Z TestFramework (ERROR): 
2025-02-27T15:19:04.988000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_🏃_20250227_151436/feature_coinstatsindex_131' to consolidate all logs
2025-02-27T15:19:04.988000Z TestFramework (ERROR): 
2025-02-27T15:19:04.988000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2025-02-27T15:19:04.989000Z TestFramework (ERROR): https://github.com/Bitcoin-ABC/bitcoin-abc/issues
2025-02-27T15:19:04.989000Z TestFramework (ERROR):
====== Bitcoin ABC functional tests with the next upgrade activated: feature_coinstatsindex.py ======

------- Stdout: -------
2025-02-27T15:21:46.176000Z TestFramework (INFO): Initializing test directory /work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_🏃_20250227_151905/feature_coinstatsindex_131
2025-02-27T15:21:49.339000Z TestFramework (INFO): Test that gettxoutsetinfo() output is consistent with or without coinstatsindex option
2025-02-27T15:21:49.580000Z TestFramework (INFO): Test that gettxoutsetinfo() can get fetch data on specific heights with index
2025-02-27T15:21:50.099000Z TestFramework (INFO): Test gettxoutsetinfo() with index and verbose flag
2025-02-27T15:21:50.730000Z TestFramework (INFO): Test that the index is robust across restarts
2025-02-27T15:21:52.122000Z TestFramework (INFO): Test that the index works with -reindex
2025-02-27T15:21:53.880000Z TestFramework (INFO): Test that the index works with -reindex-chainstate
2025-02-27T15:21:55.414000Z TestFramework (INFO): Test use_index option for nodes running the index
2025-02-27T15:21:55.607000Z TestFramework (INFO): Test that index can handle reorgs
2025-02-27T15:21:56.377000Z TestFramework (INFO): Test that the rpc raises if the legacy hash is passed with the index
2025-02-27T15:21:56.659000Z TestFramework (INFO): Test a reorg while the index is deactivated
2025-02-27T15:22:58.042000Z TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 149, in main
    self._run_test_internal()
  File "/work/test/functional/test_framework/test_framework.py", line 139, in _run_test_internal
    self.run_test()
  File "/work/test/functional/feature_coinstatsindex.py", line 41, in run_test
    self._test_init_index_after_reorg()
  File "/work/test/functional/feature_coinstatsindex.py", line 354, in _test_init_index_after_reorg
    self.generatetoaddress(index_node, 5, getnewdestination()[2])
  File "/work/test/functional/test_framework/test_framework.py", line 787, in generatetoaddress
    sync_fun() if sync_fun else self.sync_all()
                                ^^^^^^^^^^^^^^^
  File "/work/test/functional/test_framework/test_framework.py", line 869, in sync_all
    self.sync_blocks(nodes)
  File "/work/test/functional/test_framework/test_framework.py", line 813, in sync_blocks
    raise AssertionError(f"Block sync timed out after {timeout}s:{best_hashes}")
AssertionError: Block sync timed out after 60s:
  '1c56e9df20f75341ceb2cc026a6d8308e0e57a3cab0a7a9be798aa506b7df28e'
  '2430bef48290414a6214c5d7075a3755264b700fd614e578b8213bbca32bb428'
2025-02-27T15:22:58.097000Z TestFramework (INFO): Stopping nodes
2025-02-27T15:22:58.404000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_🏃_20250227_151905/feature_coinstatsindex_131
2025-02-27T15:22:58.404000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_🏃_20250227_151905/feature_coinstatsindex_131/test_framework.log
2025-02-27T15:22:58.404000Z TestFramework (ERROR): 
2025-02-27T15:22:58.404000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_🏃_20250227_151905/feature_coinstatsindex_131' to consolidate all logs
2025-02-27T15:22:58.404000Z TestFramework (ERROR): 
2025-02-27T15:22:58.404000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2025-02-27T15:22:58.404000Z TestFramework (ERROR): https://github.com/Bitcoin-ABC/bitcoin-abc/issues
2025-02-27T15:22:58.404000Z TestFramework (ERROR):

Each failure log is accessible here:
Bitcoin ABC functional tests: feature_coinstatsindex.py
Bitcoin ABC functional tests with the next upgrade activated: feature_coinstatsindex.py

fix clang-tidy style error, fix test flakiness (mine enough new block to trigger automatic unparking on the other node, 5 was not enough)

test/functional/feature_coinstatsindex.py
356 ↗(On Diff #52813)

The number of blocks mined here made the test pass 10 times out of 10 for me. But there is a part of uncertainty, as the automatic unparking behavior depends on accumulated PoW differences between the two chains which is by nature random.
If we see sporadic failures we can raise the number of blocks or just use "-noparkdeepreorg" on node0

Fabien requested changes to this revision.Thu, Feb 27, 20:38
Fabien added a subscriber: Fabien.
Fabien added inline comments.
test/functional/feature_coinstatsindex.py
356 ↗(On Diff #52813)

use "-noparkdeepreorg"

This is a much better solution, it's self documenting and it prevents breaking the test in the future

This revision now requires changes to proceed.Thu, Feb 27, 20:38