Page MenuHomePhabricator
Differential D11145

Allow maintaining the blockfilterindex when using prune
AbandonedPublic
Actions

Authored by PiRK on Mar 8 2022, 14:51.

Details

Reviewers
Fabien
Group Reviewers
Restricted Project
Summary

PR description:

Maintaining the blockfilterindexes in prune mode is possible and may lead to efficient p2p based rescans of wallets (restore backups, import/sweep keys) beyond the prune height (rescans not part of that PR).

This PR allows running the blockfilterindex(es) in conjunction with pruning.

  • Bitcoind/Qt will shutdown during startup when missing block data has been detected ([re]enable -blockfilterindex when we already have pruned)
  • manual block pruning is disabled during blockfilterindex sync
  • auto-pruning is delayed during blockfilterindex sync

Allow blockfilter in conjunction with prune

https://github.com/bitcoin/bitcoin/pull/15946/commits/6abe9f5b11cd4a5ecb6caca8443fe2950a417842

index: Fix backwards search for bestblock

https://github.com/bitcoin/bitcoin/pull/23365/commits/698c524698c33595a4d555eaa9e21bc19b4d3e93

Avoid accessing nullpointer in BaseIndex::GetSummary()

https://github.com/bitcoin/bitcoin/pull/15946/commits/00d57ff76854938ead800767fb673a8af46eac8e

Avoid pruning below the blockfilterindex sync height

https://github.com/bitcoin/bitcoin/pull/15946/commits/5e112269c311a559bfded814d3c3c438349a1986

This is a backport of core#15946 [1/2] and core#23365 [1/2]

PR23365 fixes a rare issue when only the genesis block is indexed.

Test Plan

ninja all check-all

src/bitcoind -prune=2000 -blockfilterindex=1

Diff Detail

Event Timeline

PiRK created this revision.Mar 8 2022, 14:51
Herald added a reviewer: Restricted Project. · View Herald TranscriptMar 8 2022, 14:51
PiRK requested review of this revision.Mar 8 2022, 14:51
teamcity edited the summary of this revision. (Show Details)Mar 8 2022, 14:51
Build Bitcoin ABC Diffs / Diff Testing (build-clang) failed.

Tail of the build log:

Build 'Bitcoin ABC Diffs / Diff Testing' #43657, branch 'refs/tags/phabricator/diff/32633'
Triggered 2022-03-08 14:51:47 by 'Phabricator Staging (phabricator-staging)'
Started 2022-03-08 14:51:50 on agent 'buildagent1'
Finished 2022-03-08 14:52:04 with status FAILURE 'Snapshot dependency failed to start: Automated Deployments / Bitcoin ABC Infra / Bitcoin-ABC Infra Checkout'
VCS revisions: 'BitcoinABC_BitcoinAbcStaging' (Git, instance id 5): 'N/A' (checkout rules: '+:. => ./bitcoin-abc')
TeamCity URL https://build.bitcoinabc.org/viewLog.html?buildId=364167&buildTypeId=BitcoinABC_BitcoinAbcStaging 
TeamCity server version is 2019.2.4 (build 72059), server timezone: GMT (UTC)

[14:51:47]E: bt15 (running for 16s)
[14:51:47]i: TeamCity server version is 2019.2.4 (build 72059)
[14:51:47] : Collecting changes in 2 VCS roots
[14:51:47] :	 [Collecting changes in 2 VCS roots] VCS Root details
[14:51:47] :		 [VCS Root details] "Bitcoin ABC Staging" {instance id=5, parent internal id=3, parent id=BitcoinABC_BitcoinAbcStaging, description: "ssh://vcs@reviews.bitcoinabc.org:2221/source/bitcoin-abc-staging.git#refs/heads/master"}
[14:51:47] :		 [VCS Root details] "abc-infrastructure" {instance id=7, parent internal id=7, parent id=AutomatedDeployments_BitcoinAbcDeveloperTools_AbcInfrastructure, description: "ssh://vcs@reviews.bitcoinabc.org:2221/source/infrastructure.git#refs/heads/master"}
[14:51:50] : The build is removed from the queue to be prepared for the start
[14:52:04]W: This build has not been started because some of the builds it depends on failed to start
[14:52:04]E: Snapshot dependency "Automated Deployments / Bitcoin ABC Infra / Bitcoin-ABC Infra Checkout" failed
[14:52:04] : Build finished
Harbormaster completed remote builds in B18431: Diff 32633.Mar 8 2022, 14:53
PiRK added a child revision: D11146: Add functional test for blockfilterindex in prune-mode.Mar 8 2022, 14:57
teamcity edited the summary of this revision. (Show Details)Mar 8 2022, 14:58
PiRK mentioned this in D11147: test: Add edge case of pruning up to index height.Mar 8 2022, 15:02
PiRK edited the summary of this revision. (Show Details)Mar 8 2022, 15:05
PiRK updated this revision to Diff 32647.Mar 10 2022, 08:55

rebase

teamcity added a comment.Mar 10 2022, 08:55
Build Bitcoin ABC Diffs / Diff Testing (build-clang) failed.

Tail of the build log:

Build 'Bitcoin ABC Diffs / Diff Testing' #43696, branch 'refs/tags/phabricator/diff/32647'
Triggered 2022-03-10 08:55:22 by 'Phabricator Staging (phabricator-staging)'
Started 2022-03-10 08:55:25 on agent 'buildagent1'
Finished 2022-03-10 08:55:30 with status FAILURE 'Snapshot dependency failed to start: Automated Deployments / Bitcoin ABC Infra / Bitcoin-ABC Infra Checkout'
VCS revisions: 'PhabBitcoinAbcStagingRo' (Git, instance id 18): 'N/A' (checkout rules: '+:. => ./bitcoin-abc')
TeamCity URL https://build.bitcoinabc.org/viewLog.html?buildId=364312&buildTypeId=BitcoinABC_BitcoinAbcStaging 
TeamCity server version is 2019.2.4 (build 72059), server timezone: GMT (UTC)

[08:55:22]E: bt15 (running for 7s)
[08:55:22]i: TeamCity server version is 2019.2.4 (build 72059)
[08:55:22] : Collecting changes in 2 VCS roots
[08:55:22] :	 [Collecting changes in 2 VCS roots] VCS Root details
[08:55:22] :		 [VCS Root details] "phab-bitcoin-abc-staging-ro" {instance id=18, parent internal id=14, parent id=PhabBitcoinAbcStagingRo, description: "https://reviews.bitcoinabc.org/source/bitcoin-abc-staging.git#refs/heads/master"}
[08:55:22] :		 [VCS Root details] "abc-infrastructure" {instance id=7, parent internal id=7, parent id=AutomatedDeployments_BitcoinAbcDeveloperTools_AbcInfrastructure, description: "ssh://vcs@reviews.bitcoinabc.org:2221/source/infrastructure.git#refs/heads/master"}
[08:55:25] : The build is removed from the queue to be prepared for the start
[08:55:30]W: This build has not been started because some of the builds it depends on failed to start
[08:55:30]E: Snapshot dependency "Automated Deployments / Bitcoin ABC Infra / Bitcoin-ABC Infra Checkout" failed
[08:55:30] : Build finished
Harbormaster completed remote builds in B18442: Diff 32647.Mar 10 2022, 08:55
teamcity added a comment.Mar 10 2022, 08:55
Build Bitcoin ABC Diffs / Diff Testing (lint-circular-dependencies) failed.

Tail of the build log:

Build 'Bitcoin ABC Diffs / Diff Testing' #43697, branch 'refs/tags/phabricator/diff/32647'
Triggered 2022-03-10 08:55:23 by 'Phabricator Staging (phabricator-staging)'
Started 2022-03-10 08:55:33 on agent 'buildagent1'
Finished 2022-03-10 08:55:39 with status FAILURE 'Snapshot dependency failed to start: Automated Deployments / Bitcoin ABC Infra / Bitcoin-ABC Infra Checkout'
VCS revisions: 'PhabBitcoinAbcStagingRo' (Git, instance id 18): 'N/A' (checkout rules: '+:. => ./bitcoin-abc')
TeamCity URL https://build.bitcoinabc.org/viewLog.html?buildId=364322&buildTypeId=BitcoinABC_BitcoinAbcStaging 
TeamCity server version is 2019.2.4 (build 72059), server timezone: GMT (UTC)

[08:55:23]E: bt15 (running for 15s)
[08:55:23]i: TeamCity server version is 2019.2.4 (build 72059)
[08:55:23] : Collecting changes in 2 VCS roots (7s)
[08:55:23] :	 [Collecting changes in 2 VCS roots] VCS Root details
[08:55:23] :		 [VCS Root details] "phab-bitcoin-abc-staging-ro" {instance id=18, parent internal id=14, parent id=PhabBitcoinAbcStagingRo, description: "https://reviews.bitcoinabc.org/source/bitcoin-abc-staging.git#refs/heads/master"}
[08:55:23] :		 [VCS Root details] "abc-infrastructure" {instance id=7, parent internal id=7, parent id=AutomatedDeployments_BitcoinAbcDeveloperTools_AbcInfrastructure, description: "ssh://vcs@reviews.bitcoinabc.org:2221/source/infrastructure.git#refs/heads/master"}
[08:55:33] : The build is removed from the queue to be prepared for the start
[08:55:39]W: This build has not been started because some of the builds it depends on failed to start
[08:55:39]E: Snapshot dependency "Automated Deployments / Bitcoin ABC Infra / Bitcoin-ABC Infra Checkout" failed
[08:55:39] : Build finished
teamcity added a comment.Mar 10 2022, 08:55
Build Bitcoin ABC Diffs / Diff Testing (build-clang-tidy) failed.

Tail of the build log:

Build 'Bitcoin ABC Diffs / Diff Testing' #43698, branch 'refs/tags/phabricator/diff/32647'
Triggered 2022-03-10 08:55:23 by 'Phabricator Staging (phabricator-staging)'
Started 2022-03-10 08:55:40 on agent 'buildagent2'
Finished 2022-03-10 08:55:45 with status FAILURE 'Snapshot dependency failed to start: Automated Deployments / Bitcoin ABC Infra / Bitcoin-ABC Infra Checkout'
VCS revisions: 'PhabBitcoinAbcStagingRo' (Git, instance id 18): 'N/A' (checkout rules: '+:. => ./bitcoin-abc')
TeamCity URL https://build.bitcoinabc.org/viewLog.html?buildId=364314&buildTypeId=BitcoinABC_BitcoinAbcStaging 
TeamCity server version is 2019.2.4 (build 72059), server timezone: GMT (UTC)

[08:55:23]E: bt15 (running for 22s)
[08:55:23]i: TeamCity server version is 2019.2.4 (build 72059)
[08:55:23] : Collecting changes in 2 VCS roots (15s)
[08:55:23] :	 [Collecting changes in 2 VCS roots] VCS Root details
[08:55:23] :		 [VCS Root details] "phab-bitcoin-abc-staging-ro" {instance id=18, parent internal id=14, parent id=PhabBitcoinAbcStagingRo, description: "https://reviews.bitcoinabc.org/source/bitcoin-abc-staging.git#refs/heads/master"}
[08:55:23] :		 [VCS Root details] "abc-infrastructure" {instance id=7, parent internal id=7, parent id=AutomatedDeployments_BitcoinAbcDeveloperTools_AbcInfrastructure, description: "ssh://vcs@reviews.bitcoinabc.org:2221/source/infrastructure.git#refs/heads/master"}
[08:55:23]i:	 [Collecting changes in 2 VCS roots] Loading current repository state for VCS root 'abc-infrastructure' (15s)
[08:55:40] : The build is removed from the queue to be prepared for the start
[08:55:45]W: This build has not been started because some of the builds it depends on failed to start
[08:55:45]E: Snapshot dependency "Automated Deployments / Bitcoin ABC Infra / Bitcoin-ABC Infra Checkout" failed
[08:55:46] : Build finished
teamcity added a comment.Mar 10 2022, 08:56
Build Bitcoin ABC Diffs / Diff Testing (build-debug) failed.

Tail of the build log:

Build 'Bitcoin ABC Diffs / Diff Testing' #43699, branch 'refs/tags/phabricator/diff/32647'
Triggered 2022-03-10 08:55:23 by 'Phabricator Staging (phabricator-staging)'
Started 2022-03-10 08:55:56 on agent 'buildagent1'
Finished 2022-03-10 08:56:00 with status FAILURE 'Snapshot dependency failed to start: Automated Deployments / Bitcoin ABC Infra / Bitcoin-ABC Infra Checkout'
VCS revisions: 'PhabBitcoinAbcStagingRo' (Git, instance id 18): 'N/A' (checkout rules: '+:. => ./bitcoin-abc')
TeamCity URL https://build.bitcoinabc.org/viewLog.html?buildId=364316&buildTypeId=BitcoinABC_BitcoinAbcStaging 
TeamCity server version is 2019.2.4 (build 72059), server timezone: GMT (UTC)

[08:55:23]E: bt15 (running for 37s)
[08:55:23]i: TeamCity server version is 2019.2.4 (build 72059)
[08:55:23] : Collecting changes in 2 VCS roots (31s)
[08:55:23] :	 [Collecting changes in 2 VCS roots] VCS Root details
[08:55:23] :		 [VCS Root details] "phab-bitcoin-abc-staging-ro" {instance id=18, parent internal id=14, parent id=PhabBitcoinAbcStagingRo, description: "https://reviews.bitcoinabc.org/source/bitcoin-abc-staging.git#refs/heads/master"}
[08:55:23] :		 [VCS Root details] "abc-infrastructure" {instance id=7, parent internal id=7, parent id=AutomatedDeployments_BitcoinAbcDeveloperTools_AbcInfrastructure, description: "ssh://vcs@reviews.bitcoinabc.org:2221/source/infrastructure.git#refs/heads/master"}
[08:55:23]i:	 [Collecting changes in 2 VCS roots] Loading current repository state for VCS root 'phab-bitcoin-abc-staging-ro' (15s)
[08:55:56] : The build is removed from the queue to be prepared for the start
[08:56:00]W: This build has not been started because some of the builds it depends on failed to start
[08:56:00]E: Snapshot dependency "Automated Deployments / Bitcoin ABC Infra / Bitcoin-ABC Infra Checkout" failed
[08:56:01] : Build finished
teamcity added a comment.Mar 10 2022, 08:56
Build Bitcoin ABC Diffs / Diff Testing (build-diff) failed.

Tail of the build log:

Build 'Bitcoin ABC Diffs / Diff Testing' #43700, branch 'refs/tags/phabricator/diff/32647'
Triggered 2022-03-10 08:55:23 by 'Phabricator Staging (phabricator-staging)'
Started 2022-03-10 08:56:00 on agent 'buildagent2'
Finished 2022-03-10 08:56:07 with status FAILURE 'Snapshot dependency failed to start: Automated Deployments / Bitcoin ABC Infra / Bitcoin-ABC Infra Checkout'
VCS revisions: 'PhabBitcoinAbcStagingRo' (Git, instance id 18): 'N/A' (checkout rules: '+:. => ./bitcoin-abc')
TeamCity URL https://build.bitcoinabc.org/viewLog.html?buildId=364318&buildTypeId=BitcoinABC_BitcoinAbcStaging 
TeamCity server version is 2019.2.4 (build 72059), server timezone: GMT (UTC)

[08:55:23]E: bt15 (running for 44s)
[08:55:23]i: TeamCity server version is 2019.2.4 (build 72059)
[08:55:23] : Collecting changes in 2 VCS roots (35s)
[08:55:23] :	 [Collecting changes in 2 VCS roots] VCS Root details
[08:55:23] :		 [VCS Root details] "phab-bitcoin-abc-staging-ro" {instance id=18, parent internal id=14, parent id=PhabBitcoinAbcStagingRo, description: "https://reviews.bitcoinabc.org/source/bitcoin-abc-staging.git#refs/heads/master"}
[08:55:23] :		 [VCS Root details] "abc-infrastructure" {instance id=7, parent internal id=7, parent id=AutomatedDeployments_BitcoinAbcDeveloperTools_AbcInfrastructure, description: "ssh://vcs@reviews.bitcoinabc.org:2221/source/infrastructure.git#refs/heads/master"}
[08:55:23]i:	 [Collecting changes in 2 VCS roots] Loading current repository state for VCS root 'phab-bitcoin-abc-staging-ro' (31s)
[08:55:23]i:	 [Collecting changes in 2 VCS roots] Loading current repository state for VCS root 'abc-infrastructure' (35s)
[08:56:00] : The build is removed from the queue to be prepared for the start
[08:56:07]W: This build has not been started because some of the builds it depends on failed to start
[08:56:07]E: Snapshot dependency "Automated Deployments / Bitcoin ABC Infra / Bitcoin-ABC Infra Checkout" failed
[08:56:07] : Build finished
Fabien requested changes to this revision.Mar 10 2022, 09:48
Fabien added a subscriber: Fabien.

You need to run IBD with pruning enabled in your test plan

src/validation.cpp
2129 ↗(On Diff #32633)

Please clean this comment mess

This revision now requires changes to proceed.Mar 10 2022, 09:48
Fabien mentioned this in D11146: Add functional test for blockfilterindex in prune-mode.Mar 10 2022, 09:50
PiRK edited the test plan for this revision. (Show Details)Apr 1 2022, 12:58
PiRK abandoned this revision.Apr 4 2022, 08:04

replaced by D11299

Revision Contents
Changeset List

PathSize
src/
index/
49 lines
4 lines
2 lines
25 lines

Diff 32647

View Options

src/index/base.cpp

Loading...
View Options

src/init.cpp

Loading...
View Options

src/validation.h

Loading...
View Options

src/validation.cpp

Loading...