Page MenuHomePhabricator
Differential D16023

blockstorage: Adjust fastprune limit if block exceeds blockfile size
ClosedPublic
Actions

Authored by PiRK on Apr 19 2024, 11:37.

Details

Reviewers
Fabien
Group Reviewers
Restricted Project
Commits
rABCb5e5bf5848a0: blockstorage: Adjust fastprune limit if block exceeds blockfile size
Summary

If the added block exceeds the blockfile size in test-only
-fastprune mode, the node would get stuck in an infinite loop and
run out of memory.

Avoid this by raising the blockfile size to the size of the added block
in this situation.

Co-authored-by: TheCharlatan <seb.kung@gmail.com>

This is a backport of core#27191

Depends on D16028

Note that this fix does not address the non-test issue of blocks larger than 128 MiB. But at least now we get a meaningful assertion failure rather than an OOM crash.

Test Plan

ninja all check-all

Check that without the change in blockstorage.cpp the new test slowly consumes all the memory and the test eventually fails.

Diff Detail

Repository
rABC Bitcoin ABC
Branch
pr27746
Lint
Lint Passed
SeverityLocationCodeMessage
Auto-Fixtest/functional/feature_fastprune.py:1ISORTSorting Python imports
Unit
No Test Coverage
Build Status
Buildable 28665
Build 56871: Build Difflint-circular-dependencies · build-without-wallet · build-clang-tidy · build-diff · build-clang · build-debug
Build 56870: arc lint + arc unit

Event Timeline

PiRK created this revision.Apr 19 2024, 11:37
Herald added a reviewer: Restricted Project. · View Herald TranscriptApr 19 2024, 11:37
teamcity edited the summary of this revision. (Show Details)Apr 19 2024, 11:37
Harbormaster completed remote builds in B28665: Diff 47304.Apr 19 2024, 11:48
teamcity added a comment.Apr 19 2024, 11:48
Build Bitcoin ABC Diffs / Diff Testing (build-without-wallet) failed.

Failed tests logs:

====== Bitcoin ABC functional tests: feature_fastprune.py ======

------- Stdout: -------
2024-04-19T11:46:52.860000Z TestFramework (INFO): Initializing test directory /work/abc-ci-builds/build-without-wallet/test/tmp/test_runner_₿₵_  _20240419_114357/feature_fastprune_263
2024-04-19T11:46:53.133000Z TestFramework (INFO): ensure that large blocks don't crash or freeze in -fastprune
2024-04-19T11:46:53.133000Z TestFramework (ERROR): Unexpected exception caught during testing
Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 147, in main
    self._run_test_internal()
  File "/work/test/functional/test_framework/test_framework.py", line 137, in _run_test_internal
    self.run_test()
  File "/work/test/functional/feature_fastprune.py", line 24, in run_test
    tx = wallet.create_self_transfer()["tx"]
  File "/work/test/functional/test_framework/wallet.py", line 160, in create_self_transfer
    utxo_to_spend = utxo_to_spend or self.get_utxo()
  File "/work/test/functional/test_framework/wallet.py", line 127, in get_utxo
    index = self._utxos.index(next(utxo_filter))
StopIteration
2024-04-19T11:46:53.184000Z TestFramework (INFO): Stopping nodes
2024-04-19T11:46:53.385000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/build-without-wallet/test/tmp/test_runner_₿₵_  _20240419_114357/feature_fastprune_263
2024-04-19T11:46:53.385000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/build-without-wallet/test/tmp/test_runner_₿₵_  _20240419_114357/feature_fastprune_263/test_framework.log
2024-04-19T11:46:53.386000Z TestFramework (ERROR): 
2024-04-19T11:46:53.386000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/build-without-wallet/test/tmp/test_runner_₿₵_  _20240419_114357/feature_fastprune_263' to consolidate all logs
2024-04-19T11:46:53.386000Z TestFramework (ERROR): 
2024-04-19T11:46:53.386000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-04-19T11:46:53.386000Z TestFramework (ERROR): https://github.com/Bitcoin-ABC/bitcoin-abc/issues
2024-04-19T11:46:53.386000Z TestFramework (ERROR):

Each failure log is accessible here:
Bitcoin ABC functional tests: feature_fastprune.py

teamcity added a comment.Apr 19 2024, 11:51
Build Bitcoin ABC Diffs / Diff Testing (build-debug) failed.

Failed tests logs:

====== Bitcoin ABC functional tests: feature_fastprune.py ======

------- Stdout: -------
2024-04-19T11:50:20.182000Z TestFramework (INFO): Initializing test directory /work/abc-ci-builds/build-debug/test/tmp/test_runner_₿₵_  _20240419_114236/feature_fastprune_263
2024-04-19T11:50:20.765000Z TestFramework (INFO): ensure that large blocks don't crash or freeze in -fastprune
2024-04-19T11:50:20.765000Z TestFramework (ERROR): Unexpected exception caught during testing
Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 147, in main
    self._run_test_internal()
  File "/work/test/functional/test_framework/test_framework.py", line 137, in _run_test_internal
    self.run_test()
  File "/work/test/functional/feature_fastprune.py", line 24, in run_test
    tx = wallet.create_self_transfer()["tx"]
  File "/work/test/functional/test_framework/wallet.py", line 160, in create_self_transfer
    utxo_to_spend = utxo_to_spend or self.get_utxo()
  File "/work/test/functional/test_framework/wallet.py", line 127, in get_utxo
    index = self._utxos.index(next(utxo_filter))
StopIteration
2024-04-19T11:50:20.816000Z TestFramework (INFO): Stopping nodes
2024-04-19T11:50:20.918000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/build-debug/test/tmp/test_runner_₿₵_  _20240419_114236/feature_fastprune_263
2024-04-19T11:50:20.918000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/build-debug/test/tmp/test_runner_₿₵_  _20240419_114236/feature_fastprune_263/test_framework.log
2024-04-19T11:50:20.918000Z TestFramework (ERROR): 
2024-04-19T11:50:20.918000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/build-debug/test/tmp/test_runner_₿₵_  _20240419_114236/feature_fastprune_263' to consolidate all logs
2024-04-19T11:50:20.918000Z TestFramework (ERROR): 
2024-04-19T11:50:20.918000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-04-19T11:50:20.918000Z TestFramework (ERROR): https://github.com/Bitcoin-ABC/bitcoin-abc/issues
2024-04-19T11:50:20.918000Z TestFramework (ERROR):

Each failure log is accessible here:
Bitcoin ABC functional tests: feature_fastprune.py

teamcity added a comment.Apr 19 2024, 11:56
Build Bitcoin ABC Diffs / Diff Testing (build-diff) failed.

Failed tests logs:

====== Bitcoin ABC functional tests: feature_fastprune.py ======

------- Stdout: -------
2024-04-19T11:49:16.716000Z TestFramework (INFO): Initializing test directory /work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_  _20240419_114450/feature_fastprune_263
2024-04-19T11:49:17.017000Z TestFramework (INFO): ensure that large blocks don't crash or freeze in -fastprune
2024-04-19T11:49:17.017000Z TestFramework (ERROR): Unexpected exception caught during testing
Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 147, in main
    self._run_test_internal()
  File "/work/test/functional/test_framework/test_framework.py", line 137, in _run_test_internal
    self.run_test()
  File "/work/test/functional/feature_fastprune.py", line 24, in run_test
    tx = wallet.create_self_transfer()["tx"]
  File "/work/test/functional/test_framework/wallet.py", line 160, in create_self_transfer
    utxo_to_spend = utxo_to_spend or self.get_utxo()
  File "/work/test/functional/test_framework/wallet.py", line 127, in get_utxo
    index = self._utxos.index(next(utxo_filter))
StopIteration
2024-04-19T11:49:17.068000Z TestFramework (INFO): Stopping nodes
2024-04-19T11:49:17.169000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_  _20240419_114450/feature_fastprune_263
2024-04-19T11:49:17.169000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_  _20240419_114450/feature_fastprune_263/test_framework.log
2024-04-19T11:49:17.169000Z TestFramework (ERROR): 
2024-04-19T11:49:17.170000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_  _20240419_114450/feature_fastprune_263' to consolidate all logs
2024-04-19T11:49:17.170000Z TestFramework (ERROR): 
2024-04-19T11:49:17.170000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-04-19T11:49:17.170000Z TestFramework (ERROR): https://github.com/Bitcoin-ABC/bitcoin-abc/issues
2024-04-19T11:49:17.170000Z TestFramework (ERROR):
====== Bitcoin ABC functional tests with the next upgrade activated: feature_fastprune.py ======

------- Stdout: -------
2024-04-19T11:53:33.512000Z TestFramework (INFO): Initializing test directory /work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_  _20240419_114918/feature_fastprune_263
2024-04-19T11:53:33.814000Z TestFramework (INFO): ensure that large blocks don't crash or freeze in -fastprune
2024-04-19T11:53:33.814000Z TestFramework (ERROR): Unexpected exception caught during testing
Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 147, in main
    self._run_test_internal()
  File "/work/test/functional/test_framework/test_framework.py", line 137, in _run_test_internal
    self.run_test()
  File "/work/test/functional/feature_fastprune.py", line 24, in run_test
    tx = wallet.create_self_transfer()["tx"]
  File "/work/test/functional/test_framework/wallet.py", line 160, in create_self_transfer
    utxo_to_spend = utxo_to_spend or self.get_utxo()
  File "/work/test/functional/test_framework/wallet.py", line 127, in get_utxo
    index = self._utxos.index(next(utxo_filter))
StopIteration
2024-04-19T11:53:33.865000Z TestFramework (INFO): Stopping nodes
2024-04-19T11:53:33.966000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_  _20240419_114918/feature_fastprune_263
2024-04-19T11:53:33.966000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_  _20240419_114918/feature_fastprune_263/test_framework.log
2024-04-19T11:53:33.966000Z TestFramework (ERROR): 
2024-04-19T11:53:33.967000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_  _20240419_114918/feature_fastprune_263' to consolidate all logs
2024-04-19T11:53:33.967000Z TestFramework (ERROR): 
2024-04-19T11:53:33.967000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-04-19T11:53:33.967000Z TestFramework (ERROR): https://github.com/Bitcoin-ABC/bitcoin-abc/issues
2024-04-19T11:53:33.967000Z TestFramework (ERROR):

Each failure log is accessible here:
Bitcoin ABC functional tests: feature_fastprune.py
Bitcoin ABC functional tests with the next upgrade activated: feature_fastprune.py

PiRK updated this revision to Diff 47315.Apr 19 2024, 13:50

rebase after backporting all the pieces needed for the new test, squash with https://github.com/bitcoin/bitcoin/pull/27553

PiRK published this revision for review.Apr 19 2024, 13:50
PiRK edited the summary of this revision. (Show Details)Apr 19 2024, 13:54
PiRK edited the test plan for this revision. (Show Details)
PiRK added a parent revision: D16028: test: simplify tests by using the pre-mined chain.
Fabien accepted this revision.Apr 19 2024, 14:22
This revision is now accepted and ready to land.Apr 19 2024, 14:22
Harbormaster completed remote builds in B28674: Diff 47315.Apr 19 2024, 14:23
Closed by commit rABCb5e5bf5848a0: blockstorage: Adjust fastprune limit if block exceeds blockfile size (authored by Martin Zumsande <mzumsande@gmail.com>, committed by PiRK). · Explain WhyApr 19 2024, 14:47
This revision was automatically updated to reflect the committed changes.
PiRK added a commit: rABCb5e5bf5848a0: blockstorage: Adjust fastprune limit if block exceeds blockfile size.

Revision Contents
Changeset List

PathSize
src/
node/
19 lines
test/
functional/
41 lines
test_framework/
3 lines

Diff 47304

View Options

src/node/blockstorage.cpp

Loading...
View Options

test/functional/feature_fastprune.py

Loading...
View Options

test/functional/test_framework/blocktools.py

Loading...