core#28589 (race fixes)
core#28590 (getchainstates return a list of chainstates)
core#28625 ( check that loading snapshot not matching AssumeUTXO parameters fails)
https://github.com/bitcoin/bitcoin/pull/28639/commits/fafde92f84fb7c245bc3c1cd946a32c891861e5e
core#28647 (Add assumeutxo test for wrong hash)
core#28652 (fail early if snapshot block hash doesn't match AssumeUTXO parameters)
core#28562 (self.no_op, self.wait_until)
core#28666 (assumeutxo file with unknown block hash)
core#28669 (check au file with changed outpoint index)
core#28685 (add tests for coin maleation)
core#29215 (spend coin from snapshot chainstate after loading)
https://github.com/bitcoin/bitcoin/pull/29726/commits/b7ba60f81a33db876f88b5f9af1e5025d679b5be (add coverage for -reindex and assumeutxo)
core#29354 (Assumeutxo with more than just coinbase transactions)
core#29394 (Add test to ensure failure when mempool not empty)
https://github.com/bitcoin/bitcoin/pull/29478/commits/2bc1ecfaa9b69a20388e913ec64967de2f506cd3 (Remove unnecessary sync_blocks in assumeutxo tests)
core#29478 (Add test for loadtxoutset when headers are not synced)
core#29370 (RPC test for fake nTx and nChainTX values, stale block CheckBlockIndex crash test & assumeutxo snapshot block CheckBlockIndex crash test)
core#29617 (test for coin_height > base_height & amount > money_supply)
https://github.com/bitcoin/bitcoin/pull/28685/commits/f6213929c519d0e615cacd3d6f479f1517be1662 (Check deserialized coins for out of range values)
core#30053 (coverage for "Couldn't open file..." error)
core#29973 (ensure failure when importing a snapshot twice)

test: add assumeutxo wallet test

initial commit: core#28838
https://github.com/bitcoin/bitcoin/pull/29354/commits/fa5cd66f0a47d1b759c93d01524ee4558432c0cc (Assumeutxo with more than just coinbase transactions)
https://github.com/bitcoin/bitcoin/pull/29478/commits/2bc1ecfaa9b69a20388e913ec64967de2f506cd3 (Remove unnecessary sync_blocks in assumeutxo tests)
https://github.com/bitcoin/bitcoin/pull/30678/commits/7e3dbe4180cbeb65e59b53d9fa98509e9189549d (functional test only)
https://github.com/bitcoin/bitcoin/pull/30678/commits/f20fe33e94c6752e5d2ed92511c0bf51a10716ee (Add basic balance coverage)
https://github.com/bitcoin/bitcoin/pull/31556/commits/bc43ecaf6dc0830a27296d3a29428814fed07bb1 (test for balance after snapshot completion)
https://github.com/bitcoin/bitcoin/pull/30909/commits/595edee169045b6735b76ff9721677f0e43f13e5 (import descriptors during background sync)

Test Plan

ninja all check-all

Diff Detail

Repository

rABC Bitcoin ABC

Lint

Lint Passed

Unit

No Test Coverage

Build Status

Buildable 32965
Build 65414: Build Diff	lint-circular-dependencies · build-without-wallet · build-diff · build-debug · build-clang · build-clang-tidy
Build 65413: arc lint + arc unit

Event Timeline

PiRK created this revision.Mon, Apr 14, 19:11

Herald added a reviewer: Restricted Project. · View Herald TranscriptMon, Apr 14, 19:11

PiRK requested review of this revision.Mon, Apr 14, 19:11

teamcity edited the summary of this revision. (Show Details)Mon, Apr 14, 19:11

Harbormaster completed remote builds in B32961: Diff 53478.Mon, Apr 14, 19:21

Build Bitcoin ABC Diffs / Diff Testing (build-debug) failed.

Failed tests logs:

====== Bitcoin ABC functional tests: feature_assumeutxo.py ======

------- Stdout: -------
2025-04-14T19:20:32.450000Z TestFramework (INFO): Initializing test directory /work/abc-ci-builds/build-debug/test/tmp/test_runner_₿₵_🏃_20250414_191514/feature_assumeutxo_285
2025-04-14T19:20:34.136000Z TestFramework (INFO): -- Testing assumeutxo + some indexes + pruning
2025-04-14T19:20:34.137000Z TestFramework (INFO): Creating a UTXO snapshot at height 299
2025-04-14T19:20:34.184000Z TestFramework (INFO): Test loading snapshot when headers are not synced
2025-04-14T19:20:35.919000Z TestFramework (INFO): Test bitcoind should fail when mempool not empty.
2025-04-14T19:20:37.126000Z TestFramework (INFO): Test different scenarios of loading invalid snapshot files
2025-04-14T19:20:37.126000Z TestFramework (INFO):   - snapshot file refering to a block that is not in the assumeutxo parameters
2025-04-14T19:20:37.128000Z TestFramework (INFO):   - snapshot file with wrong number of coins
2025-04-14T19:20:37.423000Z TestFramework (INFO):   - snapshot file with alternated UTXO data
2025-04-14T19:20:38.134000Z TestFramework (INFO): Test bitcoind should fail when file path is invalid.
2025-04-14T19:20:38.135000Z TestFramework (INFO): Loading snapshot into second node from /work/abc-ci-builds/build-debug/test/tmp/test_runner_₿₵_🏃_20250414_191514/feature_assumeutxo_285/node0/regtest/utxos.dat
2025-04-14T19:20:38.484000Z TestFramework (INFO): Submit a stale block that forked off the chain before the snapshot
2025-04-14T19:20:38.492000Z TestFramework (INFO): Submit a spending transaction for a snapshot chainstate coin to the mempool
2025-04-14T19:20:38.503000Z TestFramework (INFO): Restarting node to stop at height 359
2025-04-14T19:20:40.410000Z TestFramework (INFO): Checking that blocks are segmented on disk
2025-04-14T19:20:40.410000Z TestFramework (INFO): Restarted node before snapshot validation completed, reloading...
2025-04-14T19:20:41.540000Z TestFramework (ERROR): Unexpected exception caught during testing
Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 152, in main
    self._run_test_internal()
  File "/work/test/functional/test_framework/test_framework.py", line 142, in _run_test_internal
    self.run_test()
  File "/work/test/functional/feature_assumeutxo.py", line 413, in run_test
    n1.submitblock(snapshot_block)
  File "/work/test/functional/test_framework/coverage.py", line 46, in __call__
    return_val = self.auth_service_proxy_instance.__call__(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/test/functional/test_framework/authproxy.py", line 146, in __call__
    response, status = self._request(
                       ^^^^^^^^^^^^^^
  File "/work/test/functional/test_framework/authproxy.py", line 115, in _request
    return self._get_response()
           ^^^^^^^^^^^^^^^^^^^^
  File "/work/test/functional/test_framework/authproxy.py", line 187, in _get_response
    http_response = self.__conn.getresponse()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/http/client.py", line 1374, in getresponse
    response.begin()
  File "/usr/lib/python3.11/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
2025-04-14T19:20:41.592000Z TestFramework (INFO): Stopping nodes
[node 2] Cleaning up leftover process
[node 1] Cleaning up leftover process
[node 0] Cleaning up leftover process
------- Stderr: -------
Traceback (most recent call last):
  File "/work/test/functional/feature_assumeutxo.py", line 550, in <module>
    AssumeutxoTest().main()
  File "/work/test/functional/test_framework/test_framework.py", line 175, in main
    exit_code = self.shutdown()
                ^^^^^^^^^^^^^^^
  File "/work/test/functional/test_framework/test_framework.py", line 404, in shutdown
    self.stop_nodes()
  File "/work/test/functional/test_framework/test_framework.py", line 659, in stop_nodes
    node.stop_node(wait=wait, wait_until_stopped=False)
  File "/work/test/functional/test_framework/test_node.py", line 532, in stop_node
    self.stop(wait=wait)
  File "/work/test/functional/test_framework/coverage.py", line 46, in __call__
    return_val = self.auth_service_proxy_instance.__call__(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/test/functional/test_framework/authproxy.py", line 146, in __call__
    response, status = self._request(
                       ^^^^^^^^^^^^^^
  File "/work/test/functional/test_framework/authproxy.py", line 114, in _request
    self.__conn.request(method, path, postdata, headers)
  File "/usr/lib/python3.11/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.11/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.11/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.11/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/lib/python3.11/http/client.py", line 975, in send
    self.connect()
  File "/usr/lib/python3.11/http/client.py", line 941, in connect
    self.sock = self._create_connection(
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/socket.py", line 851, in create_connection
    raise exceptions[0]
  File "/usr/lib/python3.11/socket.py", line 836, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

Each failure log is accessible here:
Bitcoin ABC functional tests: feature_assumeutxo.py

Review suggestion:

check loadtxoutset against https://github.com/bitcoin/bitcoin/blob/faa30a4c566c5b720c7994c55f276352a119129f/src/rpc/blockchain.cpp

check feature_assumeutxo.py vs https://github.com/bitcoin/bitcoin/blob/b259b0e8d360726b062c4b0453d1cf5a68e1933f/test/functional/feature_assumeutxo.py

check wallet_assumeutxo.py vs https://github.com/bitcoin/bitcoin/blob/595edee169045b6735b76ff9721677f0e43f13e5/test/functional/wallet_assumeutxo.py

There are still additional tests and fixes to be included in the tests, but I wanted to limit the number of files changed in this diff. Following tests will depened on a change in format of the snapshot file after core#29612

fix release notes, investigate test failure on CI

move release notes to proper file, split MiniWallet determinism into its own diff D17933, fix file permissions and shebangs for tests

PiRK edited the summary of this revision. (Show Details)Tue, Apr 15, 07:05

Harbormaster completed remote builds in B32965: Diff 53482.Tue, Apr 15, 07:13

Build Bitcoin ABC Diffs / Diff Testing (build-debug) failed.

Failed tests logs:

====== Bitcoin ABC functional tests: feature_assumeutxo.py ======

------- Stdout: -------
2025-04-15T07:12:41.104000Z TestFramework (INFO): Initializing test directory /work/abc-ci-builds/build-debug/test/tmp/test_runner_₿₵_🏃_20250415_070720/feature_assumeutxo_285
2025-04-15T07:12:42.980000Z TestFramework (INFO): -- Testing assumeutxo + some indexes + pruning
2025-04-15T07:12:42.981000Z TestFramework (INFO): Creating a UTXO snapshot at height 299
2025-04-15T07:12:43.017000Z TestFramework (INFO): Test loading snapshot when headers are not synced
2025-04-15T07:12:44.738000Z TestFramework (INFO): Test bitcoind should fail when mempool not empty.
2025-04-15T07:12:45.943000Z TestFramework (INFO): Test different scenarios of loading invalid snapshot files
2025-04-15T07:12:45.943000Z TestFramework (INFO):   - snapshot file refering to a block that is not in the assumeutxo parameters
2025-04-15T07:12:45.946000Z TestFramework (INFO):   - snapshot file with wrong number of coins
2025-04-15T07:12:46.239000Z TestFramework (INFO):   - snapshot file with alternated UTXO data
2025-04-15T07:12:46.956000Z TestFramework (INFO): Test bitcoind should fail when file path is invalid.
2025-04-15T07:12:46.957000Z TestFramework (INFO): Loading snapshot into second node from /work/abc-ci-builds/build-debug/test/tmp/test_runner_₿₵_🏃_20250415_070720/feature_assumeutxo_285/node0/regtest/utxos.dat
2025-04-15T07:12:47.219000Z TestFramework (INFO): Submit a stale block that forked off the chain before the snapshot
2025-04-15T07:12:47.228000Z TestFramework (INFO): Submit a spending transaction for a snapshot chainstate coin to the mempool
2025-04-15T07:12:47.239000Z TestFramework (INFO): Restarting node to stop at height 359
2025-04-15T07:12:49.348000Z TestFramework (INFO): Checking that blocks are segmented on disk
2025-04-15T07:12:49.348000Z TestFramework (INFO): Restarted node before snapshot validation completed, reloading...
2025-04-15T07:12:50.225000Z TestFramework (ERROR): Unexpected exception caught during testing
Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 152, in main
    self._run_test_internal()
  File "/work/test/functional/test_framework/test_framework.py", line 142, in _run_test_internal
    self.run_test()
  File "/work/test/functional/feature_assumeutxo.py", line 412, in run_test
    n1.submitblock(snapshot_block)
  File "/work/test/functional/test_framework/coverage.py", line 46, in __call__
    return_val = self.auth_service_proxy_instance.__call__(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/test/functional/test_framework/authproxy.py", line 146, in __call__
    response, status = self._request(
                       ^^^^^^^^^^^^^^
  File "/work/test/functional/test_framework/authproxy.py", line 115, in _request
    return self._get_response()
           ^^^^^^^^^^^^^^^^^^^^
  File "/work/test/functional/test_framework/authproxy.py", line 187, in _get_response
    http_response = self.__conn.getresponse()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/http/client.py", line 1374, in getresponse
    response.begin()
  File "/usr/lib/python3.11/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
2025-04-15T07:12:50.276000Z TestFramework (INFO): Stopping nodes
[node 2] Cleaning up leftover process
[node 1] Cleaning up leftover process
[node 0] Cleaning up leftover process
------- Stderr: -------
Traceback (most recent call last):
  File "/work/test/functional/feature_assumeutxo.py", line 549, in <module>
    AssumeutxoTest().main()
  File "/work/test/functional/test_framework/test_framework.py", line 175, in main
    exit_code = self.shutdown()
                ^^^^^^^^^^^^^^^
  File "/work/test/functional/test_framework/test_framework.py", line 404, in shutdown
    self.stop_nodes()
  File "/work/test/functional/test_framework/test_framework.py", line 659, in stop_nodes
    node.stop_node(wait=wait, wait_until_stopped=False)
  File "/work/test/functional/test_framework/test_node.py", line 532, in stop_node
    self.stop(wait=wait)
  File "/work/test/functional/test_framework/coverage.py", line 46, in __call__
    return_val = self.auth_service_proxy_instance.__call__(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/test/functional/test_framework/authproxy.py", line 146, in __call__
    response, status = self._request(
                       ^^^^^^^^^^^^^^
  File "/work/test/functional/test_framework/authproxy.py", line 114, in _request
    self.__conn.request(method, path, postdata, headers)
  File "/usr/lib/python3.11/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.11/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.11/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.11/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/lib/python3.11/http/client.py", line 975, in send
    self.connect()
  File "/usr/lib/python3.11/http/client.py", line 941, in connect
    self.sock = self._create_connection(
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/socket.py", line 851, in create_connection
    raise exceptions[0]
  File "/usr/lib/python3.11/socket.py", line 836, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

Each failure log is accessible here:
Bitcoin ABC functional tests: feature_assumeutxo.py

PiRK added inline comments.Tue, Apr 15, 07:21

test/functional/feature_assumeutxo.py
130	Here and in the next case we differ from Core because for our utxo_snaphshot the first coin in the dump happens to be a coinbase coin, so "\x81" would be the correct value. "\x80" is the code for the same block height but no coinbase flag. "\x83" is the same coinbase flag but a different height (+1) `coin code = (height << 1) \| is_coinbase`
152	Here the offset used by the original commit does not work for us, so I early backported a fix (`offset = offset - 3`) from https://github.com/bitcoin/bitcoin/pull/31907/commits/e5ff4e416ecc8a51367022eb8a7a291f8cbc0c65 See comment in the commit message: In addition, the undocumented offset is set surprisingly high (39 bytes is well into the serialization of the amount which starts at offset 36). This would be after the core#29612 optimized the format of the snapshot, so for us it it offset 38 rather than 36
test/functional/wallet_assumeutxo.py
230–232	This issue is likely caused by missing wallet backports, and has no direct influence on assumeutxo operations