Page MenuHomePhabricator

[chronik] add a block-headers endpoint
ClosedPublic

Authored by PiRK on Jul 23 2024, 15:49.

Details

Reviewers
Fabien
Group Reviewers
Restricted Project
Commits
rABCe2a322504024: [chronik] add a block-headers endpoint
Summary

This adds another endpoint to fetch a range of block headers.

The behavior is the same as the "blocks" endpoint wrt boundary checks:

  • when the end height is more than the number of blocks in the active chain return less blocks than requested
  • require 0 <= start_height <= end_height
  • start_height and end_height are inclusive

Depends on D16508

Test Plan

ninja all check-all

Diff Detail

Repository
rABC Bitcoin ABC
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

dedup code that checks for range boundary consistency

fix accidental deletion of code in block_txs

PiRK published this revision for review.Jul 23 2024, 16:11
PiRK edited the summary of this revision. (Show Details)

Failed tests logs:

====== Bitcoin ABC functional tests: chronik_block_txs.py ======

------- Stdout: -------
2024-07-23T16:18:21.382000Z TestFramework (INFO): Initializing test directory /work/abc-ci-builds/build-chronik/test/tmp/test_runner_โ‚ฟโ‚ต_๐Ÿƒ_20240723_161337/chronik_block_txs_186
2024-07-23T16:18:22.037000Z TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 147, in main
    self._run_test_internal()
  File "/work/test/functional/test_framework/test_framework.py", line 137, in _run_test_internal
    self.run_test()
  File "/work/test/functional/chronik_block_txs.py", line 62, in run_test
    chronik.block_txs("00" * 32, page=0, page_size=201).err(400).msg,
  File "/work/test/functional/test_framework/chronik/client.py", line 42, in err
    raise AssertionError(
AssertionError: Expected error response status 400, but got different error status 404, error: msg: "404: Block not found: 0000000000000000000000000000000000000000000000000000000000000000"

2024-07-23T16:18:22.088000Z TestFramework (INFO): Stopping nodes
2024-07-23T16:18:22.240000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/build-chronik/test/tmp/test_runner_โ‚ฟโ‚ต_๐Ÿƒ_20240723_161337/chronik_block_txs_186
2024-07-23T16:18:22.240000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/build-chronik/test/tmp/test_runner_โ‚ฟโ‚ต_๐Ÿƒ_20240723_161337/chronik_block_txs_186/test_framework.log
2024-07-23T16:18:22.240000Z TestFramework (ERROR): 
2024-07-23T16:18:22.240000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/build-chronik/test/tmp/test_runner_โ‚ฟโ‚ต_๐Ÿƒ_20240723_161337/chronik_block_txs_186' to consolidate all logs
2024-07-23T16:18:22.240000Z TestFramework (ERROR): 
2024-07-23T16:18:22.240000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-07-23T16:18:22.240000Z TestFramework (ERROR): https://github.com/Bitcoin-ABC/bitcoin-abc/issues
2024-07-23T16:18:22.240000Z TestFramework (ERROR):

Each failure log is accessible here:
Bitcoin ABC functional tests: chronik_block_txs.py

Failed tests logs:

====== Bitcoin ABC functional tests: chronik_block_txs.py ======

------- Stdout: -------
2024-07-23T16:18:27.584000Z TestFramework (INFO): Initializing test directory /work/abc-ci-builds/build-chronik-plugins/test/tmp/test_runner_โ‚ฟโ‚ต_๐Ÿƒ_20240723_161354/chronik_block_txs_186
2024-07-23T16:18:28.237000Z TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 147, in main
    self._run_test_internal()
  File "/work/test/functional/test_framework/test_framework.py", line 137, in _run_test_internal
    self.run_test()
  File "/work/test/functional/chronik_block_txs.py", line 62, in run_test
    chronik.block_txs("00" * 32, page=0, page_size=201).err(400).msg,
  File "/work/test/functional/test_framework/chronik/client.py", line 42, in err
    raise AssertionError(
AssertionError: Expected error response status 400, but got different error status 404, error: msg: "404: Block not found: 0000000000000000000000000000000000000000000000000000000000000000"

2024-07-23T16:18:28.288000Z TestFramework (INFO): Stopping nodes
2024-07-23T16:18:28.439000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/build-chronik-plugins/test/tmp/test_runner_โ‚ฟโ‚ต_๐Ÿƒ_20240723_161354/chronik_block_txs_186
2024-07-23T16:18:28.440000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/build-chronik-plugins/test/tmp/test_runner_โ‚ฟโ‚ต_๐Ÿƒ_20240723_161354/chronik_block_txs_186/test_framework.log
2024-07-23T16:18:28.440000Z TestFramework (ERROR): 
2024-07-23T16:18:28.440000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/build-chronik-plugins/test/tmp/test_runner_โ‚ฟโ‚ต_๐Ÿƒ_20240723_161354/chronik_block_txs_186' to consolidate all logs
2024-07-23T16:18:28.440000Z TestFramework (ERROR): 
2024-07-23T16:18:28.440000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-07-23T16:18:28.440000Z TestFramework (ERROR): https://github.com/Bitcoin-ABC/bitcoin-abc/issues
2024-07-23T16:18:28.440000Z TestFramework (ERROR):

Each failure log is accessible here:
Bitcoin ABC functional tests: chronik_block_txs.py

Tail of the build log:

  File "/work/test/functional/setup_scripts/../test_framework/util.py", line 296, in wait_until_helper
    raise AssertionError(
AssertionError: Predicate ''''
            self.wait_until(lambda: is_finalblock(next_blockhash))
''' not true after 60.0 seconds
2024-07-23T16:47:17.651000Z TestFramework (INFO): Stopping nodes
2024-07-23T16:47:17.905000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_โ‚ฟโ‚ต_๐Ÿƒ_20240723_164604/setup_scripts/chronik-client_websocket_0
2024-07-23T16:47:17.905000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_โ‚ฟโ‚ต_๐Ÿƒ_20240723_164604/setup_scripts/chronik-client_websocket_0/test_framework.log
2024-07-23T16:47:17.905000Z TestFramework (ERROR): 
2024-07-23T16:47:17.906000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_โ‚ฟโ‚ต_๐Ÿƒ_20240723_164604/setup_scripts/chronik-client_websocket_0' to consolidate all logs
2024-07-23T16:47:17.906000Z TestFramework (ERROR): 
2024-07-23T16:47:17.906000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-07-23T16:47:17.907000Z TestFramework (ERROR): https://github.com/Bitcoin-ABC/bitcoin-abc/issues
2024-07-23T16:47:17.907000Z TestFramework (ERROR): 
Running Unit Tests for Test Framework Modules
setup_scripts/chronik-client_websocket.py started
setup_scripts/chronik-client_websocket.py failed, Duration: 73 s

stdout:

stderr:


TEST                                      | STATUS    | DURATION

setup_scripts/chronik-client_websocket.py | โœ– Failed  | 73 s

ALL                                       | โœ– Failed  | 73 s (accumulated) 
Runtime: 73 s

Test runner for chronik-client_websocket completed with code 1
-----------------------|---------|----------|---------|---------|-----------------------------------
File                   | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s                 
-----------------------|---------|----------|---------|---------|-----------------------------------
All files              |   28.35 |     8.64 |   24.57 |   28.35 |                                   
 chronik-client        |     100 |      100 |     100 |     100 |                                   
  index.ts             |     100 |      100 |     100 |     100 |                                   
 chronik-client/proto  |    21.6 |     6.07 |    13.9 |   21.74 |                                   
  chronik.ts           |     6.1 |     0.83 |    2.54 |    6.09 | ...,3978-3985,3990-4027,4031-4036 
  chronikNode.ts       |   33.03 |    10.56 |    23.6 |   33.28 | ...,4991-5030,5038-5111,5146-5151 
 chronik-client/src    |   65.77 |    42.39 |   63.41 |   65.38 |                                   
  ChronikClient.ts     |    4.24 |        0 |       0 |    4.29 | 33-163,178-222,290-692            
  ChronikClientNode.ts |   90.84 |     72.8 |   96.38 |   90.84 | ...,1068,1078,1103,1115,1121,1127 
  failoverProxy.ts     |   75.22 |    51.61 |   62.06 |   74.52 | ...67,275-285,294,301,305,310,314 
  hex.ts               |   89.47 |       50 |      75 |   87.87 | 58,66-68                          
  validation.ts        |   93.33 |    81.81 |     100 |   92.59 | 33,39                             
-----------------------|---------|----------|---------|---------|-----------------------------------

##teamcity[blockOpened name='Code Coverage Summary']
##teamcity[buildStatisticValue key='CodeCoverageAbsBCovered' value='1188']
##teamcity[buildStatisticValue key='CodeCoverageAbsBTotal' value='4189']
##teamcity[buildStatisticValue key='CodeCoverageAbsRCovered' value='337']
##teamcity[buildStatisticValue key='CodeCoverageAbsRTotal' value='3897']
##teamcity[buildStatisticValue key='CodeCoverageAbsMCovered' value='187']
##teamcity[buildStatisticValue key='CodeCoverageAbsMTotal' value='761']
##teamcity[buildStatisticValue key='CodeCoverageAbsLCovered' value='1176']
##teamcity[buildStatisticValue key='CodeCoverageAbsLTotal' value='4148']
##teamcity[blockClosed name='Code Coverage Summary']
mv: cannot stat 'test_results/chronik-client-integration-tests-junit.xml': No such file or directory
Build chronik-client-integration-tests failed with exit code 1
Fabien requested changes to this revision.Jul 23 2024, 18:59
Fabien added a subscriber: Fabien.

back to your queue

This revision now requires changes to proceed.Jul 23 2024, 18:59

This is weird. It looks intermittent

Fabien requested changes to this revision.Jul 24 2024, 12:24

The deduplication was correct, you just called the boundary check later in the code flow which changed the behavior.

This revision now requires changes to proceed.Jul 24 2024, 12:24

rededuplicate boundary checks, keep them early in the function this time

Tail of the build log:

  File "/work/test/functional/setup_scripts/../test_framework/util.py", line 296, in wait_until_helper
    raise AssertionError(
AssertionError: Predicate ''''
            self.wait_until(lambda: is_finalblock(next_blockhash))
''' not true after 60.0 seconds
2024-07-24T13:42:42.037000Z TestFramework (INFO): Stopping nodes
2024-07-24T13:42:42.242000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_โ‚ฟโ‚ต_๐Ÿƒ_20240724_134129/setup_scripts/chronik-client_websocket_0
2024-07-24T13:42:42.242000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_โ‚ฟโ‚ต_๐Ÿƒ_20240724_134129/setup_scripts/chronik-client_websocket_0/test_framework.log
2024-07-24T13:42:42.243000Z TestFramework (ERROR): 
2024-07-24T13:42:42.243000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_โ‚ฟโ‚ต_๐Ÿƒ_20240724_134129/setup_scripts/chronik-client_websocket_0' to consolidate all logs
2024-07-24T13:42:42.244000Z TestFramework (ERROR): 
2024-07-24T13:42:42.244000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-07-24T13:42:42.244000Z TestFramework (ERROR): https://github.com/Bitcoin-ABC/bitcoin-abc/issues
2024-07-24T13:42:42.244000Z TestFramework (ERROR): 
Running Unit Tests for Test Framework Modules
setup_scripts/chronik-client_websocket.py started
setup_scripts/chronik-client_websocket.py failed, Duration: 73 s

stdout:

stderr:


TEST                                      | STATUS    | DURATION

setup_scripts/chronik-client_websocket.py | โœ– Failed  | 73 s

ALL                                       | โœ– Failed  | 73 s (accumulated) 
Runtime: 73 s

Test runner for chronik-client_websocket completed with code 1
-----------------------|---------|----------|---------|---------|-----------------------------------
File                   | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s                 
-----------------------|---------|----------|---------|---------|-----------------------------------
All files              |   28.35 |     8.64 |   24.57 |   28.35 |                                   
 chronik-client        |     100 |      100 |     100 |     100 |                                   
  index.ts             |     100 |      100 |     100 |     100 |                                   
 chronik-client/proto  |    21.6 |     6.07 |    13.9 |   21.74 |                                   
  chronik.ts           |     6.1 |     0.83 |    2.54 |    6.09 | ...,3978-3985,3990-4027,4031-4036 
  chronikNode.ts       |   33.03 |    10.56 |    23.6 |   33.28 | ...,4991-5030,5038-5111,5146-5151 
 chronik-client/src    |   65.77 |    42.39 |   63.41 |   65.38 |                                   
  ChronikClient.ts     |    4.24 |        0 |       0 |    4.29 | 33-163,178-222,290-692            
  ChronikClientNode.ts |   90.84 |     72.8 |   96.38 |   90.84 | ...,1068,1078,1103,1115,1121,1127 
  failoverProxy.ts     |   75.22 |    51.61 |   62.06 |   74.52 | ...67,275-285,294,301,305,310,314 
  hex.ts               |   89.47 |       50 |      75 |   87.87 | 58,66-68                          
  validation.ts        |   93.33 |    81.81 |     100 |   92.59 | 33,39                             
-----------------------|---------|----------|---------|---------|-----------------------------------

##teamcity[blockOpened name='Code Coverage Summary']
##teamcity[buildStatisticValue key='CodeCoverageAbsBCovered' value='1188']
##teamcity[buildStatisticValue key='CodeCoverageAbsBTotal' value='4189']
##teamcity[buildStatisticValue key='CodeCoverageAbsRCovered' value='337']
##teamcity[buildStatisticValue key='CodeCoverageAbsRTotal' value='3897']
##teamcity[buildStatisticValue key='CodeCoverageAbsMCovered' value='187']
##teamcity[buildStatisticValue key='CodeCoverageAbsMTotal' value='761']
##teamcity[buildStatisticValue key='CodeCoverageAbsLCovered' value='1176']
##teamcity[buildStatisticValue key='CodeCoverageAbsLTotal' value='4148']
##teamcity[blockClosed name='Code Coverage Summary']
mv: cannot stat 'test_results/chronik-client-integration-tests-junit.xml': No such file or directory
Build chronik-client-integration-tests failed with exit code 1

I'm wondering if the way each individual CBlockIndex is fetched with the cs_main lock could be an issue. I guess there could be more efficient ways to do it, but mostly I wonder if there is a risk of returning an inconsistent list of headers if a reorg happens between two locks.

@bot chronik-client-integration-test

The build failed due to an unexpected infrastructure outage. The administrators have been notified to investigate. Sorry for the inconvenience.

The setup_scripts/chronik-client_websocket.py failure above seems unrelated to me. It still is a bit suspicious, as I haven't seen an unrelated chronik ws related failure in a while.

rebase and shorten chronik_bridge::ffi -> ffi

This revision is now accepted and ready to land.Jul 25 2024, 14:08

hold cs_main while fetching all the headers instead of holding and releasing it for each individual header, to avoid getting an inconsistent chain

chronik/chronik-bridge/src/ffi.rs
129โ€“134 โ†—(On Diff #48951)
chronik/chronik-cpp/chronik_bridge.cpp
215โ€“216 โ†—(On Diff #48951)

In this diff this check is redundant with the chech already done at the callsite, but I plan to reuse this function in a future diff where the check will be more relaxed (no limit of 500 blocks)

rename API (headers -> block-headers, pb.Header -> pb.BlockHeaders)

handle_header_range -> handle_block_headers

PiRK retitled this revision from [chronik] add a headers endpoint to [chronik] add a block-headers endpoint.Aug 1 2024, 09:12

Tail of the build log:

  File "/work/abc-ci-builds/ecash-lib-integration-tests/test/functional/test_runner.py", line 361, in main
    os.makedirs(tmpdir)
  File "/usr/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/work/abc-ci-builds/ecash-lib-integration-tests/test/tmp/test_runner_โ‚ฟโ‚ต_๐Ÿƒ_20240801_104729'
Test runner completed with code 1
----------------------------|---------|----------|---------|---------|------------------------------
File                        | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s            
----------------------------|---------|----------|---------|---------|------------------------------
All files                   |   58.73 |    49.01 |   56.39 |   58.66 |                              
 ecash-lib                  |       0 |        0 |       0 |       0 |                              
  eslint.config.js          |       0 |        0 |       0 |       0 |                              
 ecash-lib/src              |   73.12 |    56.58 |   73.07 |   72.88 |                              
  ecc.ts                    |   57.14 |    83.33 |      40 |   57.14 | 23-31                        
  hash.ts                   |   88.88 |    83.33 |      80 |   88.88 | 14                           
  index.ts                  |       0 |        0 |       0 |       0 |                              
  indexBrowser.ts           |       0 |        0 |       0 |       0 |                              
  indexNodeJs.ts            |       0 |        0 |       0 |       0 |                              
  initBrowser.ts            |       0 |      100 |       0 |       0 | 11-13                        
  initNodeJs.ts             |     100 |      100 |     100 |     100 |                              
  op.ts                     |   49.15 |     50.9 |      80 |   49.15 | ...89,92,104,107,109,117-122 
  opcode.ts                 |     100 |    83.33 |     100 |     100 | 1                            
  script.ts                 |   52.63 |    38.09 |      60 |    50.9 | ...4-135,146,156,166,188-199 
  sigHashType.ts            |   77.77 |       44 |   85.71 |   77.77 | 26-38                        
  tx.ts                     |   93.47 |    79.16 |    90.9 |   93.18 | 123-125                      
  txBuilder.ts              |   52.74 |    46.42 |   66.66 |   51.72 | ...3-107,124-179,214,244-248 
  unsignedTx.ts             |    73.8 |    57.14 |   78.57 |   74.07 | ...3,151,159,184,192,198-201 
 ecash-lib/src/ffi          |   28.26 |    15.94 |   16.98 |   28.98 |                              
  ecash_lib_wasm_browser.js |       0 |        0 |       0 |       0 | 3-336                        
  ecash_lib_wasm_nodejs.js  |    61.9 |       55 |   39.13 |   62.75 | ...1,197-215,237,250-251,255 
 ecash-lib/src/io           |   59.55 |    60.29 |   70.58 |   58.77 |                              
  bytes.ts                  |     7.4 |    71.42 |    12.5 |     7.4 | 13-64                        
  hex.ts                    |   82.05 |     62.5 |      80 |   82.35 | 41-45,50,58                  
  int.ts                    |       0 |        0 |       0 |       0 |                              
  str.ts                    |   85.71 |    83.33 |   66.66 |   85.71 | 15                           
  varsize.ts                |      32 |    36.36 |   66.66 |      32 | 14-24,40-47                  
  writer.ts                 |       0 |        0 |       0 |       0 |                              
  writerbytes.ts            |   83.33 |    68.42 |     100 |   83.33 | 33,43,53,63,79               
  writerlength.ts           |     100 |    83.33 |     100 |     100 | 1                            
 ecash-lib/src/test         |   84.93 |    51.11 |   81.25 |   85.29 |                              
  testRunner.ts             |   84.93 |    51.11 |   81.25 |   85.29 | 69-71,83-85,108,119,162,185  
 ecash-lib/src/token        |   60.33 |    47.14 |   53.33 |   60.11 |                              
  alp.ts                    |   82.92 |    89.47 |   83.33 |   82.92 | 110-123,142                  
  common.ts                 |     100 |    83.33 |     100 |     100 | 1                            
  empp.ts                   |    92.3 |       75 |     100 |   91.66 | 12                           
  slp.ts                    |    28.2 |    13.51 |    7.69 |    28.2 | ...7,174-178,185-197,201-211 
----------------------------|---------|----------|---------|---------|------------------------------

##teamcity[blockOpened name='Code Coverage Summary']
##teamcity[buildStatisticValue key='CodeCoverageAbsBCovered' value='723']
##teamcity[buildStatisticValue key='CodeCoverageAbsBTotal' value='1231']
##teamcity[buildStatisticValue key='CodeCoverageAbsRCovered' value='224']
##teamcity[buildStatisticValue key='CodeCoverageAbsRTotal' value='457']
##teamcity[buildStatisticValue key='CodeCoverageAbsMCovered' value='119']
##teamcity[buildStatisticValue key='CodeCoverageAbsMTotal' value='211']
##teamcity[buildStatisticValue key='CodeCoverageAbsLCovered' value='704']
##teamcity[buildStatisticValue key='CodeCoverageAbsLTotal' value='1200']
##teamcity[blockClosed name='Code Coverage Summary']
mv: cannot stat 'test_results/ecash-lib-integration-tests-junit.xml': No such file or directory
Build ecash-lib-integration-tests failed with exit code 1
This revision was automatically updated to reflect the committed changes.