Page MenuHomePhabricator

[chronik] add a block-headers endpoint
ClosedPublic

Authored by PiRK on Jul 23 2024, 15:49.

Details

Reviewers
Fabien
Group Reviewers
Restricted Project
Commits
rABCe2a322504024: [chronik] add a block-headers endpoint
Summary

This adds another endpoint to fetch a range of block headers.

The behavior is the same as the "blocks" endpoint wrt boundary checks:

  • when the end height is more than the number of blocks in the active chain return less blocks than requested
  • require 0 <= start_height <= end_height
  • start_height and end_height are inclusive

Depends on D16508

Test Plan

ninja all check-all

Event Timeline

dedup code that checks for range boundary consistency

fix accidental deletion of code in block_txs

PiRK published this revision for review.Jul 23 2024, 16:11
PiRK edited the summary of this revision. (Show Details)

Failed tests logs:

====== Bitcoin ABC functional tests: chronik_block_txs.py ======

------- Stdout: -------
2024-07-23T16:18:21.382000Z TestFramework (INFO): Initializing test directory /work/abc-ci-builds/build-chronik/test/tmp/test_runner_₿₵_🏃_20240723_161337/chronik_block_txs_186
2024-07-23T16:18:22.037000Z TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 147, in main
    self._run_test_internal()
  File "/work/test/functional/test_framework/test_framework.py", line 137, in _run_test_internal
    self.run_test()
  File "/work/test/functional/chronik_block_txs.py", line 62, in run_test
    chronik.block_txs("00" * 32, page=0, page_size=201).err(400).msg,
  File "/work/test/functional/test_framework/chronik/client.py", line 42, in err
    raise AssertionError(
AssertionError: Expected error response status 400, but got different error status 404, error: msg: "404: Block not found: 0000000000000000000000000000000000000000000000000000000000000000"

2024-07-23T16:18:22.088000Z TestFramework (INFO): Stopping nodes
2024-07-23T16:18:22.240000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/build-chronik/test/tmp/test_runner_₿₵_🏃_20240723_161337/chronik_block_txs_186
2024-07-23T16:18:22.240000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/build-chronik/test/tmp/test_runner_₿₵_🏃_20240723_161337/chronik_block_txs_186/test_framework.log
2024-07-23T16:18:22.240000Z TestFramework (ERROR): 
2024-07-23T16:18:22.240000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/build-chronik/test/tmp/test_runner_₿₵_🏃_20240723_161337/chronik_block_txs_186' to consolidate all logs
2024-07-23T16:18:22.240000Z TestFramework (ERROR): 
2024-07-23T16:18:22.240000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-07-23T16:18:22.240000Z TestFramework (ERROR): https://github.com/Bitcoin-ABC/bitcoin-abc/issues
2024-07-23T16:18:22.240000Z TestFramework (ERROR):

Each failure log is accessible here:
Bitcoin ABC functional tests: chronik_block_txs.py

Failed tests logs:

====== Bitcoin ABC functional tests: chronik_block_txs.py ======

------- Stdout: -------
2024-07-23T16:18:27.584000Z TestFramework (INFO): Initializing test directory /work/abc-ci-builds/build-chronik-plugins/test/tmp/test_runner_₿₵_🏃_20240723_161354/chronik_block_txs_186
2024-07-23T16:18:28.237000Z TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 147, in main
    self._run_test_internal()
  File "/work/test/functional/test_framework/test_framework.py", line 137, in _run_test_internal
    self.run_test()
  File "/work/test/functional/chronik_block_txs.py", line 62, in run_test
    chronik.block_txs("00" * 32, page=0, page_size=201).err(400).msg,
  File "/work/test/functional/test_framework/chronik/client.py", line 42, in err
    raise AssertionError(
AssertionError: Expected error response status 400, but got different error status 404, error: msg: "404: Block not found: 0000000000000000000000000000000000000000000000000000000000000000"

2024-07-23T16:18:28.288000Z TestFramework (INFO): Stopping nodes
2024-07-23T16:18:28.439000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/build-chronik-plugins/test/tmp/test_runner_₿₵_🏃_20240723_161354/chronik_block_txs_186
2024-07-23T16:18:28.440000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/build-chronik-plugins/test/tmp/test_runner_₿₵_🏃_20240723_161354/chronik_block_txs_186/test_framework.log
2024-07-23T16:18:28.440000Z TestFramework (ERROR): 
2024-07-23T16:18:28.440000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/build-chronik-plugins/test/tmp/test_runner_₿₵_🏃_20240723_161354/chronik_block_txs_186' to consolidate all logs
2024-07-23T16:18:28.440000Z TestFramework (ERROR): 
2024-07-23T16:18:28.440000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-07-23T16:18:28.440000Z TestFramework (ERROR): https://github.com/Bitcoin-ABC/bitcoin-abc/issues
2024-07-23T16:18:28.440000Z TestFramework (ERROR):

Each failure log is accessible here:
Bitcoin ABC functional tests: chronik_block_txs.py

Tail of the build log:

  File "/work/test/functional/setup_scripts/../test_framework/util.py", line 296, in wait_until_helper
    raise AssertionError(
AssertionError: Predicate ''''
            self.wait_until(lambda: is_finalblock(next_blockhash))
''' not true after 60.0 seconds
2024-07-23T16:47:17.651000Z TestFramework (INFO): Stopping nodes
2024-07-23T16:47:17.905000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240723_164604/setup_scripts/chronik-client_websocket_0
2024-07-23T16:47:17.905000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240723_164604/setup_scripts/chronik-client_websocket_0/test_framework.log
2024-07-23T16:47:17.905000Z TestFramework (ERROR): 
2024-07-23T16:47:17.906000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240723_164604/setup_scripts/chronik-client_websocket_0' to consolidate all logs
2024-07-23T16:47:17.906000Z TestFramework (ERROR): 
2024-07-23T16:47:17.906000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-07-23T16:47:17.907000Z TestFramework (ERROR): https://github.com/Bitcoin-ABC/bitcoin-abc/issues
2024-07-23T16:47:17.907000Z TestFramework (ERROR): 
Running Unit Tests for Test Framework Modules
setup_scripts/chronik-client_websocket.py started
setup_scripts/chronik-client_websocket.py failed, Duration: 73 s

stdout:

stderr:


TEST                                      | STATUS    | DURATION

setup_scripts/chronik-client_websocket.py | ✖ Failed  | 73 s

ALL                                       | ✖ Failed  | 73 s (accumulated) 
Runtime: 73 s

Test runner for chronik-client_websocket completed with code 1
-----------------------|---------|----------|---------|---------|-----------------------------------
File                   | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s                 
-----------------------|---------|----------|---------|---------|-----------------------------------
All files              |   28.35 |     8.64 |   24.57 |   28.35 |                                   
 chronik-client        |     100 |      100 |     100 |     100 |                                   
  index.ts             |     100 |      100 |     100 |     100 |                                   
 chronik-client/proto  |    21.6 |     6.07 |    13.9 |   21.74 |                                   
  chronik.ts           |     6.1 |     0.83 |    2.54 |    6.09 | ...,3978-3985,3990-4027,4031-4036 
  chronikNode.ts       |   33.03 |    10.56 |    23.6 |   33.28 | ...,4991-5030,5038-5111,5146-5151 
 chronik-client/src    |   65.77 |    42.39 |   63.41 |   65.38 |                                   
  ChronikClient.ts     |    4.24 |        0 |       0 |    4.29 | 33-163,178-222,290-692            
  ChronikClientNode.ts |   90.84 |     72.8 |   96.38 |   90.84 | ...,1068,1078,1103,1115,1121,1127 
  failoverProxy.ts     |   75.22 |    51.61 |   62.06 |   74.52 | ...67,275-285,294,301,305,310,314 
  hex.ts               |   89.47 |       50 |      75 |   87.87 | 58,66-68                          
  validation.ts        |   93.33 |    81.81 |     100 |   92.59 | 33,39                             
-----------------------|---------|----------|---------|---------|-----------------------------------

##teamcity[blockOpened name='Code Coverage Summary']
##teamcity[buildStatisticValue key='CodeCoverageAbsBCovered' value='1188']
##teamcity[buildStatisticValue key='CodeCoverageAbsBTotal' value='4189']
##teamcity[buildStatisticValue key='CodeCoverageAbsRCovered' value='337']
##teamcity[buildStatisticValue key='CodeCoverageAbsRTotal' value='3897']
##teamcity[buildStatisticValue key='CodeCoverageAbsMCovered' value='187']
##teamcity[buildStatisticValue key='CodeCoverageAbsMTotal' value='761']
##teamcity[buildStatisticValue key='CodeCoverageAbsLCovered' value='1176']
##teamcity[buildStatisticValue key='CodeCoverageAbsLTotal' value='4148']
##teamcity[blockClosed name='Code Coverage Summary']
mv: cannot stat 'test_results/chronik-client-integration-tests-junit.xml': No such file or directory
Build chronik-client-integration-tests failed with exit code 1
Fabien requested changes to this revision.Jul 23 2024, 18:59
Fabien added a subscriber: Fabien.

back to your queue

This revision now requires changes to proceed.Jul 23 2024, 18:59

This is weird. It looks intermittent

Fabien requested changes to this revision.Jul 24 2024, 12:24

The deduplication was correct, you just called the boundary check later in the code flow which changed the behavior.

This revision now requires changes to proceed.Jul 24 2024, 12:24

rededuplicate boundary checks, keep them early in the function this time

Tail of the build log:

  File "/work/test/functional/setup_scripts/../test_framework/util.py", line 296, in wait_until_helper
    raise AssertionError(
AssertionError: Predicate ''''
            self.wait_until(lambda: is_finalblock(next_blockhash))
''' not true after 60.0 seconds
2024-07-24T13:42:42.037000Z TestFramework (INFO): Stopping nodes
2024-07-24T13:42:42.242000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240724_134129/setup_scripts/chronik-client_websocket_0
2024-07-24T13:42:42.242000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240724_134129/setup_scripts/chronik-client_websocket_0/test_framework.log
2024-07-24T13:42:42.243000Z TestFramework (ERROR): 
2024-07-24T13:42:42.243000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240724_134129/setup_scripts/chronik-client_websocket_0' to consolidate all logs
2024-07-24T13:42:42.244000Z TestFramework (ERROR): 
2024-07-24T13:42:42.244000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-07-24T13:42:42.244000Z TestFramework (ERROR): https://github.com/Bitcoin-ABC/bitcoin-abc/issues
2024-07-24T13:42:42.244000Z TestFramework (ERROR): 
Running Unit Tests for Test Framework Modules
setup_scripts/chronik-client_websocket.py started
setup_scripts/chronik-client_websocket.py failed, Duration: 73 s

stdout:

stderr:


TEST                                      | STATUS    | DURATION

setup_scripts/chronik-client_websocket.py | ✖ Failed  | 73 s

ALL                                       | ✖ Failed  | 73 s (accumulated) 
Runtime: 73 s

Test runner for chronik-client_websocket completed with code 1
-----------------------|---------|----------|---------|---------|-----------------------------------
File                   | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s                 
-----------------------|---------|----------|---------|---------|-----------------------------------
All files              |   28.35 |     8.64 |   24.57 |   28.35 |                                   
 chronik-client        |     100 |      100 |     100 |     100 |                                   
  index.ts             |     100 |      100 |     100 |     100 |                                   
 chronik-client/proto  |    21.6 |     6.07 |    13.9 |   21.74 |                                   
  chronik.ts           |     6.1 |     0.83 |    2.54 |    6.09 | ...,3978-3985,3990-4027,4031-4036 
  chronikNode.ts       |   33.03 |    10.56 |    23.6 |   33.28 | ...,4991-5030,5038-5111,5146-5151 
 chronik-client/src    |   65.77 |    42.39 |   63.41 |   65.38 |                                   
  ChronikClient.ts     |    4.24 |        0 |       0 |    4.29 | 33-163,178-222,290-692            
  ChronikClientNode.ts |   90.84 |     72.8 |   96.38 |   90.84 | ...,1068,1078,1103,1115,1121,1127 
  failoverProxy.ts     |   75.22 |    51.61 |   62.06 |   74.52 | ...67,275-285,294,301,305,310,314 
  hex.ts               |   89.47 |       50 |      75 |   87.87 | 58,66-68                          
  validation.ts        |   93.33 |    81.81 |     100 |   92.59 | 33,39                             
-----------------------|---------|----------|---------|---------|-----------------------------------

##teamcity[blockOpened name='Code Coverage Summary']
##teamcity[buildStatisticValue key='CodeCoverageAbsBCovered' value='1188']
##teamcity[buildStatisticValue key='CodeCoverageAbsBTotal' value='4189']
##teamcity[buildStatisticValue key='CodeCoverageAbsRCovered' value='337']
##teamcity[buildStatisticValue key='CodeCoverageAbsRTotal' value='3897']
##teamcity[buildStatisticValue key='CodeCoverageAbsMCovered' value='187']
##teamcity[buildStatisticValue key='CodeCoverageAbsMTotal' value='761']
##teamcity[buildStatisticValue key='CodeCoverageAbsLCovered' value='1176']
##teamcity[buildStatisticValue key='CodeCoverageAbsLTotal' value='4148']
##teamcity[blockClosed name='Code Coverage Summary']
mv: cannot stat 'test_results/chronik-client-integration-tests-junit.xml': No such file or directory
Build chronik-client-integration-tests failed with exit code 1

I'm wondering if the way each individual CBlockIndex is fetched with the cs_main lock could be an issue. I guess there could be more efficient ways to do it, but mostly I wonder if there is a risk of returning an inconsistent list of headers if a reorg happens between two locks.

@bot chronik-client-integration-test

The build failed due to an unexpected infrastructure outage. The administrators have been notified to investigate. Sorry for the inconvenience.

The setup_scripts/chronik-client_websocket.py failure above seems unrelated to me. It still is a bit suspicious, as I haven't seen an unrelated chronik ws related failure in a while.

rebase and shorten chronik_bridge::ffi -> ffi

This revision is now accepted and ready to land.Jul 25 2024, 14:08

hold cs_main while fetching all the headers instead of holding and releasing it for each individual header, to avoid getting an inconsistent chain

chronik/chronik-bridge/src/ffi.rs
129–134
chronik/chronik-cpp/chronik_bridge.cpp
215–216

In this diff this check is redundant with the chech already done at the callsite, but I plan to reuse this function in a future diff where the check will be more relaxed (no limit of 500 blocks)

rename API (headers -> block-headers, pb.Header -> pb.BlockHeaders)

handle_header_range -> handle_block_headers

PiRK retitled this revision from [chronik] add a headers endpoint to [chronik] add a block-headers endpoint.Aug 1 2024, 09:12

Tail of the build log:

  File "/work/abc-ci-builds/ecash-lib-integration-tests/test/functional/test_runner.py", line 361, in main
    os.makedirs(tmpdir)
  File "/usr/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/work/abc-ci-builds/ecash-lib-integration-tests/test/tmp/test_runner_₿₵_🏃_20240801_104729'
Test runner completed with code 1
----------------------------|---------|----------|---------|---------|------------------------------
File                        | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s            
----------------------------|---------|----------|---------|---------|------------------------------
All files                   |   58.73 |    49.01 |   56.39 |   58.66 |                              
 ecash-lib                  |       0 |        0 |       0 |       0 |                              
  eslint.config.js          |       0 |        0 |       0 |       0 |                              
 ecash-lib/src              |   73.12 |    56.58 |   73.07 |   72.88 |                              
  ecc.ts                    |   57.14 |    83.33 |      40 |   57.14 | 23-31                        
  hash.ts                   |   88.88 |    83.33 |      80 |   88.88 | 14                           
  index.ts                  |       0 |        0 |       0 |       0 |                              
  indexBrowser.ts           |       0 |        0 |       0 |       0 |                              
  indexNodeJs.ts            |       0 |        0 |       0 |       0 |                              
  initBrowser.ts            |       0 |      100 |       0 |       0 | 11-13                        
  initNodeJs.ts             |     100 |      100 |     100 |     100 |                              
  op.ts                     |   49.15 |     50.9 |      80 |   49.15 | ...89,92,104,107,109,117-122 
  opcode.ts                 |     100 |    83.33 |     100 |     100 | 1                            
  script.ts                 |   52.63 |    38.09 |      60 |    50.9 | ...4-135,146,156,166,188-199 
  sigHashType.ts            |   77.77 |       44 |   85.71 |   77.77 | 26-38                        
  tx.ts                     |   93.47 |    79.16 |    90.9 |   93.18 | 123-125                      
  txBuilder.ts              |   52.74 |    46.42 |   66.66 |   51.72 | ...3-107,124-179,214,244-248 
  unsignedTx.ts             |    73.8 |    57.14 |   78.57 |   74.07 | ...3,151,159,184,192,198-201 
 ecash-lib/src/ffi          |   28.26 |    15.94 |   16.98 |   28.98 |                              
  ecash_lib_wasm_browser.js |       0 |        0 |       0 |       0 | 3-336                        
  ecash_lib_wasm_nodejs.js  |    61.9 |       55 |   39.13 |   62.75 | ...1,197-215,237,250-251,255 
 ecash-lib/src/io           |   59.55 |    60.29 |   70.58 |   58.77 |                              
  bytes.ts                  |     7.4 |    71.42 |    12.5 |     7.4 | 13-64                        
  hex.ts                    |   82.05 |     62.5 |      80 |   82.35 | 41-45,50,58                  
  int.ts                    |       0 |        0 |       0 |       0 |                              
  str.ts                    |   85.71 |    83.33 |   66.66 |   85.71 | 15                           
  varsize.ts                |      32 |    36.36 |   66.66 |      32 | 14-24,40-47                  
  writer.ts                 |       0 |        0 |       0 |       0 |                              
  writerbytes.ts            |   83.33 |    68.42 |     100 |   83.33 | 33,43,53,63,79               
  writerlength.ts           |     100 |    83.33 |     100 |     100 | 1                            
 ecash-lib/src/test         |   84.93 |    51.11 |   81.25 |   85.29 |                              
  testRunner.ts             |   84.93 |    51.11 |   81.25 |   85.29 | 69-71,83-85,108,119,162,185  
 ecash-lib/src/token        |   60.33 |    47.14 |   53.33 |   60.11 |                              
  alp.ts                    |   82.92 |    89.47 |   83.33 |   82.92 | 110-123,142                  
  common.ts                 |     100 |    83.33 |     100 |     100 | 1                            
  empp.ts                   |    92.3 |       75 |     100 |   91.66 | 12                           
  slp.ts                    |    28.2 |    13.51 |    7.69 |    28.2 | ...7,174-178,185-197,201-211 
----------------------------|---------|----------|---------|---------|------------------------------

##teamcity[blockOpened name='Code Coverage Summary']
##teamcity[buildStatisticValue key='CodeCoverageAbsBCovered' value='723']
##teamcity[buildStatisticValue key='CodeCoverageAbsBTotal' value='1231']
##teamcity[buildStatisticValue key='CodeCoverageAbsRCovered' value='224']
##teamcity[buildStatisticValue key='CodeCoverageAbsRTotal' value='457']
##teamcity[buildStatisticValue key='CodeCoverageAbsMCovered' value='119']
##teamcity[buildStatisticValue key='CodeCoverageAbsMTotal' value='211']
##teamcity[buildStatisticValue key='CodeCoverageAbsLCovered' value='704']
##teamcity[buildStatisticValue key='CodeCoverageAbsLTotal' value='1200']
##teamcity[blockClosed name='Code Coverage Summary']
mv: cannot stat 'test_results/ecash-lib-integration-tests-junit.xml': No such file or directory
Build ecash-lib-integration-tests failed with exit code 1
This revision was automatically updated to reflect the committed changes.