Page MenuHomePhabricator

[Chronik] Relax plugin height checks at startup
ClosedPublic

Authored by tobias_ruck on Jul 31 2024, 15:23.

Details

Reviewers
Fabien
Group Reviewers
Restricted Project
Commits
rABCe3b90a9aea1b: [Chronik] Relax plugin height checks at startup
Summary

Currently, we require all plugins to be synced for every block since genesis. However, this is overkill for most plugins, as most have the first usage of their LOKAD ID much later.

In this diff, allow plugins to load if there's currently no txs that would apply for it indexed in the DB already.

Depends on D16568.

Test Plan

cargo test -p chronik-indexer && ./test/functional/test_runner.py chronik_plugins_setup

Event Timeline

Also fix a small bug where can't get the tx details for coinbase txs if the plugin index is enabled.

Could you please split the bugfix in its own diff with test ?

Fabien requested changes to this revision.Aug 1 2024, 09:45
This revision now requires changes to proceed.Aug 1 2024, 09:45

Tail of the build log:

    Checking rand v0.8.5
    Checking http-body-util v0.1.1
    Checking url v2.5.0
    Checking tracing v0.1.40
    Checking chronik-util v0.1.0 (/work/chronik/chronik-util)
    Checking sha1 v0.10.6
    Checking embedded-io v0.4.0
    Checking cobs v0.2.3
    Checking data-encoding v2.5.0
    Checking ryu v1.0.17
    Checking smallvec v1.13.2
    Checking httpdate v1.0.3
    Checking utf-8 v0.7.6
    Checking postcard v1.0.8
    Checking pin-project v1.1.5
    Checking tungstenite v0.21.0
    Checking chronik-plugin-common v0.1.0 (/work/chronik/chronik-plugin-common)
   Compiling prost-build v0.11.9
    Checking toml_datetime v0.6.6
    Checking chronik-plugin v0.1.0 (/work/chronik/chronik-plugin)
    Checking serde_spanned v0.6.6
   Compiling pyo3 v0.22.2
    Checking topo_sort v0.4.0
    Checking seahash v4.1.0
    Checking sync_wrapper v0.1.2
    Checking mime v0.3.17
    Checking winnow v0.6.13
    Checking memoffset v0.9.1
    Checking axum-core v0.4.3
    Checking serde_json v1.0.115
    Checking serde_urlencoded v0.7.1
    Checking hyper v1.2.0
    Checking tokio-tungstenite v0.21.0
    Checking tower v0.4.13
    Checking futures-executor v0.3.30
    Checking serde_path_to_error v0.1.16
    Checking hyper-util v0.1.3
    Checking matchit v0.7.3
    Checking unindent v0.2.3
    Checking sync_wrapper v1.0.1
    Checking unicode-segmentation v1.11.0
    Checking base64 v0.21.7
    Checking futures v0.3.30
    Checking tower-http v0.5.2
   Compiling chronik-proto v0.1.0 (/work/chronik/chronik-proto)
    Checking convert_case v0.6.0
    Checking toml_edit v0.22.14
   Compiling chronik-bridge v0.1.0 (/work/chronik/chronik-bridge)
   Compiling librocksdb-sys v0.11.0+8.1.1
    Checking axum v0.7.5
   Compiling pyo3-macros v0.22.2
    Checking toml v0.8.14
    Checking chronik-plugin-impl v0.1.0 (/work/chronik/chronik-plugin-impl)
    Checking rocksdb v0.21.0
    Checking chronik-db v0.1.0 (/work/chronik/chronik-db)
    Checking chronik-indexer v0.1.0 (/work/chronik/chronik-indexer)
    Checking chronik-http v0.1.0 (/work/chronik/chronik-http)
    Finished dev [unoptimized + debuginfo] target(s) in 4m 39s
ninja: build stopped: cannot make progress due to previous errors.
Build build-chronik failed with exit code 1

Tail of the build log:

   Doc-tests chronik-bridge

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests chronik-db

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests chronik-http

running 1 test
test chronik/chronik-http/src/protobuf.rs - protobuf::Protobuf (line 29) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.32s

   Doc-tests chronik-indexer

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests chronik-plugin

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests chronik-plugin-common

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests chronik-plugin-impl

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests chronik-proto

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests chronik-util

running 2 tests
test chronik/chronik-util/src/log.rs - log::log (line 65) ... ignored
test chronik/chronik-util/src/log.rs - log::log_chronik (line 87) ... ignored

test result: ok. 0 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 0.00s

ninja: build stopped: cannot make progress due to previous errors.
Build build-chronik-plugins failed with exit code 1

Tail of the build log:

  File "/work/test/functional/setup_scripts/../test_framework/util.py", line 296, in wait_until_helper
    raise AssertionError(
AssertionError: Predicate ''''
            self.wait_until(lambda: is_finalblock(next_blockhash))
''' not true after 60.0 seconds
2024-08-01T13:25:46.857000Z TestFramework (INFO): Stopping nodes
2024-08-01T13:25:47.011000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240801_132435/setup_scripts/chronik-client_websocket_0
2024-08-01T13:25:47.011000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240801_132435/setup_scripts/chronik-client_websocket_0/test_framework.log
2024-08-01T13:25:47.011000Z TestFramework (ERROR): 
2024-08-01T13:25:47.012000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240801_132435/setup_scripts/chronik-client_websocket_0' to consolidate all logs
2024-08-01T13:25:47.012000Z TestFramework (ERROR): 
2024-08-01T13:25:47.012000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-08-01T13:25:47.012000Z TestFramework (ERROR): https://github.com/Bitcoin-ABC/bitcoin-abc/issues
2024-08-01T13:25:47.013000Z TestFramework (ERROR): 
Running Unit Tests for Test Framework Modules
setup_scripts/chronik-client_websocket.py started
setup_scripts/chronik-client_websocket.py failed, Duration: 72 s

stdout:

stderr:


TEST                                      | STATUS    | DURATION

setup_scripts/chronik-client_websocket.py | ✖ Failed  | 72 s

ALL                                       | ✖ Failed  | 72 s (accumulated) 
Runtime: 72 s

Test runner for chronik-client_websocket completed with code 1
-----------------------|---------|----------|---------|---------|-----------------------------------
File                   | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s                 
-----------------------|---------|----------|---------|---------|-----------------------------------
All files              |   28.35 |     8.64 |   24.57 |   28.35 |                                   
 chronik-client        |     100 |      100 |     100 |     100 |                                   
  index.ts             |     100 |      100 |     100 |     100 |                                   
 chronik-client/proto  |    21.6 |     6.07 |    13.9 |   21.74 |                                   
  chronik.ts           |     6.1 |     0.83 |    2.54 |    6.09 | ...,3978-3985,3990-4027,4031-4036 
  chronikNode.ts       |   33.03 |    10.56 |    23.6 |   33.28 | ...,4991-5030,5038-5111,5146-5151 
 chronik-client/src    |   65.77 |    42.39 |   63.41 |   65.38 |                                   
  ChronikClient.ts     |    4.24 |        0 |       0 |    4.29 | 33-163,178-222,290-692            
  ChronikClientNode.ts |   90.84 |     72.8 |   96.38 |   90.84 | ...,1068,1078,1103,1115,1121,1127 
  failoverProxy.ts     |   75.22 |    51.61 |   62.06 |   74.52 | ...67,275-285,294,301,305,310,314 
  hex.ts               |   89.47 |       50 |      75 |   87.87 | 58,66-68                          
  validation.ts        |   93.33 |    81.81 |     100 |   92.59 | 33,39                             
-----------------------|---------|----------|---------|---------|-----------------------------------

##teamcity[blockOpened name='Code Coverage Summary']
##teamcity[buildStatisticValue key='CodeCoverageAbsBCovered' value='1188']
##teamcity[buildStatisticValue key='CodeCoverageAbsBTotal' value='4189']
##teamcity[buildStatisticValue key='CodeCoverageAbsRCovered' value='337']
##teamcity[buildStatisticValue key='CodeCoverageAbsRTotal' value='3897']
##teamcity[buildStatisticValue key='CodeCoverageAbsMCovered' value='187']
##teamcity[buildStatisticValue key='CodeCoverageAbsMTotal' value='761']
##teamcity[buildStatisticValue key='CodeCoverageAbsLCovered' value='1176']
##teamcity[buildStatisticValue key='CodeCoverageAbsLTotal' value='4148']
##teamcity[blockClosed name='Code Coverage Summary']
mv: cannot stat 'test_results/chronik-client-integration-tests-junit.xml': No such file or directory
Build chronik-client-integration-tests failed with exit code 1
Fabien requested changes to this revision.Aug 4 2024, 08:19
Fabien added inline comments.
chronik/chronik-indexer/src/indexer.rs
214 ↗(On Diff #49004)

if you invalidate the block you will start banning your peers also because your node thinks they are on an invalid chain

238 ↗(On Diff #49004)
1162 ↗(On Diff #49004)

Style: it's more a bit more readable imo: you are searching for the minimum so it seems more logical to use <

1190 ↗(On Diff #49004)

dito

1213 ↗(On Diff #49004)

Is there a way to improve the message a bit? If several plugins are out of sync you will use the one with the earliest tx num which is good, but also show it as the single unsynched plugin. Can you print the list of plugins instead (but still a single tx) ?

This revision now requires changes to proceed.Aug 4 2024, 08:19
This revision is now accepted and ready to land.Aug 6 2024, 14:39