[Chronik] Add `PluginsWriter::update_sync_height`, enforce plugins are synced
ClosedPublic
Actions

Authored by tobias_ruck on Jul 27 2024, 15:05.

Details

Reviewers

Fabien
PiRK

Group Reviewers

Restricted Project

Commits

rABCe5fc6a5832e6: [Chronik] Add `PluginsWriter::update_sync_height`, enforce plugins are synced

Summary

Currently, if a plugin desyncs with the DB, nothing happens, but this could lead to an inconsistent state of Chronik, e.g. where a plugin seems to be up-to-date but actually skipped a few blocks resulting in wrong data being reported to the user.

To avoid this, we previously already added a field sync_height to store the plugin's synced height, but it was unused. In this diff, we enforce that all plugins must be synced at startup, or instruct the user to either reindex Chronik or disable the plugin.

In the future, we can automatically reindex only the transactions that are relevant for the plugin using the LOKAD ID index, but since this is non-trivial this will be implemented later. Prioritizing getting a plugin system with mediocre UX out sooner.

Depends on D16532.

Test Plan

cargo test -p chronik-db && ./test/functional/test_runner.py chronik_plugins_setup

Diff Detail

Repository

rABC Bitcoin ABC

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

tobias_ruck created this revision.Jul 27 2024, 15:05

Herald added a reviewer: Restricted Project. · View Herald TranscriptJul 27 2024, 15:05

initialize to -1 by default

tobias_ruck edited the summary of this revision. (Show Details)Jul 27 2024, 15:26

tobias_ruck added a parent revision: D16532: [Chronik] Add PluginsWrite::{insert, delete}.

Harbormaster completed remote builds in B29732: Diff 48861.Jul 27 2024, 15:45

Build Bitcoin ABC Diffs / Diff Testing (chronik-client-integration-tests) failed.

Tail of the build log:

  File "/work/test/functional/setup_scripts/../test_framework/util.py", line 296, in wait_until_helper
    raise AssertionError(
AssertionError: Predicate ''''
            self.wait_until(lambda: is_finalblock(next_blockhash))
''' not true after 60.0 seconds
2024-07-27T15:44:44.937000Z TestFramework (INFO): Stopping nodes
2024-07-27T15:44:45.191000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240727_154332/setup_scripts/chronik-client_websocket_0
2024-07-27T15:44:45.191000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240727_154332/setup_scripts/chronik-client_websocket_0/test_framework.log
2024-07-27T15:44:45.192000Z TestFramework (ERROR): 
2024-07-27T15:44:45.192000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240727_154332/setup_scripts/chronik-client_websocket_0' to consolidate all logs
2024-07-27T15:44:45.192000Z TestFramework (ERROR): 
2024-07-27T15:44:45.192000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-07-27T15:44:45.192000Z TestFramework (ERROR): https://github.com/Bitcoin-ABC/bitcoin-abc/issues
2024-07-27T15:44:45.192000Z TestFramework (ERROR): 
Running Unit Tests for Test Framework Modules
[1msetup_scripts/chronik-client_websocket.py[0m started
[1msetup_scripts/chronik-client_websocket.py[0m failed, Duration: 73 s

[1mstdout:[0m

[1mstderr:[0m

[1m
TEST                                      | STATUS    | DURATION

[0m[0;31msetup_scripts/chronik-client_websocket.py | ✖ Failed  | 73 s
[0m[0;31m[1m
ALL                                       | ✖ Failed  | 73 s (accumulated) 
[0m[0mRuntime: 73 s

Test runner for chronik-client_websocket completed with code 1
-----------------------|---------|----------|---------|---------|-----------------------------------
File                   | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s                 
-----------------------|---------|----------|---------|---------|-----------------------------------
All files              |   28.35 |     8.64 |   24.57 |   28.35 |                                   
 chronik-client        |     100 |      100 |     100 |     100 |                                   
  index.ts             |     100 |      100 |     100 |     100 |                                   
 chronik-client/proto  |    21.6 |     6.07 |    13.9 |   21.74 |                                   
  chronik.ts           |     6.1 |     0.83 |    2.54 |    6.09 | ...,3978-3985,3990-4027,4031-4036 
  chronikNode.ts       |   33.03 |    10.56 |    23.6 |   33.28 | ...,4991-5030,5038-5111,5146-5151 
 chronik-client/src    |   65.77 |    42.39 |   63.41 |   65.38 |                                   
  ChronikClient.ts     |    4.24 |        0 |       0 |    4.29 | 33-163,178-222,290-692            
  ChronikClientNode.ts |   90.84 |     72.8 |   96.38 |   90.84 | ...,1068,1078,1103,1115,1121,1127 
  failoverProxy.ts     |   75.22 |    51.61 |   62.06 |   74.52 | ...67,275-285,294,301,305,310,314 
  hex.ts               |   89.47 |       50 |      75 |   87.87 | 58,66-68                          
  validation.ts        |   93.33 |    81.81 |     100 |   92.59 | 33,39                             
-----------------------|---------|----------|---------|---------|-----------------------------------

##teamcity[blockOpened name='Code Coverage Summary']
##teamcity[buildStatisticValue key='CodeCoverageAbsBCovered' value='1188']
##teamcity[buildStatisticValue key='CodeCoverageAbsBTotal' value='4189']
##teamcity[buildStatisticValue key='CodeCoverageAbsRCovered' value='337']
##teamcity[buildStatisticValue key='CodeCoverageAbsRTotal' value='3897']
##teamcity[buildStatisticValue key='CodeCoverageAbsMCovered' value='187']
##teamcity[buildStatisticValue key='CodeCoverageAbsMTotal' value='761']
##teamcity[buildStatisticValue key='CodeCoverageAbsLCovered' value='1176']
##teamcity[buildStatisticValue key='CodeCoverageAbsLTotal' value='4148']
##teamcity[blockClosed name='Code Coverage Summary']
mv: cannot stat 'test_results/chronik-client-integration-tests-junit.xml': No such file or directory
Build chronik-client-integration-tests failed with exit code 1

plugin_writer -> plugins_writer

Harbormaster completed remote builds in B29731: Diff 48860.Jul 27 2024, 15:50

Harbormaster completed remote builds in B29733: Diff 48862.Jul 27 2024, 16:09

tobias_ruck added a child revision: D16537: [Chronik] Enable indexing txs by plugins + return protobuf.Jul 27 2024, 20:55

tobias_ruck published this revision for review.Jul 27 2024, 20:58

PiRK accepted this revision.Jul 29 2024, 07:29

This revision is now accepted and ready to land.Jul 29 2024, 07:29

Closed by commit rABCe5fc6a5832e6: [Chronik] Add `PluginsWriter::update_sync_height`, enforce plugins are synced (authored by tobias_ruck). · Explain WhyJul 29 2024, 12:38

This revision was automatically updated to reflect the committed changes.

tobias_ruck added a commit: rABCe5fc6a5832e6: [Chronik] Add `PluginsWriter::update_sync_height`, enforce plugins are synced.