Details

Reviewers

Group Reviewers

Restricted Project

Commits

rABC3c106476b316: [Chronik] Add LOKAD ID index

Summary

Protocols on eCash are specified using a LOKAD ID, a 4 byte identifier. There are a few popular protocols, like SLP and ALP, however, it's not easy adding other protocols them as the infrastructure is not quite ready.

This diff adds a simple index, which finds txs with LOKAD IDs and groups them. A UTXO index is not added because LOKAD IDs are per-tx, not per-output.

The index is enabled by default, however, existing instances without the LOKAD ID index will have to manually specify -chroniklokadidindex=1, which will reindex the LOKAD IDs from genesis.

Specifying -chroniklokadidindex=0 manually on an instance with the index enabled will wipe the index.

This index is also useful for the Plugin interface, i.e. when a plugin is added for a LOKAD ID, we know exactly which txs will have to be indexed already and don't need to search all blocks again.

Depends on D15913.

Test Plan

./test/functional/test_runner.py chronik_lokad_id_group

Diff Detail

Repository

rABC Bitcoin ABC

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

tobias_ruck created this revision.Apr 8 2024, 00:26

Herald added a reviewer: Restricted Project. · View Herald TranscriptApr 8 2024, 00:26

tobias_ruck requested review of this revision.Apr 8 2024, 00:26

tobias_ruck edited the summary of this revision. (Show Details)Apr 8 2024, 00:27

tobias_ruck edited the test plan for this revision. (Show Details)

remove rocksdb features

Harbormaster completed remote builds in B28401: Diff 46937.Apr 8 2024, 00:47

Harbormaster completed remote builds in B28402: Diff 46938.Apr 8 2024, 01:04

bytesofman added a subscriber: bytesofman.Apr 8 2024, 14:04

bytesofman added inline comments.

chronik/chronik-db/src/mem/mempool.rs
37 ↗	(On Diff #46938)	prob needed for `is_token_index_enabled` as well and outside scope of this diff -- there should be a way for a user to ask a chronik instance what indexing it has enabled.
chronik/chronik-http/src/server.rs
213 ↗	(On Diff #46938)	could we query the server with more than one filter? say, "get all txs of given lokadId involving given outputScript"? easy enough for an appdev to filter with the current logic, but it does mean we have a pattern of app dev requests too much info from chronik --> then throws out what isn't being used Since chronik is indexing both, it would be nice if we could get a more-filtered result. That said, we do not want slpdb complexity in queries.
chronik/chronik-http/src/ws.rs
189 ↗	(On Diff #46938)	very nice
chronik/chronik-indexer/src/indexer.rs
214 ↗	(On Diff #46938)	why is this using the `is_lokad_id_index_enabled` bool for lokad but the `params` bool for token index?

Test coverage looks good and afaict the code is correct. I can't really review the API itself, I'm not in the loop regarding the intended use cases.

tobias_ruck added inline comments.Apr 12 2024, 13:36

chronik/chronik-db/src/mem/mempool.rs
37 ↗	(On Diff #46938)	Good idea—a `struct IndexingCapabilities` or so that has the flags. Utility be more obvious once we add the Plugin index but I can do it next.
chronik/chronik-http/src/server.rs
213 ↗	(On Diff #46938)	Unfortunately this is not trivial because they live in separate indexes, but later it shouldn't be too hard to "merge" those two when querying what you ask for, since the history of a script/token ID/LOKAD ID are sorted (making looking up if a it O(log n)). At some point we might wanna add a proper query language (uff), at least once we get like 3-5 more useful indices.
chronik/chronik-indexer/src/indexer.rs
214 ↗	(On Diff #46938)	This is because the token ID index is always enabled if `enable_token_index` is true, whereas the LOKAD ID index has a bit more complex activation logic. If indexing from genesis it's enabled, unless explicitly disabled, and if there's already a DB without the index, it's disabled, unless explicitly enabled.

Fabien requested changes to this revision.Apr 12 2024, 14:09

Fabien added inline comments.

chronik/chronik-cpp/chronik.cpp
57 ↗	(On Diff #46938)	You should avoid this and move the logic in the node, in the InitParameterInteraction() method: The option lives in the node anyway There are more possible issues with these options that can be handled, e.g. if -chroniklokadidindex=1 but -chroniktokenindex=0 we should either warn and disable the lokad id index or silently enable the token indexing.

This revision now requires changes to proceed.Apr 12 2024, 14:09

tobias_ruck added inline comments.Apr 12 2024, 19:14

chronik/chronik-cpp/chronik.cpp
57 ↗	(On Diff #46938)	Hmm, how would the node know whether Chronik has a DB or not? Or should we instead just have everyone reindex on the next version and tell them to opt out if they don't want the index? It seems like a simple and straightforward solution to me. Note: The chroniklokadidindex and chroniktokenindex are completely independent of each other

Fabien added inline comments.Apr 12 2024, 19:18

chronik/chronik-cpp/chronik.cpp
57 ↗	(On Diff #46938)	You still have to pass whether it's activated or not, but you can avoid passing if it's set. If you're not indexing tokens, indexing lokad ids seems useless to me. What would be the use case ?

tobias_ruck added inline comments.Apr 12 2024, 19:40

chronik/chronik-cpp/chronik.cpp
57 ↗	(On Diff #46938)	You still have to pass whether it's activated or not, but you can avoid passing if it's set. I don't get how I can avoid passing if it's set or not in this situation and get the same behavoir. There's three states we need to encode: explicit chroniklokadidindex=1, explicit chroniklokadidindex=0, flag not set (and should be derived from the state of the DB). If you're not indexing tokens, indexing lokad ids seems useless to me. What would be the use case ? Stamp would be an example that would benefit from a LOKAD ID index without a token index, or the memo.cash system Ethan is working on. Many such cases—the LOKAD ID spec predates the SLP spec.

Fabien added inline comments.Apr 12 2024, 19:53

chronik/chronik-cpp/chronik.cpp
57 ↗	(On Diff #46938)	I don't get how I can avoid passing if it's set or not in this situation and get the same behavoir. There's three states we need to encode: explicit chroniklokadidindex=1, explicit chroniklokadidindex=0, flag not set (and should be derived from the state of the DB). If indexing from genesis it's enabled, unless explicitly disabled, and if there's already a DB without the index, it's disabled, unless explicitly enabled. I think this is because this behavior is uncommon (unique if memory serves) and I missed it. Usually whatever the default is it takes precedence to whatever is in the db, which I think is fine. Either you reindex or you explicitely disable it (if it's on by default). Please note that now that chronik is officially released, this kind of change deserves a release note. Stamp would be an example that would benefit from a LOKAD ID index without a token index, or the memo.cash system Ethan is working on. Many such cases—the LOKAD ID spec predates the SLP spec. OK if there are use cases for this then it's fine.

tobias_ruck added inline comments.Apr 13 2024, 00:17

chronik/chronik-cpp/chronik.cpp
57 ↗	(On Diff #46938)	Ok, I can change it to default to reindex and make sure that people are informed that it’ll do that

always enable LOKAD ID index by default, toggle only by -chroniklokadidindex

Harbormaster completed remote builds in B28685: Diff 47335.Apr 20 2024, 10:54

Build Bitcoin ABC Diffs / Diff Testing (chronik-client-integration-tests) failed.

Failed tests logs:

====== Get blocktxs, txs, and history for SLP NFT1 token txs: "before each" hook for "Gets an SLP NFT1 send tx from the mempool".Get blocktxs, txs, and history for SLP NFT1 token txs "before each" hook for "Gets an SLP NFT1 send tx from the mempool" ======
Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/work/modules/chronik-client/test/integration/token_slp_nft1.ts)
    at listOnTimeout (node:internal/timers:573:17)
    at processTimers (node:internal/timers:514:7)

Each failure log is accessible here:
Get blocktxs, txs, and history for SLP NFT1 token txs: "before each" hook for "Gets an SLP NFT1 send tx from the mempool".Get blocktxs, txs, and history for SLP NFT1 token txs "before each" hook for "Gets an SLP NFT1 send tx from the mempool"

rerun CI

Harbormaster completed remote builds in B28689: Diff 47341.Apr 20 2024, 14:02

Fabien added inline comments.Apr 22 2024, 07:43

chronik/chronik-db/Cargo.toml
47 ↗	(On Diff #47341)	This will enable compression ?

remove rocksdb features again

Fabien accepted this revision.Apr 22 2024, 16:24

This revision is now accepted and ready to land.Apr 22 2024, 16:24

Harbormaster completed remote builds in B28709: Diff 47371.Apr 22 2024, 16:41

The build failed due to an unexpected infrastructure outage. The administrators have been notified to investigate. Sorry for the inconvenience.

Build Bitcoin ABC Diffs / Diff Testing (chronik-client-integration-tests) failed.

Tail of the build log:

  File "/work/test/functional/setup_scripts/../test_framework/util.py", line 297, in wait_until_helper
    raise AssertionError(
AssertionError: Predicate ''''
            self.wait_until(lambda: is_finalblock(next_blockhash))
''' not true after 60.0 seconds
2024-04-22T16:49:35.331000Z TestFramework (INFO): Stopping nodes
2024-04-22T16:49:35.836000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240422_164822/setup_scripts/chronik-client_websocket_0
2024-04-22T16:49:35.836000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240422_164822/setup_scripts/chronik-client_websocket_0/test_framework.log
2024-04-22T16:49:35.837000Z TestFramework (ERROR): 
2024-04-22T16:49:35.837000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240422_164822/setup_scripts/chronik-client_websocket_0' to consolidate all logs
2024-04-22T16:49:35.837000Z TestFramework (ERROR): 
2024-04-22T16:49:35.838000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-04-22T16:49:35.838000Z TestFramework (ERROR): https://github.com/Bitcoin-ABC/bitcoin-abc/issues
2024-04-22T16:49:35.838000Z TestFramework (ERROR): 
Running Unit Tests for Test Framework Modules
[1msetup_scripts/chronik-client_websocket.py[0m started
[1msetup_scripts/chronik-client_websocket.py[0m failed, Duration: 74 s

[1mstdout:[0m

[1mstderr:[0m

[1m
TEST                                      | STATUS    | DURATION

[0m[0;31msetup_scripts/chronik-client_websocket.py | ✖ Failed  | 74 s
[0m[0;31m[1m
ALL                                       | ✖ Failed  | 74 s (accumulated) 
[0m[0mRuntime: 74 s

Test runner for chronik-client_websocket completed with code 1
-----------------------|---------|----------|---------|---------|-----------------------------------
File                   | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s                 
-----------------------|---------|----------|---------|---------|-----------------------------------
All files              |   27.35 |    10.37 |   23.06 |   27.36 |                                   
 chronik-client        |     100 |      100 |     100 |     100 |                                   
  index.ts             |     100 |      100 |     100 |     100 |                                   
 chronik-client/proto  |   21.43 |     7.91 |   13.72 |   21.57 |                                   
  chronik.ts           |     6.1 |        1 |    2.54 |    6.09 | ...,3978-3985,3990-4027,4031-4036 
  chronikNode.ts       |   32.95 |    13.21 |   23.49 |   33.19 | ...,4882-4921,4929-5002,5037-5042 
 chronik-client/src    |   62.93 |     42.4 |   60.54 |   62.56 |                                   
  ChronikClient.ts     |    4.24 |        0 |       0 |    4.29 | 33-163,178-222,290-692            
  ChronikClientNode.ts |   89.68 |    70.14 |   95.71 |   89.68 | ...52,876,888,898,923,935,941,947 
  failoverProxy.ts     |   76.19 |    59.09 |   66.66 |   75.49 | ...61-264,267,275-285,294,303,307 
  hex.ts               |   89.47 |       75 |      75 |   87.87 | 58,66-68                          
  validation.ts        |      90 |    84.61 |     100 |   89.47 | 31,37                             
-----------------------|---------|----------|---------|---------|-----------------------------------

##teamcity[blockOpened name='Code Coverage Summary']
##teamcity[buildStatisticValue key='CodeCoverageAbsBCovered' value='1120']
##teamcity[buildStatisticValue key='CodeCoverageAbsBTotal' value='4095']
##teamcity[buildStatisticValue key='CodeCoverageAbsRCovered' value='459']
##teamcity[buildStatisticValue key='CodeCoverageAbsRTotal' value='4422']
##teamcity[buildStatisticValue key='CodeCoverageAbsMCovered' value='170']
##teamcity[buildStatisticValue key='CodeCoverageAbsMTotal' value='737']
##teamcity[buildStatisticValue key='CodeCoverageAbsLCovered' value='1110']
##teamcity[buildStatisticValue key='CodeCoverageAbsLTotal' value='4056']
##teamcity[blockClosed name='Code Coverage Summary']
mv: cannot stat 'test_results/chronik-client-integration-tests-junit.xml': No such file or directory
Build chronik-client-integration-tests failed with exit code 1

rerun CI because cash is supposed to be green

Harbormaster completed remote builds in B28723: Diff 47387.Apr 23 2024, 21:16

The build failed due to an unexpected infrastructure outage. The administrators have been notified to investigate. Sorry for the inconvenience.

Updated chronik.pay2stay.com/xec to this diff, upgrade took 3h:05m, so not too fast but not too slow.

Querying the confirmed history for "SLP\0" gives these txs, which seem like the first 25 SLP txs (which could be useful for future reference, too).

['545cba6f72a08cbcb08c7d4e8166267942e8cb9a611328805c62fa538e861ba4', '83bfe019fcf976142c55e7c0ad4a429f4be1bc2cb138bd8d0bab8dd4cd4758c4', '4f035d656ed5b6e94a884c88c09a8d2dee9c7e97901cce3adec966115e2a1ba5', '323a1e35ae0b356316093d20f2d9fbc995d19314b5c0148b78dc8d9c0dab9d35', '874306bda204d3a5dd15e03ea5732cccdca4c33a52df35162cdd64e30ea7f04e', '3a574837eaecf701ddf232ab4ad30c541be61061ee5a563d27809dde927d7ecf', '82a9c47118dd221bf528e8b9ee9daef626ca52fb824b92cbe52a83e87afb0fac', '170147548aad6de7c1df686c56e4846e0936c4573411b604a18d0ec76482dde2', 'e5ff3083cd2dcf87a40a4a4a478349a394c1a1eeffe4857c2a173b183fdd42a2', 'f56121d5a21a319204cf26ce68a6d607fefa02ba6ac42b4647fcad813b32d8b3', '660057b446cc4c930493607aa02e943e4fe7c38ae0816797ff7234ba72fea50f', '483d0198ed272bd0be7c6bbaf0e60340cce926f7d32143e2b09c5513922eaf87', '234893177b18a95dbfc1eb855d69f1c9cc256a317a6c51be8fd1b9a38ae072ce', 'a333e7ebd34f0e24b567e99ed27241e3cfda5e9952cacdaa8fab31a7ee7e544d', '89f4a12d7de52e3d117ef67e9e9be45eb9a1432afee61d99efc3576df2c7fab2', '938cc18e618967d787897bbc64b9a8d201b94ec7c69b1a9949eab0433ba5cdf8', 'ee9d3cf5153599c134147e3fac9844c68e216843f4452a1ce15a29452af6db34', '4640a734063ea79fa587a3cac38a70a2f6f3db0011e23514024185982110d0fa', 'e0f3e9ee5ab30c250c0d5fbd78c1eaedf3dfd998813ecbf29ee4c00dfb9acf12', '662bc5b6fe6fa2ab7ee3257ee31549d8490f3a7d591c52eecc053e97fc4c3a1c', 'f58c7bb144f24385c341687c725c095af274fe16666b2903b6e061f18d895ec5', '34b38d6751f38d6597eb9dbd74e009e98434b0fac6ed95637fc92aa6d7f8eb79', '550d19eb820e616a54b8a73372c4420b5a0567d8dc00f613b71c5234dc884b35', '95d460512143b636bbc5780d8b27b04fca3bfd2f22003ab48da594e2bab9cfc1', 'b36b0c7485ad569b98cc9b9614dc68a5208495f22ec3b00effcf963b135a5215']

So, seems like everything works in production, too. Will land right now.

rebase onto master to fix build?

Harbormaster completed remote builds in B28724: Diff 47388.Apr 23 2024, 21:40

Build Bitcoin ABC Diffs / Diff Testing (ecash-lib-integration-tests) failed.

Failed tests logs:

====== TxBuilder: "before all" hook for "TxBuilder P2PKH Wallet".TxBuilder "before all" hook for "TxBuilder P2PKH Wallet" ======
Error: Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/work/modules/ecash-lib/tests/txBuilder.test.ts)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)

Each failure log is accessible here:
TxBuilder: "before all" hook for "TxBuilder P2PKH Wallet".TxBuilder "before all" hook for "TxBuilder P2PKH Wallet"

rebase onto D16046

Harbormaster completed remote builds in B28726: Diff 47390.Apr 23 2024, 22:16

Build Bitcoin ABC Diffs / Diff Testing (build-chronik-plugins) failed.

Failed tests logs:

====== Bitcoin ABC functional tests: chronik_token_alp.py ======

------- Stdout: -------
2024-04-23T22:14:59.795000Z TestFramework (INFO): Initializing test directory /work/abc-ci-builds/build-chronik-plugins/test/tmp/test_runner_₿₵_  _20240423_220940/chronik_token_alp_139
2024-04-23T22:15:31.381000Z TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 627, in start_nodes
    node.wait_for_rpc_connection()
  File "/work/test/functional/test_framework/test_node.py", line 348, in wait_for_rpc_connection
    raise FailedToStartError(
test_framework.test_node.FailedToStartError: [node 0] bitcoind exited with status 1 during initialization

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 146, in main
    self.setup()
  File "/work/test/functional/test_framework/test_framework.py", line 381, in setup
    self.setup_network()
  File "/work/test/functional/test_framework/test_framework.py", line 488, in setup_network
    self.setup_nodes()
  File "/work/test/functional/test_framework/test_framework.py", line 513, in setup_nodes
    self.start_nodes()
  File "/work/test/functional/test_framework/test_framework.py", line 630, in start_nodes
    self.stop_nodes()
  File "/work/test/functional/test_framework/test_framework.py", line 645, in stop_nodes
    node.stop_node(wait=wait, wait_until_stopped=False)
  File "/work/test/functional/test_framework/test_node.py", line 501, in stop_node
    self.stop(wait=wait)
  File "/work/test/functional/test_framework/test_node.py", line 278, in __getattr__
    assert self.rpc is not None, self._node_msg("Error: RPC not initialized")
AssertionError: [node 0] Error: RPC not initialized
2024-04-23T22:15:31.432000Z TestFramework (INFO): Stopping nodes
[node 0] Cleaning up leftover process
------- Stderr: -------
Traceback (most recent call last):
  File "/work/test/functional/chronik_token_alp.py", line 666, in <module>
    ChronikTokenAlp().main()
  File "/work/test/functional/test_framework/test_framework.py", line 170, in main
    exit_code = self.shutdown()
  File "/work/test/functional/test_framework/test_framework.py", line 396, in shutdown
    self.stop_nodes()
  File "/work/test/functional/test_framework/test_framework.py", line 645, in stop_nodes
    node.stop_node(wait=wait, wait_until_stopped=False)
  File "/work/test/functional/test_framework/test_node.py", line 501, in stop_node
    self.stop(wait=wait)
  File "/work/test/functional/test_framework/test_node.py", line 278, in __getattr__
    assert self.rpc is not None, self._node_msg("Error: RPC not initialized")
AssertionError: [node 0] Error: RPC not initialized

Each failure log is accessible here:
Bitcoin ABC functional tests: chronik_token_alp.py

rerun CI, last one got Error: Chronik failed binding to 127.0.0.1:33962: Address already in use (os error 98)

Harbormaster completed remote builds in B28730: Diff 47394.Apr 24 2024, 01:32

Closed by commit rABC3c106476b316: [Chronik] Add LOKAD ID index (authored by tobias_ruck). · Explain WhyApr 24 2024, 16:29

This revision was automatically updated to reflect the committed changes.

tobias_ruck added a commit: rABC3c106476b316: [Chronik] Add LOKAD ID index.

PiRK mentioned this in D16866: [chronik] remove outdated comment for enable_lokad_id_index.Oct 8 2024, 15:46

PiRK mentioned this in rABC2fdbd11f292d: [chronik] remove outdated comment for enable_lokad_id_index.Oct 8 2024, 18:33

[Chronik] Add LOKAD ID index
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 47403

chronik/bitcoinsuite-core/src/tx/tx.rs

chronik/chronik-cpp/chronik.cpp

chronik/chronik-db/src/db.rs

chronik/chronik-db/src/groups/lokad_id.rs

chronik/chronik-db/src/groups/mod.rs

chronik/chronik-db/src/io/group_history.rs

chronik/chronik-db/src/io/group_utxos.rs

chronik/chronik-db/src/io/metadata.rs

chronik/chronik-db/src/mem/mempool.rs

chronik/chronik-http/src/handlers.rs

chronik/chronik-http/src/parse.rs

chronik/chronik-http/src/server.rs

chronik/chronik-http/src/ws.rs

chronik/chronik-indexer/src/indexer.rs

chronik/chronik-indexer/src/subs.rs

chronik/chronik-lib/src/bridge.rs

chronik/chronik-lib/src/ffi.rs

chronik/chronik-proto/proto/chronik.proto

src/init.cpp

test/functional/chronik_lokad_id_group.py

test/functional/test_framework/chronik/client.py

test/functional/test_framework/util.py

[Chronik] Add LOKAD ID indexClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 47403

chronik/bitcoinsuite-core/src/tx/tx.rs

chronik/chronik-cpp/chronik.cpp

chronik/chronik-db/src/db.rs

chronik/chronik-db/src/groups/lokad_id.rs

chronik/chronik-db/src/groups/mod.rs

chronik/chronik-db/src/io/group_history.rs

chronik/chronik-db/src/io/group_utxos.rs

chronik/chronik-db/src/io/metadata.rs

chronik/chronik-db/src/mem/mempool.rs

chronik/chronik-http/src/handlers.rs

chronik/chronik-http/src/parse.rs

chronik/chronik-http/src/server.rs

chronik/chronik-http/src/ws.rs

chronik/chronik-indexer/src/indexer.rs

chronik/chronik-indexer/src/subs.rs

chronik/chronik-lib/src/bridge.rs

chronik/chronik-lib/src/ffi.rs

chronik/chronik-proto/proto/chronik.proto

src/init.cpp

test/functional/chronik_lokad_id_group.py

test/functional/test_framework/chronik/client.py

test/functional/test_framework/util.py

[Chronik] Add LOKAD ID index
ClosedPublic
Actions

Revision Contents
Changeset List