Page MenuHomePhabricator

[Chronik] Add LOKAD ID index
ClosedPublic

Authored by tobias_ruck on Apr 8 2024, 00:26.

Details

Reviewers
Fabien
Group Reviewers
Restricted Project
Commits
rABC3c106476b316: [Chronik] Add LOKAD ID index
Summary

Protocols on eCash are specified using a LOKAD ID, a 4 byte identifier. There are a few popular protocols, like SLP and ALP, however, it's not easy adding other protocols them as the infrastructure is not quite ready.

This diff adds a simple index, which finds txs with LOKAD IDs and groups them. A UTXO index is not added because LOKAD IDs are per-tx, not per-output.

The index is enabled by default, however, existing instances without the LOKAD ID index will have to manually specify -chroniklokadidindex=1, which will reindex the LOKAD IDs from genesis.

Specifying -chroniklokadidindex=0 manually on an instance with the index enabled will wipe the index.

This index is also useful for the Plugin interface, i.e. when a plugin is added for a LOKAD ID, we know exactly which txs will have to be indexed already and don't need to search all blocks again.

Depends on D15913.

Test Plan

./test/functional/test_runner.py chronik_lokad_id_group

Event Timeline

tobias_ruck edited the test plan for this revision. (Show Details)
bytesofman added inline comments.
chronik/chronik-db/src/mem/mempool.rs
37 ↗(On Diff #46938)

prob needed for is_token_index_enabled as well and outside scope of this diff -- there should be a way for a user to ask a chronik instance what indexing it has enabled.

chronik/chronik-http/src/server.rs
213 ↗(On Diff #46938)

could we query the server with more than one filter? say, "get all txs of given lokadId involving given outputScript"?

easy enough for an appdev to filter with the current logic, but it does mean we have a pattern of

app dev requests too much info from chronik --> then throws out what isn't being used

Since chronik is indexing both, it would be nice if we could get a more-filtered result. That said, we do not want slpdb complexity in queries.

chronik/chronik-http/src/ws.rs
189 ↗(On Diff #46938)

very nice

chronik/chronik-indexer/src/indexer.rs
214 ↗(On Diff #46938)

why is this using the is_lokad_id_index_enabled bool for lokad but the params bool for token index?

Test coverage looks good and afaict the code is correct. I can't really review the API itself, I'm not in the loop regarding the intended use cases.

chronik/chronik-db/src/mem/mempool.rs
37 ↗(On Diff #46938)

Good idea—a struct IndexingCapabilities or so that has the flags. Utility be more obvious once we add the Plugin index but I can do it next.

chronik/chronik-http/src/server.rs
213 ↗(On Diff #46938)

Unfortunately this is not trivial because they live in separate indexes, but later it shouldn't be too hard to "merge" those two when querying what you ask for, since the history of a script/token ID/LOKAD ID are sorted (making looking up if a it O(log n)).

At some point we might wanna add a proper query language (uff), at least once we get like 3-5 more useful indices.

chronik/chronik-indexer/src/indexer.rs
214 ↗(On Diff #46938)

This is because the token ID index is always enabled if enable_token_index is true, whereas the LOKAD ID index has a bit more complex activation logic.

If indexing from genesis it's enabled, unless explicitly disabled, and if there's already a DB without the index, it's disabled, unless explicitly enabled.

Fabien requested changes to this revision.Apr 12 2024, 14:09
Fabien added inline comments.
chronik/chronik-cpp/chronik.cpp
57 ↗(On Diff #46938)

You should avoid this and move the logic in the node, in the InitParameterInteraction() method:

  1. The option lives in the node anyway
  2. There are more possible issues with these options that can be handled, e.g. if -chroniklokadidindex=1 but -chroniktokenindex=0 we should either warn and disable the lokad id index or silently enable the token indexing.
This revision now requires changes to proceed.Apr 12 2024, 14:09
chronik/chronik-cpp/chronik.cpp
57 ↗(On Diff #46938)

Hmm, how would the node know whether Chronik has a DB or not? Or should we instead just have everyone reindex on the next version and tell them to opt out if they don't want the index? It seems like a simple and straightforward solution to me.

Note: The chroniklokadidindex and chroniktokenindex are completely independent of each other

chronik/chronik-cpp/chronik.cpp
57 ↗(On Diff #46938)

You still have to pass whether it's activated or not, but you can avoid passing if it's set.
If you're not indexing tokens, indexing lokad ids seems useless to me. What would be the use case ?

chronik/chronik-cpp/chronik.cpp
57 ↗(On Diff #46938)

You still have to pass whether it's activated or not, but you can avoid passing if it's set.

I don't get how I can avoid passing if it's set or not in this situation and get the same behavoir. There's three states we need to encode: explicit chroniklokadidindex=1, explicit chroniklokadidindex=0, flag not set (and should be derived from the state of the DB).

If you're not indexing tokens, indexing lokad ids seems useless to me. What would be the use case ?

Stamp would be an example that would benefit from a LOKAD ID index without a token index, or the memo.cash system Ethan is working on. Many such cases—the LOKAD ID spec predates the SLP spec.

chronik/chronik-cpp/chronik.cpp
57 ↗(On Diff #46938)

I don't get how I can avoid passing if it's set or not in this situation and get the same behavoir. There's three states we need to encode: explicit chroniklokadidindex=1, explicit chroniklokadidindex=0, flag not set (and should be derived from the state of the DB).

If indexing from genesis it's enabled, unless explicitly disabled, and if there's already a DB without the index, it's disabled, unless explicitly enabled.

I think this is because this behavior is uncommon (unique if memory serves) and I missed it. Usually whatever the default is it takes precedence to whatever is in the db, which I think is fine. Either you reindex or you explicitely disable it (if it's on by default).

Please note that now that chronik is officially released, this kind of change deserves a release note.

Stamp would be an example that would benefit from a LOKAD ID index without a token index, or the memo.cash system Ethan is working on. Many such cases—the LOKAD ID spec predates the SLP spec.

OK if there are use cases for this then it's fine.

chronik/chronik-cpp/chronik.cpp
57 ↗(On Diff #46938)

Ok, I can change it to default to reindex and make sure that people are informed that it’ll do that

always enable LOKAD ID index by default, toggle only by -chroniklokadidindex

Failed tests logs:

====== Get blocktxs, txs, and history for SLP NFT1 token txs: "before each" hook for "Gets an SLP NFT1 send tx from the mempool".Get blocktxs, txs, and history for SLP NFT1 token txs "before each" hook for "Gets an SLP NFT1 send tx from the mempool" ======
Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/work/modules/chronik-client/test/integration/token_slp_nft1.ts)
    at listOnTimeout (node:internal/timers:573:17)
    at processTimers (node:internal/timers:514:7)

Each failure log is accessible here:
Get blocktxs, txs, and history for SLP NFT1 token txs: "before each" hook for "Gets an SLP NFT1 send tx from the mempool".Get blocktxs, txs, and history for SLP NFT1 token txs "before each" hook for "Gets an SLP NFT1 send tx from the mempool"

chronik/chronik-db/Cargo.toml
47 ↗(On Diff #47341)

This will enable compression ?

remove rocksdb features again

This revision is now accepted and ready to land.Apr 22 2024, 16:24
The build failed due to an unexpected infrastructure outage. The administrators have been notified to investigate. Sorry for the inconvenience.

Tail of the build log:

  File "/work/test/functional/setup_scripts/../test_framework/util.py", line 297, in wait_until_helper
    raise AssertionError(
AssertionError: Predicate ''''
            self.wait_until(lambda: is_finalblock(next_blockhash))
''' not true after 60.0 seconds
2024-04-22T16:49:35.331000Z TestFramework (INFO): Stopping nodes
2024-04-22T16:49:35.836000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240422_164822/setup_scripts/chronik-client_websocket_0
2024-04-22T16:49:35.836000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240422_164822/setup_scripts/chronik-client_websocket_0/test_framework.log
2024-04-22T16:49:35.837000Z TestFramework (ERROR): 
2024-04-22T16:49:35.837000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240422_164822/setup_scripts/chronik-client_websocket_0' to consolidate all logs
2024-04-22T16:49:35.837000Z TestFramework (ERROR): 
2024-04-22T16:49:35.838000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-04-22T16:49:35.838000Z TestFramework (ERROR): https://github.com/Bitcoin-ABC/bitcoin-abc/issues
2024-04-22T16:49:35.838000Z TestFramework (ERROR): 
Running Unit Tests for Test Framework Modules
setup_scripts/chronik-client_websocket.py started
setup_scripts/chronik-client_websocket.py failed, Duration: 74 s

stdout:

stderr:


TEST                                      | STATUS    | DURATION

setup_scripts/chronik-client_websocket.py | ✖ Failed  | 74 s

ALL                                       | ✖ Failed  | 74 s (accumulated) 
Runtime: 74 s

Test runner for chronik-client_websocket completed with code 1
-----------------------|---------|----------|---------|---------|-----------------------------------
File                   | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s                 
-----------------------|---------|----------|---------|---------|-----------------------------------
All files              |   27.35 |    10.37 |   23.06 |   27.36 |                                   
 chronik-client        |     100 |      100 |     100 |     100 |                                   
  index.ts             |     100 |      100 |     100 |     100 |                                   
 chronik-client/proto  |   21.43 |     7.91 |   13.72 |   21.57 |                                   
  chronik.ts           |     6.1 |        1 |    2.54 |    6.09 | ...,3978-3985,3990-4027,4031-4036 
  chronikNode.ts       |   32.95 |    13.21 |   23.49 |   33.19 | ...,4882-4921,4929-5002,5037-5042 
 chronik-client/src    |   62.93 |     42.4 |   60.54 |   62.56 |                                   
  ChronikClient.ts     |    4.24 |        0 |       0 |    4.29 | 33-163,178-222,290-692            
  ChronikClientNode.ts |   89.68 |    70.14 |   95.71 |   89.68 | ...52,876,888,898,923,935,941,947 
  failoverProxy.ts     |   76.19 |    59.09 |   66.66 |   75.49 | ...61-264,267,275-285,294,303,307 
  hex.ts               |   89.47 |       75 |      75 |   87.87 | 58,66-68                          
  validation.ts        |      90 |    84.61 |     100 |   89.47 | 31,37                             
-----------------------|---------|----------|---------|---------|-----------------------------------

##teamcity[blockOpened name='Code Coverage Summary']
##teamcity[buildStatisticValue key='CodeCoverageAbsBCovered' value='1120']
##teamcity[buildStatisticValue key='CodeCoverageAbsBTotal' value='4095']
##teamcity[buildStatisticValue key='CodeCoverageAbsRCovered' value='459']
##teamcity[buildStatisticValue key='CodeCoverageAbsRTotal' value='4422']
##teamcity[buildStatisticValue key='CodeCoverageAbsMCovered' value='170']
##teamcity[buildStatisticValue key='CodeCoverageAbsMTotal' value='737']
##teamcity[buildStatisticValue key='CodeCoverageAbsLCovered' value='1110']
##teamcity[buildStatisticValue key='CodeCoverageAbsLTotal' value='4056']
##teamcity[blockClosed name='Code Coverage Summary']
mv: cannot stat 'test_results/chronik-client-integration-tests-junit.xml': No such file or directory
Build chronik-client-integration-tests failed with exit code 1

rerun CI because cash is supposed to be green

The build failed due to an unexpected infrastructure outage. The administrators have been notified to investigate. Sorry for the inconvenience.

Updated chronik.pay2stay.com/xec to this diff, upgrade took 3h:05m, so not too fast but not too slow.

Querying the confirmed history for "SLP\0" gives these txs, which seem like the first 25 SLP txs (which could be useful for future reference, too).

['545cba6f72a08cbcb08c7d4e8166267942e8cb9a611328805c62fa538e861ba4', '83bfe019fcf976142c55e7c0ad4a429f4be1bc2cb138bd8d0bab8dd4cd4758c4', '4f035d656ed5b6e94a884c88c09a8d2dee9c7e97901cce3adec966115e2a1ba5', '323a1e35ae0b356316093d20f2d9fbc995d19314b5c0148b78dc8d9c0dab9d35', '874306bda204d3a5dd15e03ea5732cccdca4c33a52df35162cdd64e30ea7f04e', '3a574837eaecf701ddf232ab4ad30c541be61061ee5a563d27809dde927d7ecf', '82a9c47118dd221bf528e8b9ee9daef626ca52fb824b92cbe52a83e87afb0fac', '170147548aad6de7c1df686c56e4846e0936c4573411b604a18d0ec76482dde2', 'e5ff3083cd2dcf87a40a4a4a478349a394c1a1eeffe4857c2a173b183fdd42a2', 'f56121d5a21a319204cf26ce68a6d607fefa02ba6ac42b4647fcad813b32d8b3', '660057b446cc4c930493607aa02e943e4fe7c38ae0816797ff7234ba72fea50f', '483d0198ed272bd0be7c6bbaf0e60340cce926f7d32143e2b09c5513922eaf87', '234893177b18a95dbfc1eb855d69f1c9cc256a317a6c51be8fd1b9a38ae072ce', 'a333e7ebd34f0e24b567e99ed27241e3cfda5e9952cacdaa8fab31a7ee7e544d', '89f4a12d7de52e3d117ef67e9e9be45eb9a1432afee61d99efc3576df2c7fab2', '938cc18e618967d787897bbc64b9a8d201b94ec7c69b1a9949eab0433ba5cdf8', 'ee9d3cf5153599c134147e3fac9844c68e216843f4452a1ce15a29452af6db34', '4640a734063ea79fa587a3cac38a70a2f6f3db0011e23514024185982110d0fa', 'e0f3e9ee5ab30c250c0d5fbd78c1eaedf3dfd998813ecbf29ee4c00dfb9acf12', '662bc5b6fe6fa2ab7ee3257ee31549d8490f3a7d591c52eecc053e97fc4c3a1c', 'f58c7bb144f24385c341687c725c095af274fe16666b2903b6e061f18d895ec5', '34b38d6751f38d6597eb9dbd74e009e98434b0fac6ed95637fc92aa6d7f8eb79', '550d19eb820e616a54b8a73372c4420b5a0567d8dc00f613b71c5234dc884b35', '95d460512143b636bbc5780d8b27b04fca3bfd2f22003ab48da594e2bab9cfc1', 'b36b0c7485ad569b98cc9b9614dc68a5208495f22ec3b00effcf963b135a5215']

So, seems like everything works in production, too. Will land right now.

rebase onto master to fix build?

Failed tests logs:

====== TxBuilder: "before all" hook for "TxBuilder P2PKH Wallet".TxBuilder "before all" hook for "TxBuilder P2PKH Wallet" ======
Error: Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/work/modules/ecash-lib/tests/txBuilder.test.ts)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)

Each failure log is accessible here:
TxBuilder: "before all" hook for "TxBuilder P2PKH Wallet".TxBuilder "before all" hook for "TxBuilder P2PKH Wallet"

Failed tests logs:

====== Bitcoin ABC functional tests: chronik_token_alp.py ======

------- Stdout: -------
2024-04-23T22:14:59.795000Z TestFramework (INFO): Initializing test directory /work/abc-ci-builds/build-chronik-plugins/test/tmp/test_runner_₿₵_  _20240423_220940/chronik_token_alp_139
2024-04-23T22:15:31.381000Z TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 627, in start_nodes
    node.wait_for_rpc_connection()
  File "/work/test/functional/test_framework/test_node.py", line 348, in wait_for_rpc_connection
    raise FailedToStartError(
test_framework.test_node.FailedToStartError: [node 0] bitcoind exited with status 1 during initialization

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 146, in main
    self.setup()
  File "/work/test/functional/test_framework/test_framework.py", line 381, in setup
    self.setup_network()
  File "/work/test/functional/test_framework/test_framework.py", line 488, in setup_network
    self.setup_nodes()
  File "/work/test/functional/test_framework/test_framework.py", line 513, in setup_nodes
    self.start_nodes()
  File "/work/test/functional/test_framework/test_framework.py", line 630, in start_nodes
    self.stop_nodes()
  File "/work/test/functional/test_framework/test_framework.py", line 645, in stop_nodes
    node.stop_node(wait=wait, wait_until_stopped=False)
  File "/work/test/functional/test_framework/test_node.py", line 501, in stop_node
    self.stop(wait=wait)
  File "/work/test/functional/test_framework/test_node.py", line 278, in __getattr__
    assert self.rpc is not None, self._node_msg("Error: RPC not initialized")
AssertionError: [node 0] Error: RPC not initialized
2024-04-23T22:15:31.432000Z TestFramework (INFO): Stopping nodes
[node 0] Cleaning up leftover process
------- Stderr: -------
Traceback (most recent call last):
  File "/work/test/functional/chronik_token_alp.py", line 666, in <module>
    ChronikTokenAlp().main()
  File "/work/test/functional/test_framework/test_framework.py", line 170, in main
    exit_code = self.shutdown()
  File "/work/test/functional/test_framework/test_framework.py", line 396, in shutdown
    self.stop_nodes()
  File "/work/test/functional/test_framework/test_framework.py", line 645, in stop_nodes
    node.stop_node(wait=wait, wait_until_stopped=False)
  File "/work/test/functional/test_framework/test_node.py", line 501, in stop_node
    self.stop(wait=wait)
  File "/work/test/functional/test_framework/test_node.py", line 278, in __getattr__
    assert self.rpc is not None, self._node_msg("Error: RPC not initialized")
AssertionError: [node 0] Error: RPC not initialized

Each failure log is accessible here:
Bitcoin ABC functional tests: chronik_token_alp.py

rerun CI, last one got Error: Chronik failed binding to 127.0.0.1:33962: Address already in use (os error 98)

This revision was automatically updated to reflect the committed changes.