Page MenuHomePhabricator

[chronik-client] Support lokad id endpoints
ClosedPublic

Authored by bytesofman on Tue, Apr 30, 23:56.

Details

Reviewers
tobias_ruck
Group Reviewers
Restricted Project
Commits
rABC893e80e4a44a: [chronik-client] Support lokad id endpoints
Summary

Support new lokad id endpoints for chronik instances that index them

Test Plan

CI

Diff Detail

Repository
rABC Bitcoin ABC
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Failed tests logs:

====== Get blocktxs, txs, and history for SLP 2 mint vault token txs: "before each" hook for "Gets a badly constructed SLP v2 Vault Mint tx from the mempool".Get blocktxs, txs, and history for SLP 2 mint vault token txs "before each" hook for "Gets a badly constructed SLP v2 Vault Mint tx from the mempool" ======
Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/work/modules/chronik-client/test/integration/token_slp_mint_vault.ts)
    at listOnTimeout (node:internal/timers:573:17)
    at processTimers (node:internal/timers:514:7)

Each failure log is accessible here:
Get blocktxs, txs, and history for SLP 2 mint vault token txs: "before each" hook for "Gets a badly constructed SLP v2 Vault Mint tx from the mempool".Get blocktxs, txs, and history for SLP 2 mint vault token txs "before each" hook for "Gets a badly constructed SLP v2 Vault Mint tx from the mempool"

commenced converting chronik_lokad_id_group.py to chronik-client test

progress on tests, some TODOs to go

Failed tests logs:

====== History endpoints and websocket for LOKAD ID: Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN.History endpoints and websocket for LOKAD ID Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN ======
AssertionError: expected [] to deeply equal [ { type: 'Tx', …(2) }, …(1) ]
    at /work/modules/chronik-client/test/integration/lokad_id.ts:328:47
    at Generator.next (<anonymous>)
    at fulfilled (test/integration/lokad_id.ts:31:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

      + expected - actual

      -[]
      +[
      +  {
      +    "msgType": "TX_ADDED_TO_MEMPOOL"
      +    "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
      +    "type": "Tx"
      +  }
      +  {
      +    "msgType": "TX_ADDED_TO_MEMPOOL"
      +    "txid": "a989e07ae47f3925e911942e6dfbb0a5217fbd6617975b5c7a03a235c0c8a554"
      +    "type": "Tx"
      +  }
      +]
====== History endpoints and websocket for LOKAD ID: After these txs are mined, chronik-client returns this info appropriately.History endpoints and websocket for LOKAD ID After these txs are mined, chronik-client returns this info appropriately ======
AssertionError: expected [ { type: 'Tx', …(2) }, …(4) ] to deeply equal [ { type: 'Tx', …(2) }, …(2) ]
    at /work/modules/chronik-client/test/integration/lokad_id.ts:397:47
    at Generator.next (<anonymous>)
    at fulfilled (test/integration/lokad_id.ts:31:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

      + expected - actual

       [
         {
      -    "msgType": "TX_ADDED_TO_MEMPOOL"
      -    "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_ADDED_TO_MEMPOOL"
      -    "txid": "a989e07ae47f3925e911942e6dfbb0a5217fbd6617975b5c7a03a235c0c8a554"
      -    "type": "Tx"
      -  }
      -  {
           "msgType": "TX_CONFIRMED"
           "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
           "type": "Tx"
         }

Each failure log is accessible here:
History endpoints and websocket for LOKAD ID: Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN.History endpoints and websocket for LOKAD ID Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN
History endpoints and websocket for LOKAD ID: After these txs are mined, chronik-client returns this info appropriately.History endpoints and websocket for LOKAD ID After these txs are mined, chronik-client returns this info appropriately

test outline present, need to deal with websocket msgs coming late (well, prob not late, but after expected step)

Failed tests logs:

====== History endpoints and websocket for LOKAD ID: Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN.History endpoints and websocket for LOKAD ID Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN ======
AssertionError: expected [] to deeply equal [ { type: 'Tx', …(2) }, …(1) ]
    at /work/modules/chronik-client/test/integration/lokad_id.ts:326:47
    at Generator.next (<anonymous>)
    at fulfilled (test/integration/lokad_id.ts:31:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

      + expected - actual

      -[]
      +[
      +  {
      +    "msgType": "TX_ADDED_TO_MEMPOOL"
      +    "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
      +    "type": "Tx"
      +  }
      +  {
      +    "msgType": "TX_ADDED_TO_MEMPOOL"
      +    "txid": "a989e07ae47f3925e911942e6dfbb0a5217fbd6617975b5c7a03a235c0c8a554"
      +    "type": "Tx"
      +  }
      +]
====== History endpoints and websocket for LOKAD ID: After these txs are mined, chronik-client returns this info appropriately.History endpoints and websocket for LOKAD ID After these txs are mined, chronik-client returns this info appropriately ======
AssertionError: expected [] to deeply equal [ { type: 'Tx', …(2) }, …(2) ]
    at /work/modules/chronik-client/test/integration/lokad_id.ts:388:47
    at Generator.next (<anonymous>)
    at fulfilled (test/integration/lokad_id.ts:31:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

      + expected - actual

      -[]
      +[
      +  {
      +    "msgType": "TX_CONFIRMED"
      +    "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
      +    "type": "Tx"
      +  }
      +  {
      +    "msgType": "TX_CONFIRMED"
      +    "txid": "963850dd9433358993b41184960ea73ee24c754fbccb512ff45976b594cb8876"
      +    "type": "Tx"
      +  }
      +  {
      +    "msgType": "TX_CONFIRMED"
      +    "txid": "a989e07ae47f3925e911942e6dfbb0a5217fbd6617975b5c7a03a235c0c8a554"
      +    "type": "Tx"
      +  }
      +]
====== History endpoints and websocket for LOKAD ID: We can detect a mix of confirmed and unconfirmed txs by lokad id.History endpoints and websocket for LOKAD ID We can detect a mix of confirmed and unconfirmed txs by lokad id ======
AssertionError: expected [ { type: 'Tx', …(2) }, …(5) ] to deeply equal [ { type: 'Tx', …(2) } ]
    at /work/modules/chronik-client/test/integration/lokad_id.ts:472:47
    at Generator.next (<anonymous>)
    at fulfilled (test/integration/lokad_id.ts:31:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

      + expected - actual

       [
         {
           "msgType": "TX_ADDED_TO_MEMPOOL"
      -    "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_ADDED_TO_MEMPOOL"
      -    "txid": "a989e07ae47f3925e911942e6dfbb0a5217fbd6617975b5c7a03a235c0c8a554"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_CONFIRMED"
      -    "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_CONFIRMED"
      -    "txid": "963850dd9433358993b41184960ea73ee24c754fbccb512ff45976b594cb8876"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_CONFIRMED"
      -    "txid": "a989e07ae47f3925e911942e6dfbb0a5217fbd6617975b5c7a03a235c0c8a554"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_ADDED_TO_MEMPOOL"
           "txid": "5d8e3f080fa9f7b399be2abd5ab78c9cd6857daadc0f84db1dcfd308ef209080"
           "type": "Tx"
         }
       ]

Each failure log is accessible here:
History endpoints and websocket for LOKAD ID: Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN.History endpoints and websocket for LOKAD ID Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN
History endpoints and websocket for LOKAD ID: After these txs are mined, chronik-client returns this info appropriately.History endpoints and websocket for LOKAD ID After these txs are mined, chronik-client returns this info appropriately
History endpoints and websocket for LOKAD ID: We can detect a mix of confirmed and unconfirmed txs by lokad id.History endpoints and websocket for LOKAD ID We can detect a mix of confirmed and unconfirmed txs by lokad id

still some work to do on tests and timing

Failed tests logs:

====== History endpoints and websocket for LOKAD ID: After these txs are mined, chronik-client returns this info appropriately.History endpoints and websocket for LOKAD ID After these txs are mined, chronik-client returns this info appropriately ======
AssertionError: expected [ { type: 'Tx', …(2) } ] to deeply equal [ { type: 'Tx', …(2) }, …(2) ]
    at /work/modules/chronik-client/test/integration/lokad_id.ts:413:59
    at Generator.next (<anonymous>)
    at fulfilled (test/integration/lokad_id.ts:31:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

      + expected - actual

       [
         {
      +    "msgType": "TX_ADDED_TO_MEMPOOL"
      +    "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
      +    "type": "Tx"
      +  }
      +  {
      +    "msgType": "TX_ADDED_TO_MEMPOOL"
      +    "txid": "a989e07ae47f3925e911942e6dfbb0a5217fbd6617975b5c7a03a235c0c8a554"
      +    "type": "Tx"
      +  }
      +  {
           "msgType": "TX_CONFIRMED"
           "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
           "type": "Tx"
         }
====== History endpoints and websocket for LOKAD ID: We can detect a mix of confirmed and unconfirmed txs by lokad id.History endpoints and websocket for LOKAD ID We can detect a mix of confirmed and unconfirmed txs by lokad id ======
AssertionError: expected [ { type: 'Tx', …(2) }, …(5) ] to deeply equal [ { type: 'Tx', …(2) } ]
    at /work/modules/chronik-client/test/integration/lokad_id.ts:490:47
    at Generator.next (<anonymous>)
    at fulfilled (test/integration/lokad_id.ts:31:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

      + expected - actual

       [
         {
           "msgType": "TX_ADDED_TO_MEMPOOL"
      -    "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_ADDED_TO_MEMPOOL"
      -    "txid": "a989e07ae47f3925e911942e6dfbb0a5217fbd6617975b5c7a03a235c0c8a554"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_CONFIRMED"
      -    "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_CONFIRMED"
      -    "txid": "963850dd9433358993b41184960ea73ee24c754fbccb512ff45976b594cb8876"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_CONFIRMED"
      -    "txid": "a989e07ae47f3925e911942e6dfbb0a5217fbd6617975b5c7a03a235c0c8a554"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_ADDED_TO_MEMPOOL"
           "txid": "5d8e3f080fa9f7b399be2abd5ab78c9cd6857daadc0f84db1dcfd308ef209080"
           "type": "Tx"
         }
       ]

Each failure log is accessible here:
History endpoints and websocket for LOKAD ID: After these txs are mined, chronik-client returns this info appropriately.History endpoints and websocket for LOKAD ID After these txs are mined, chronik-client returns this info appropriately
History endpoints and websocket for LOKAD ID: We can detect a mix of confirmed and unconfirmed txs by lokad id.History endpoints and websocket for LOKAD ID We can detect a mix of confirmed and unconfirmed txs by lokad id

tests behave as expected, though we still have some flakiness

reduce flakiness by using const instead of ipc_msg for txids

remove unnecessary chronik client redefinitions, update tokenId copypasta to lokadId

bytesofman published this revision for review.Thu, May 2, 12:42
bytesofman added inline comments.
modules/chronik-client/proto/chronikNode.ts
660 ↗(On Diff #47562)

note that this file is generated by running npm run build-proto

modules/chronik-client/src/ChronikClientNode.ts
374 ↗(On Diff #47562)

tempting to abstract this as it is *almost* the same as ScriptEndpoint and also TokenIdEndpoint

subtle differences remain though that imo would force too many shims into the abstraction

  • TokenIdEndpoint returns utxos in a different shape from ScriptEndpoint
  • LokadIdEndpoint does not have a utxos method
modules/chronik-client/test/integration/lokad_id.ts
115 ↗(On Diff #47562)

We have an interesting model in this diff for some tactics to improve flakiness of existing tests

This diff had repeatable flakiness locally. Websocket msgs would frequently (about 50% of the time) come at wrong steps.

This was reduced by asserting websocket behavior in the py setup script.

Further reduced (haven't repeated it since 🤞 nm still get this though less often) by removing ipc message sends for txids.

The py script approach to txids is what should be used as a model going forward. (use node mocktime, but increment it by 1 in between txs so we can effectively test tx order for txs that are expected to be sorted by timeFirstSeen).

221 ↗(On Diff #47562)

"in a spend input"

I'm not really sure if this is what is going on here or if I am describing it appropriately

test/functional/setup_scripts/chronik-client_lokad_id.py
110 ↗(On Diff #47562)

++

really good idea for chronik tests. Copying it here from the chronik lokad test. Should be tried for other chronik-client tests where we are observing flakiness.

112 ↗(On Diff #47562)

what is going on with this lok1? I'm not sure I am interpreting it correctly.

It's getting indexed as a spent input? How would this come up in on-chain eCash use?

Failed tests logs:

====== History endpoints and websocket for LOKAD ID: Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and the same lokadId in OP_RETURN (non-EMPP).History endpoints and websocket for LOKAD ID Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and the same lokadId in OP_RETURN (non-EMPP) ======
AssertionError: expected [] to deeply equal [ { type: 'Tx', …(2) } ]
    at /work/modules/chronik-client/test/integration/lokad_id.ts:279:47
    at Generator.next (<anonymous>)
    at fulfilled (test/integration/lokad_id.ts:31:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

      + expected - actual

      -[]
      +[
      +  {
      +    "msgType": "TX_ADDED_TO_MEMPOOL"
      +    "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
      +    "type": "Tx"
      +  }
      +]
====== History endpoints and websocket for LOKAD ID: Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN.History endpoints and websocket for LOKAD ID Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN ======
AssertionError: expected [] to deeply equal [ { type: 'Tx', …(2) } ]
    at /work/modules/chronik-client/test/integration/lokad_id.ts:334:47
    at Generator.next (<anonymous>)
    at fulfilled (test/integration/lokad_id.ts:31:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

      + expected - actual

      -[]
      +[
      +  {
      +    "msgType": "TX_ADDED_TO_MEMPOOL"
      +    "txid": "a989e07ae47f3925e911942e6dfbb0a5217fbd6617975b5c7a03a235c0c8a554"
      +    "type": "Tx"
      +  }
      +]
====== History endpoints and websocket for LOKAD ID: After these txs are mined, chronik-client returns this info appropriately.History endpoints and websocket for LOKAD ID After these txs are mined, chronik-client returns this info appropriately ======
AssertionError: expected [] to deeply equal [ { type: 'Tx', …(2) }, …(2) ]
    at /work/modules/chronik-client/test/integration/lokad_id.ts:393:47
    at Generator.next (<anonymous>)
    at fulfilled (test/integration/lokad_id.ts:31:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

      + expected - actual

      -[]
      +[
      +  {
      +    "msgType": "TX_CONFIRMED"
      +    "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
      +    "type": "Tx"
      +  }
      +  {
      +    "msgType": "TX_CONFIRMED"
      +    "txid": "963850dd9433358993b41184960ea73ee24c754fbccb512ff45976b594cb8876"
      +    "type": "Tx"
      +  }
      +  {
      +    "msgType": "TX_CONFIRMED"
      +    "txid": "a989e07ae47f3925e911942e6dfbb0a5217fbd6617975b5c7a03a235c0c8a554"
      +    "type": "Tx"
      +  }
      +]
====== History endpoints and websocket for LOKAD ID: We can detect a mix of confirmed and unconfirmed txs by lokad id.History endpoints and websocket for LOKAD ID We can detect a mix of confirmed and unconfirmed txs by lokad id ======
AssertionError: expected [ { type: 'Tx', …(2) }, …(5) ] to deeply equal [ { type: 'Tx', …(2) } ]
    at /work/modules/chronik-client/test/integration/lokad_id.ts:472:47
    at Generator.next (<anonymous>)
    at fulfilled (test/integration/lokad_id.ts:31:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

      + expected - actual

       [
         {
           "msgType": "TX_ADDED_TO_MEMPOOL"
      -    "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_ADDED_TO_MEMPOOL"
      -    "txid": "a989e07ae47f3925e911942e6dfbb0a5217fbd6617975b5c7a03a235c0c8a554"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_CONFIRMED"
      -    "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_CONFIRMED"
      -    "txid": "963850dd9433358993b41184960ea73ee24c754fbccb512ff45976b594cb8876"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_CONFIRMED"
      -    "txid": "a989e07ae47f3925e911942e6dfbb0a5217fbd6617975b5c7a03a235c0c8a554"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_ADDED_TO_MEMPOOL"
           "txid": "5d8e3f080fa9f7b399be2abd5ab78c9cd6857daadc0f84db1dcfd308ef209080"
           "type": "Tx"
         }
       ]

Each failure log is accessible here:
History endpoints and websocket for LOKAD ID: Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and the same lokadId in OP_RETURN (non-EMPP).History endpoints and websocket for LOKAD ID Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and the same lokadId in OP_RETURN (non-EMPP)
History endpoints and websocket for LOKAD ID: Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN.History endpoints and websocket for LOKAD ID Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN
History endpoints and websocket for LOKAD ID: After these txs are mined, chronik-client returns this info appropriately.History endpoints and websocket for LOKAD ID After these txs are mined, chronik-client returns this info appropriately
History endpoints and websocket for LOKAD ID: We can detect a mix of confirmed and unconfirmed txs by lokad id.History endpoints and websocket for LOKAD ID We can detect a mix of confirmed and unconfirmed txs by lokad id

version bump, readme changelog

Tail of the build log:

  File "/work/test/functional/setup_scripts/../test_framework/util.py", line 297, in wait_until_helper
    raise AssertionError(
AssertionError: Predicate ''''
            self.wait_until(lambda: is_finalblock(next_blockhash))
''' not true after 60.0 seconds
2024-05-02T13:11:58.129000Z TestFramework (INFO): Stopping nodes
2024-05-02T13:11:58.786000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240502_131045/setup_scripts/chronik-client_websocket_0
2024-05-02T13:11:58.786000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240502_131045/setup_scripts/chronik-client_websocket_0/test_framework.log
2024-05-02T13:11:58.786000Z TestFramework (ERROR): 
2024-05-02T13:11:58.787000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240502_131045/setup_scripts/chronik-client_websocket_0' to consolidate all logs
2024-05-02T13:11:58.787000Z TestFramework (ERROR): 
2024-05-02T13:11:58.787000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-05-02T13:11:58.788000Z TestFramework (ERROR): https://github.com/Bitcoin-ABC/bitcoin-abc/issues
2024-05-02T13:11:58.788000Z TestFramework (ERROR): 
Running Unit Tests for Test Framework Modules
setup_scripts/chronik-client_websocket.py started
setup_scripts/chronik-client_websocket.py failed, Duration: 73 s

stdout:

stderr:


TEST                                      | STATUS    | DURATION

setup_scripts/chronik-client_websocket.py | ✖ Failed  | 73 s

ALL                                       | ✖ Failed  | 73 s (accumulated) 
Runtime: 73 s

Test runner for chronik-client_websocket completed with code 1
-----------------------|---------|----------|---------|---------|-----------------------------------
File                   | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s                 
-----------------------|---------|----------|---------|---------|-----------------------------------
All files              |   27.61 |    10.72 |   23.57 |   27.61 |                                   
 chronik-client        |     100 |      100 |     100 |     100 |                                   
  index.ts             |     100 |      100 |     100 |     100 |                                   
 chronik-client/proto  |    21.2 |     7.88 |    13.4 |   21.34 |                                   
  chronik.ts           |     6.1 |        1 |    2.54 |    6.09 | ...,3978-3985,3990-4027,4031-4036 
  chronikNode.ts       |   32.33 |    13.08 |   22.67 |   32.57 | ...,4991-5030,5038-5111,5146-5151 
 chronik-client/src    |   64.39 |    45.29 |   62.02 |   64.01 |                                   
  ChronikClient.ts     |    4.24 |        0 |       0 |    4.29 | 33-163,178-222,290-692            
  ChronikClientNode.ts |   89.64 |    71.79 |   94.93 |   89.64 | ...,1018,1028,1053,1065,1071,1077 
  failoverProxy.ts     |    75.7 |    59.09 |   64.28 |      75 | ...64,267,275-285,294,301,307,311 
  hex.ts               |   89.47 |       75 |      75 |   87.87 | 58,66-68                          
  validation.ts        |      92 |    86.66 |     100 |    91.3 | 32,38                             
-----------------------|---------|----------|---------|---------|-----------------------------------

##teamcity[blockOpened name='Code Coverage Summary']
##teamcity[buildStatisticValue key='CodeCoverageAbsBCovered' value='1151']
##teamcity[buildStatisticValue key='CodeCoverageAbsBTotal' value='4168']
##teamcity[buildStatisticValue key='CodeCoverageAbsRCovered' value='481']
##teamcity[buildStatisticValue key='CodeCoverageAbsRTotal' value='4485']
##teamcity[buildStatisticValue key='CodeCoverageAbsMCovered' value='178']
##teamcity[buildStatisticValue key='CodeCoverageAbsMTotal' value='755']
##teamcity[buildStatisticValue key='CodeCoverageAbsLCovered' value='1140']
##teamcity[buildStatisticValue key='CodeCoverageAbsLTotal' value='4128']
##teamcity[blockClosed name='Code Coverage Summary']
mv: cannot stat 'test_results/chronik-client-integration-tests-junit.xml': No such file or directory
Build chronik-client-integration-tests failed with exit code 1
emack added inline comments.
modules/chronik-client/src/ChronikClientNode.ts
619 ↗(On Diff #47572)

should this callsite catch the exception and then return so it doesn't continue onto the lines below?

632 ↗(On Diff #47572)

Is there a reason verifyLokadId is not being called here? If it's invalid then you can avoid traversing through lokadIds and return early.

bytesofman added inline comments.
modules/chronik-client/src/ChronikClientNode.ts
619 ↗(On Diff #47572)

It will throw an error if invalid, so we won't continue onto the lines below

added integration test to confirm error sub is not added to ws.subs.lokadIds

632 ↗(On Diff #47572)

this function only deals with stuff that has been successfully added to ws.subs.lokadIds -- since we verify on the way in, we cannot have anything invalid here.

The validation condition here is more appropriate -- we can only unsub to something that exists here.

bytesofman marked 2 inline comments as done.

confirm error lokad id sub is not added to ws.subs.lokadIds

Failed tests logs:

====== History endpoints and websocket for LOKAD ID: Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and the same lokadId in OP_RETURN (non-EMPP).History endpoints and websocket for LOKAD ID Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and the same lokadId in OP_RETURN (non-EMPP) ======
AssertionError: expected [] to deeply equal [ { type: 'Tx', …(2) } ]
    at /work/modules/chronik-client/test/integration/lokad_id.ts:256:47
    at Generator.next (<anonymous>)
    at fulfilled (test/integration/lokad_id.ts:31:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

      + expected - actual

      -[]
      +[
      +  {
      +    "msgType": "TX_ADDED_TO_MEMPOOL"
      +    "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
      +    "type": "Tx"
      +  }
      +]
====== History endpoints and websocket for LOKAD ID: Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN.History endpoints and websocket for LOKAD ID Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN ======
AssertionError: expected [] to deeply equal [ { type: 'Tx', …(2) } ]
    at /work/modules/chronik-client/test/integration/lokad_id.ts:309:47
    at Generator.next (<anonymous>)
    at fulfilled (test/integration/lokad_id.ts:31:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

      + expected - actual

      -[]
      +[
      +  {
      +    "msgType": "TX_ADDED_TO_MEMPOOL"
      +    "txid": "a989e07ae47f3925e911942e6dfbb0a5217fbd6617975b5c7a03a235c0c8a554"
      +    "type": "Tx"
      +  }
      +]
====== History endpoints and websocket for LOKAD ID: After these txs are mined, chronik-client returns this info appropriately.History endpoints and websocket for LOKAD ID After these txs are mined, chronik-client returns this info appropriately ======
AssertionError: expected [] to deeply equal [ { type: 'Tx', …(2) }, …(2) ]
    at /work/modules/chronik-client/test/integration/lokad_id.ts:368:47
    at Generator.next (<anonymous>)
    at fulfilled (test/integration/lokad_id.ts:31:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

      + expected - actual

      -[]
      +[
      +  {
      +    "msgType": "TX_CONFIRMED"
      +    "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
      +    "type": "Tx"
      +  }
      +  {
      +    "msgType": "TX_CONFIRMED"
      +    "txid": "963850dd9433358993b41184960ea73ee24c754fbccb512ff45976b594cb8876"
      +    "type": "Tx"
      +  }
      +  {
      +    "msgType": "TX_CONFIRMED"
      +    "txid": "a989e07ae47f3925e911942e6dfbb0a5217fbd6617975b5c7a03a235c0c8a554"
      +    "type": "Tx"
      +  }
      +]
====== History endpoints and websocket for LOKAD ID: We can detect a mix of confirmed and unconfirmed txs by lokad id.History endpoints and websocket for LOKAD ID We can detect a mix of confirmed and unconfirmed txs by lokad id ======
AssertionError: expected [ { type: 'Tx', …(2) }, …(5) ] to deeply equal [ { type: 'Tx', …(2) } ]
    at /work/modules/chronik-client/test/integration/lokad_id.ts:443:47
    at Generator.next (<anonymous>)
    at fulfilled (test/integration/lokad_id.ts:31:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

      + expected - actual

       [
         {
           "msgType": "TX_ADDED_TO_MEMPOOL"
      -    "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_ADDED_TO_MEMPOOL"
      -    "txid": "a989e07ae47f3925e911942e6dfbb0a5217fbd6617975b5c7a03a235c0c8a554"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_CONFIRMED"
      -    "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_CONFIRMED"
      -    "txid": "963850dd9433358993b41184960ea73ee24c754fbccb512ff45976b594cb8876"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_CONFIRMED"
      -    "txid": "a989e07ae47f3925e911942e6dfbb0a5217fbd6617975b5c7a03a235c0c8a554"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_ADDED_TO_MEMPOOL"
           "txid": "5d8e3f080fa9f7b399be2abd5ab78c9cd6857daadc0f84db1dcfd308ef209080"
           "type": "Tx"
         }
       ]

Each failure log is accessible here:
History endpoints and websocket for LOKAD ID: Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and the same lokadId in OP_RETURN (non-EMPP).History endpoints and websocket for LOKAD ID Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and the same lokadId in OP_RETURN (non-EMPP)
History endpoints and websocket for LOKAD ID: Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN.History endpoints and websocket for LOKAD ID Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN
History endpoints and websocket for LOKAD ID: After these txs are mined, chronik-client returns this info appropriately.History endpoints and websocket for LOKAD ID After these txs are mined, chronik-client returns this info appropriately
History endpoints and websocket for LOKAD ID: We can detect a mix of confirmed and unconfirmed txs by lokad id.History endpoints and websocket for LOKAD ID We can detect a mix of confirmed and unconfirmed txs by lokad id

Tail of the build log:

  File "/work/test/functional/setup_scripts/../test_framework/util.py", line 297, in wait_until_helper
    raise AssertionError(
AssertionError: Predicate ''''
            self.wait_until(lambda: is_finalblock(next_blockhash))
''' not true after 60.0 seconds
2024-05-02T14:44:26.049000Z TestFramework (INFO): Stopping nodes
2024-05-02T14:44:26.505000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240502_144313/setup_scripts/chronik-client_websocket_0
2024-05-02T14:44:26.505000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240502_144313/setup_scripts/chronik-client_websocket_0/test_framework.log
2024-05-02T14:44:26.505000Z TestFramework (ERROR): 
2024-05-02T14:44:26.506000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/chronik-client-integration-tests/test/tmp/test_runner_₿₵_🏃_20240502_144313/setup_scripts/chronik-client_websocket_0' to consolidate all logs
2024-05-02T14:44:26.506000Z TestFramework (ERROR): 
2024-05-02T14:44:26.506000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-05-02T14:44:26.507000Z TestFramework (ERROR): https://github.com/Bitcoin-ABC/bitcoin-abc/issues
2024-05-02T14:44:26.507000Z TestFramework (ERROR): 
Running Unit Tests for Test Framework Modules
setup_scripts/chronik-client_websocket.py started
setup_scripts/chronik-client_websocket.py failed, Duration: 73 s

stdout:

stderr:


TEST                                      | STATUS    | DURATION

setup_scripts/chronik-client_websocket.py | ✖ Failed  | 73 s

ALL                                       | ✖ Failed  | 73 s (accumulated) 
Runtime: 73 s

Test runner for chronik-client_websocket completed with code 1
-----------------------|---------|----------|---------|---------|-----------------------------------
File                   | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s                 
-----------------------|---------|----------|---------|---------|-----------------------------------
All files              |   27.95 |    10.81 |   23.97 |   27.95 |                                   
 chronik-client        |     100 |      100 |     100 |     100 |                                   
  index.ts             |     100 |      100 |     100 |     100 |                                   
 chronik-client/proto  |   21.51 |     7.98 |   13.73 |   21.65 |                                   
  chronik.ts           |     6.1 |        1 |    2.54 |    6.09 | ...,3978-3985,3990-4027,4031-4036 
  chronikNode.ts       |   32.87 |    13.25 |   23.29 |   33.11 | ...,4991-5030,5038-5111,5146-5151 
 chronik-client/src    |   64.87 |    45.29 |   62.65 |   64.51 |                                   
  ChronikClient.ts     |    4.24 |        0 |       0 |    4.29 | 33-163,178-222,290-692            
  ChronikClientNode.ts |   90.71 |    71.79 |    96.2 |   90.71 | ...,1018,1028,1053,1065,1071,1077 
  failoverProxy.ts     |    75.7 |    59.09 |   64.28 |      75 | ...64,267,275-285,294,301,307,311 
  hex.ts               |   89.47 |       75 |      75 |   87.87 | 58,66-68                          
  validation.ts        |      92 |    86.66 |     100 |    91.3 | 32,38                             
-----------------------|---------|----------|---------|---------|-----------------------------------

##teamcity[blockOpened name='Code Coverage Summary']
##teamcity[buildStatisticValue key='CodeCoverageAbsBCovered' value='1165']
##teamcity[buildStatisticValue key='CodeCoverageAbsBTotal' value='4168']
##teamcity[buildStatisticValue key='CodeCoverageAbsRCovered' value='485']
##teamcity[buildStatisticValue key='CodeCoverageAbsRTotal' value='4485']
##teamcity[buildStatisticValue key='CodeCoverageAbsMCovered' value='181']
##teamcity[buildStatisticValue key='CodeCoverageAbsMTotal' value='755']
##teamcity[buildStatisticValue key='CodeCoverageAbsLCovered' value='1154']
##teamcity[buildStatisticValue key='CodeCoverageAbsLTotal' value='4128']
##teamcity[blockClosed name='Code Coverage Summary']
mv: cannot stat 'test_results/chronik-client-integration-tests-junit.xml': No such file or directory
Build chronik-client-integration-tests failed with exit code 1

rebase on flaky test fix diff

tobias_ruck added inline comments.
test/functional/setup_scripts/chronik-client_lokad_id.py
57 ↗(On Diff #47578)

is there a reason to do this?

bytesofman added inline comments.
test/functional/setup_scripts/chronik-client_lokad_id.py
57 ↗(On Diff #47578)

Initially I did not do this. I would get the expected websocket msgs, but not always at the right step.

Probably some kind of issue with websocket msgs taking time to be sent?

Adding the websocket assertions here solved the problem, I'm guessing by making sure each step in this py script actually ends when it is expected to end.

tobias_ruck added inline comments.
test/functional/setup_scripts/chronik-client_lokad_id.py
57 ↗(On Diff #47578)

Probably what you want then is a version of the chronik_sub_lokad_id function that doesn’t subscribe, only check the logs

But seems fine for now imo, but could you change the comment to explain the actual reason?

This revision now requires changes to proceed.Sat, May 4, 13:07
bytesofman marked an inline comment as done.

Add comment explaining setup-script websocket subscription

Fabien added inline comments.
modules/chronik-client/proto/chronikNode.ts
5153 ↗(On Diff #47615)

?

Remove artefact in built chronikNode.ts, otherwise LGTM, good test coverage

modules/chronik-client/proto/chronikNode.ts
5153 ↗(On Diff #47615)

I can't reproduce this, running npm run build-proto && arc lint doesn't give this artefact on my machine

This revision is now accepted and ready to land.Mon, May 6, 13:13
modules/chronik-client/test/integration/lokad_id.ts
221 ↗(On Diff #47562)

the same LOKAD ID is in two different txs

test/functional/setup_scripts/chronik-client_lokad_id.py
110 ↗(On Diff #47562)

Good idea

112 ↗(On Diff #47562)

See D16111 for a rationale

bytesofman marked 4 inline comments as done.

remove build artifact, update test name per comment

Failed tests logs:

====== History endpoints and websocket for LOKAD ID: Websocket and endpoints correctly recognize txs with a lokadId in an input scriptSig and the same lokadId in OP_RETURN (non-EMPP).History endpoints and websocket for LOKAD ID Websocket and endpoints correctly recognize txs with a lokadId in an input scriptSig and the same lokadId in OP_RETURN (non-EMPP) ======
AssertionError: expected [] to deeply equal [ { type: 'Tx', …(2) } ]
    at /work/modules/chronik-client/test/integration/lokad_id.ts:254:47
    at Generator.next (<anonymous>)
    at fulfilled (test/integration/lokad_id.ts:31:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

      + expected - actual

      -[]
      +[
      +  {
      +    "msgType": "TX_ADDED_TO_MEMPOOL"
      +    "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
      +    "type": "Tx"
      +  }
      +]
====== History endpoints and websocket for LOKAD ID: Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN.History endpoints and websocket for LOKAD ID Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN ======
AssertionError: expected [ { type: 'Tx', …(2) }, …(1) ] to deeply equal [ { type: 'Tx', …(2) } ]
    at /work/modules/chronik-client/test/integration/lokad_id.ts:307:47
    at Generator.next (<anonymous>)
    at fulfilled (test/integration/lokad_id.ts:31:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

      + expected - actual

       [
         {
           "msgType": "TX_ADDED_TO_MEMPOOL"
      -    "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_ADDED_TO_MEMPOOL"
           "txid": "a989e07ae47f3925e911942e6dfbb0a5217fbd6617975b5c7a03a235c0c8a554"
           "type": "Tx"
         }
       ]
====== History endpoints and websocket for LOKAD ID: After these txs are mined, chronik-client returns this info appropriately.History endpoints and websocket for LOKAD ID After these txs are mined, chronik-client returns this info appropriately ======
AssertionError: expected [ { type: 'Tx', …(2) }, …(1) ] to deeply equal [ { type: 'Tx', …(2) } ]
    at /work/modules/chronik-client/test/integration/lokad_id.ts:372:59
    at Generator.next (<anonymous>)
    at fulfilled (test/integration/lokad_id.ts:31:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

      + expected - actual

           "msgType": "TX_CONFIRMED"
           "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
           "type": "Tx"
         }
      -  {
      -    "msgType": "TX_CONFIRMED"
      -    "txid": "a989e07ae47f3925e911942e6dfbb0a5217fbd6617975b5c7a03a235c0c8a554"
      -    "type": "Tx"
      -  }
       ]

Each failure log is accessible here:
History endpoints and websocket for LOKAD ID: Websocket and endpoints correctly recognize txs with a lokadId in an input scriptSig and the same lokadId in OP_RETURN (non-EMPP).History endpoints and websocket for LOKAD ID Websocket and endpoints correctly recognize txs with a lokadId in an input scriptSig and the same lokadId in OP_RETURN (non-EMPP)
History endpoints and websocket for LOKAD ID: Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN.History endpoints and websocket for LOKAD ID Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN
History endpoints and websocket for LOKAD ID: After these txs are mined, chronik-client returns this info appropriately.History endpoints and websocket for LOKAD ID After these txs are mined, chronik-client returns this info appropriately

Failed tests logs:

====== History endpoints and websocket for LOKAD ID: Websocket and endpoints correctly recognize txs with a lokadId in an input scriptSig and the same lokadId in OP_RETURN (non-EMPP).History endpoints and websocket for LOKAD ID Websocket and endpoints correctly recognize txs with a lokadId in an input scriptSig and the same lokadId in OP_RETURN (non-EMPP) ======
AssertionError: expected [] to deeply equal [ { type: 'Tx', …(2) } ]
    at /work/modules/chronik-client/test/integration/lokad_id.ts:254:47
    at Generator.next (<anonymous>)
    at fulfilled (test/integration/lokad_id.ts:31:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

      + expected - actual

      -[]
      +[
      +  {
      +    "msgType": "TX_ADDED_TO_MEMPOOL"
      +    "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
      +    "type": "Tx"
      +  }
      +]
====== History endpoints and websocket for LOKAD ID: Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN.History endpoints and websocket for LOKAD ID Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN ======
AssertionError: expected [ { type: 'Tx', …(2) }, …(1) ] to deeply equal [ { type: 'Tx', …(2) } ]
    at /work/modules/chronik-client/test/integration/lokad_id.ts:307:47
    at Generator.next (<anonymous>)
    at fulfilled (test/integration/lokad_id.ts:31:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

      + expected - actual

       [
         {
           "msgType": "TX_ADDED_TO_MEMPOOL"
      -    "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
      -    "type": "Tx"
      -  }
      -  {
      -    "msgType": "TX_ADDED_TO_MEMPOOL"
           "txid": "a989e07ae47f3925e911942e6dfbb0a5217fbd6617975b5c7a03a235c0c8a554"
           "type": "Tx"
         }
       ]
====== History endpoints and websocket for LOKAD ID: After these txs are mined, chronik-client returns this info appropriately.History endpoints and websocket for LOKAD ID After these txs are mined, chronik-client returns this info appropriately ======
AssertionError: expected [ { type: 'Tx', …(2) }, …(1) ] to deeply equal [ { type: 'Tx', …(2) } ]
    at /work/modules/chronik-client/test/integration/lokad_id.ts:372:59
    at Generator.next (<anonymous>)
    at fulfilled (test/integration/lokad_id.ts:31:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

      + expected - actual

           "msgType": "TX_CONFIRMED"
           "txid": "1ef4bebdb5a1298877aed9b7f741e3e0903820514316a35f3e7c05957308471a"
           "type": "Tx"
         }
      -  {
      -    "msgType": "TX_CONFIRMED"
      -    "txid": "a989e07ae47f3925e911942e6dfbb0a5217fbd6617975b5c7a03a235c0c8a554"
      -    "type": "Tx"
      -  }
       ]

Each failure log is accessible here:
History endpoints and websocket for LOKAD ID: Websocket and endpoints correctly recognize txs with a lokadId in an input scriptSig and the same lokadId in OP_RETURN (non-EMPP).History endpoints and websocket for LOKAD ID Websocket and endpoints correctly recognize txs with a lokadId in an input scriptSig and the same lokadId in OP_RETURN (non-EMPP)
History endpoints and websocket for LOKAD ID: Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN.History endpoints and websocket for LOKAD ID Websocket and endpoints correctly recognize a tx with a new lokadId in a spend input and a new lokad id in EMPP OP_RETURN
History endpoints and websocket for LOKAD ID: After these txs are mined, chronik-client returns this info appropriately.History endpoints and websocket for LOKAD ID After these txs are mined, chronik-client returns this info appropriately

this is related to websocket msgs occasionally being sent in the next step. I'm able to repeat it locally -- fails 3 out of 10 times.

...will be a perpetual flakiness issue if is not fixed now.

wait for expected ws msgs before testing them

This revision is now accepted and ready to land.Mon, May 6, 16:55
modules/chronik-client/test/integration/lokad_id.ts
254 ↗(On Diff #47646)

got tests passing 20 times in a row locally after adding this

you can also see the runs where the waiting happens, e.g. by adding a console.log here -- seems to be working as expected