Page MenuHomePhabricator

Add 'sequence' zmq publisher to track all block (dis)connects, mempool deltas
ClosedPublic

Authored by PiRK on Oct 12 2021, 08:26.

Details

Summary

Using the zmq notifications to avoid excessive mempool polling can be difficult
given the current notifications available. It announces all transactions
being added to mempool or included in blocks, but announces no evictions
and gives no indication if the transaction is in the mempool or a block.

Block notifications for zmq are also substandard, in that it only announces
block tips, while all block transactions are still announced.

This commit adds a unified stream which can be used to closely track mempool:

  1. getrawmempool to fill out mempool knowledge
  2. if txhash is announced, add or remove from set based on add/remove flag
  3. if blockhash is announced, get block txn list, remove from those transactions local view of mempool
  4. if we drop a sequence number, go to (1)

The mempool sequence number starts at the value 1, and
increments each time a transaction enters the mempool,
or is evicted from the mempool for any reason, including
block inclusion. The mempool sequence number is published
via ZMQ for any transaction-related notification.

These features allow for ZMQ/RPC consumer to track mempool
state in a more exacting way, without unnecesarily polling
getrawmempool. See interface_zmq.py::test_mempool_sync for
example usage.

This is a backport of core#19572 [2/4]
https://github.com/bitcoin/bitcoin/pull/19572/commits/e76fc2b84d065c9d06010d0a10b316f1f9d36fb9

See for D2304 for the reason of the change in the check-doc.py linter.

Depends on D10303

Test Plan

ninja all check-all

Event Timeline

PiRK requested review of this revision.Oct 12 2021, 08:26
PiRK planned changes to this revision.Oct 12 2021, 08:31

I spotted a mistake in client.cpp

fix arg name in client.cpp, move a few comments to their own line

Fabien requested changes to this revision.Oct 12 2021, 09:38
Fabien added a subscriber: Fabien.
Fabien added inline comments.
src/zmq/zmqpublishnotifier.cpp
231 ↗(On Diff #30446)

That's a BlockHash

245 ↗(On Diff #30446)

dito

259 ↗(On Diff #30446)

And that would better be a TxId

This revision now requires changes to proceed.Oct 12 2021, 09:38

use BlockHash and TxId instead of uint256

This revision is now accepted and ready to land.Oct 13 2021, 13:10