Page MenuHomePhabricator

[rpc] add unbroadcast info to mempool entries and getmempoolinfo
ClosedPublic

Authored by PiRK on Jan 22 2021, 02:06.

Details

Summary

a7ebe48b94c5a9195c8eabd193204c499cb4bfdb

  • expose info about number of txns in unbroadcast set and whether a mempool entry's tx has passed initial broadcast
  • makes rpcs more informative and allows for more explicit testing, eg tracking if tx is in unbroadcast set before and after originating node connects to peers (adds this in mempool_unbroadcast.py)
  • adds mempool method IsUnbroadcastTx to query for tx inclusion in mempool's unbroadcast set ------

651f1d816f054cb9c637f8a99c9360bba381ef58

  • mempool entry 'unbroadcast' field changes when tx passes initial broadcast (receive getdata), so anytime you compare mempool entries as a whole, you must wait for all broadcasts to complete ('unbroadcast' = False) otherwise the state may change in between calls
  • update P2PTxInvStore to send msg_getdata for invs and add functionality to wait for a list of txids to complete initial broadcast
  • make mempool_packages.py wait because it compares entries using getrawmempool and getmempoolentry ------

This is a backport of Core PR18895 [2/3]
https://github.com/bitcoin/bitcoin/pull/18895/commits/a7ebe48b94c5a9195c8eabd193204c499cb4bfdb
https://github.com/bitcoin/bitcoin/pull/18895/commits/651f1d816f054cb9c637f8a99c9360bba381ef58
Depends on D9023

Test Plan

ninja && test/functional/test_runner.py mempool_*

Event Timeline

PiRK requested review of this revision.Jan 22 2021, 02:06

Failed tests logs:

====== Bitcoin ABC functional tests: mempool_packages.py ======

------- Stdout: -------
2021-01-22T02:13:27.527000Z TestFramework (INFO): Initializing test directory /work/abc-ci-builds/build-debug/test/tmp/test_runner_₿₵_  _20210122_021152/mempool_packages_590
2021-01-22T02:13:29.417000Z TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 120, in main
    self.run_test()
  File "/work/test/functional/mempool_packages.py", line 87, in run_test
    assert_equal(entry, mempool[x])
  File "/work/test/functional/test_framework/util.py", line 60, in assert_equal
    for arg in (thing1, thing2) + args)))
AssertionError: not({'fees': {'base': Decimal('0.00010000'), 'modified': Decimal('0.00010000'), 'ancestor': Decimal('0.00400000'), 'descendant': Decimal('0.00110000')}, 'size': 191, 'fee': Decimal('0.00010000'), 'modifiedfee': Decimal('0.00010000'), 'time': 1611281608, 'height': 301, 'descendantcount': 11, 'descendantsize': 2101, 'descendantfees': 110000, 'ancestorcount': 40, 'ancestorsize': 7640, 'ancestorfees': 400000, 'depends': ['608577d9f39e3cdc713e7eb3551779a4d5d40b7fc0609a4f8b5a20a0ca28f45d'], 'spentby': ['d546721fba78299000e8f910889d074b0acb63a34461fa51730e95a48ebd9f3e'], 'unbroadcast': False} == {'fees': {'base': Decimal('0.00010000'), 'modified': Decimal('0.00010000'), 'ancestor': Decimal('0.00400000'), 'descendant': Decimal('0.00110000')}, 'size': 191, 'fee': Decimal('0.00010000'), 'modifiedfee': Decimal('0.00010000'), 'time': 1611281608, 'height': 301, 'descendantcount': 11, 'descendantsize': 2101, 'descendantfees': 110000, 'ancestorcount': 40, 'ancestorsize': 7640, 'ancestorfees': 400000, 'depends': ['608577d9f39e3cdc713e7eb3551779a4d5d40b7fc0609a4f8b5a20a0ca28f45d'], 'spentby': ['d546721fba78299000e8f910889d074b0acb63a34461fa51730e95a48ebd9f3e'], 'unbroadcast': True})
2021-01-22T02:13:29.468000Z TestFramework (INFO): Stopping nodes
2021-01-22T02:13:29.670000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/build-debug/test/tmp/test_runner_₿₵_  _20210122_021152/mempool_packages_590
2021-01-22T02:13:29.670000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/build-debug/test/tmp/test_runner_₿₵_  _20210122_021152/mempool_packages_590/test_framework.log
2021-01-22T02:13:29.670000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/build-debug/test/tmp/test_runner_₿₵_  _20210122_021152/mempool_packages_590' to consolidate all logs
====== Bitcoin ABC functional tests: mempool_unbroadcast.py ======

------- Stdout: -------
2021-01-22T02:15:22.430000Z TestFramework (INFO): Initializing test directory /work/abc-ci-builds/build-debug/test/tmp/test_runner_₿₵_  _20210122_021152/mempool_unbroadcast_724
2021-01-22T02:15:22.845000Z TestFramework (INFO): Test that mempool reattempts delivery of locally submitted transaction
2021-01-22T02:15:23.296000Z TestFramework (INFO): Generate transactions that only node 0 knows about
2021-01-22T02:15:23.428000Z TestFramework (ERROR): JSONRPC error
Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 120, in main
    self.run_test()
  File "/work/test/functional/mempool_unbroadcast.py", line 28, in run_test
    self.test_broadcast()
  File "/work/test/functional/mempool_unbroadcast.py", line 55, in test_broadcast
    rpc_tx_hsh = node.sendrawtransaction(txFS["hex"])
  File "/work/test/functional/test_framework/coverage.py", line 48, in __call__
    return_val = self.auth_service_proxy_instance.__call__(*args, **kwargs)
  File "/work/test/functional/test_framework/authproxy.py", line 159, in __call__
    raise JSONRPCException(response['error'], status)
test_framework.authproxy.JSONRPCException: txn-mempool-conflict (-26)
2021-01-22T02:15:23.479000Z TestFramework (INFO): Stopping nodes
2021-01-22T02:15:23.584000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/build-debug/test/tmp/test_runner_₿₵_  _20210122_021152/mempool_unbroadcast_724
2021-01-22T02:15:23.584000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/build-debug/test/tmp/test_runner_₿₵_  _20210122_021152/mempool_unbroadcast_724/test_framework.log
2021-01-22T02:15:23.585000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/build-debug/test/tmp/test_runner_₿₵_  _20210122_021152/mempool_unbroadcast_724' to consolidate all logs

Each failure log is accessible here:
Bitcoin ABC functional tests: mempool_packages.py
Bitcoin ABC functional tests: mempool_unbroadcast.py

PiRK planned changes to this revision.Jan 22 2021, 02:31

squash D9024 and D9026. The last commit in the PR fixes a test broken in the second commit.

PiRK edited the summary of this revision. (Show Details)

Failed tests logs:

====== Bitcoin ABC functional tests: mempool_packages.py ======

------- Stdout: -------
2021-01-22T07:50:04.556000Z TestFramework (INFO): Initializing test directory /work/abc-ci-builds/build-debug/test/tmp/test_runner_₿₵_  _20210122_074830/mempool_packages_131
2021-01-22T07:50:06.291000Z TestFramework (ERROR): Unexpected exception caught during testing
Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 120, in main
    self.run_test()
  File "/work/test/functional/mempool_packages.py", line 77, in run_test
    self.nodes[0].p2p.wait_for_broadcast(chain)
  File "/work/test/functional/test_framework/mininode.py", line 769, in wait_for_broadcast
    self.wait_until(lambda: set(self.get_invs()) == set(
AttributeError: 'P2PTxInvStore' object has no attribute 'wait_until'
2021-01-22T07:50:06.342000Z TestFramework (INFO): Stopping nodes
2021-01-22T07:50:06.794000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/build-debug/test/tmp/test_runner_₿₵_  _20210122_074830/mempool_packages_131
2021-01-22T07:50:06.794000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/build-debug/test/tmp/test_runner_₿₵_  _20210122_074830/mempool_packages_131/test_framework.log
2021-01-22T07:50:06.794000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/build-debug/test/tmp/test_runner_₿₵_  _20210122_074830/mempool_packages_131' to consolidate all logs

Each failure log is accessible here:
Bitcoin ABC functional tests: mempool_packages.py

Failed tests logs:

====== Bitcoin ABC functional tests: mempool_packages.py ======

------- Stdout: -------
2021-01-22T07:51:44.131000Z TestFramework (INFO): Initializing test directory /work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_  _20210122_075017/mempool_packages_315
2021-01-22T07:51:45.275000Z TestFramework (ERROR): Unexpected exception caught during testing
Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 120, in main
    self.run_test()
  File "/work/test/functional/mempool_packages.py", line 77, in run_test
    self.nodes[0].p2p.wait_for_broadcast(chain)
  File "/work/test/functional/test_framework/mininode.py", line 769, in wait_for_broadcast
    self.wait_until(lambda: set(self.get_invs()) == set(
AttributeError: 'P2PTxInvStore' object has no attribute 'wait_until'
2021-01-22T07:51:45.325000Z TestFramework (INFO): Stopping nodes
2021-01-22T07:51:45.679000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_  _20210122_075017/mempool_packages_315
2021-01-22T07:51:45.679000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_  _20210122_075017/mempool_packages_315/test_framework.log
2021-01-22T07:51:45.679000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_  _20210122_075017/mempool_packages_315' to consolidate all logs
====== Bitcoin ABC functional tests with the next upgrade activated: mempool_packages.py ======

------- Stdout: -------
2021-01-22T07:54:45.936000Z TestFramework (INFO): Initializing test directory /work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_  _20210122_075325/mempool_packages_75
2021-01-22T07:54:48.810000Z TestFramework (ERROR): Unexpected exception caught during testing
Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 120, in main
    self.run_test()
  File "/work/test/functional/mempool_packages.py", line 77, in run_test
    self.nodes[0].p2p.wait_for_broadcast(chain)
  File "/work/test/functional/test_framework/mininode.py", line 769, in wait_for_broadcast
    self.wait_until(lambda: set(self.get_invs()) == set(
AttributeError: 'P2PTxInvStore' object has no attribute 'wait_until'
2021-01-22T07:54:48.862000Z TestFramework (INFO): Stopping nodes
2021-01-22T07:54:49.566000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_  _20210122_075325/mempool_packages_75
2021-01-22T07:54:49.566000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_  _20210122_075325/mempool_packages_75/test_framework.log
2021-01-22T07:54:49.567000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_  _20210122_075325/mempool_packages_75' to consolidate all logs

Each failure log is accessible here:
Bitcoin ABC functional tests: mempool_packages.py
Bitcoin ABC functional tests with the next upgrade activated: mempool_packages.py

PiRK planned changes to this revision.Jan 22 2021, 08:01

It seems like PR18617 is a dependency for this one.

This revision is now accepted and ready to land.Jan 22 2021, 09:03

Failed tests logs:

====== Bitcoin ABC functional tests with the next upgrade activated: mempool_unbroadcast.py ======

------- Stdout: -------
2021-01-22T09:06:21.473000Z TestFramework (INFO): Initializing test directory /work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_  _20210122_090318/mempool_unbroadcast_396
2021-01-22T09:06:21.839000Z TestFramework (INFO): Test that mempool reattempts delivery of locally submitted transaction
2021-01-22T09:06:22.029000Z TestFramework (INFO): Generate transactions that only node 0 knows about
2021-01-22T09:06:22.114000Z TestFramework (ERROR): JSONRPC error
Traceback (most recent call last):
  File "/work/test/functional/test_framework/test_framework.py", line 122, in main
    self.run_test()
  File "/work/test/functional/mempool_unbroadcast.py", line 28, in run_test
    self.test_broadcast()
  File "/work/test/functional/mempool_unbroadcast.py", line 55, in test_broadcast
    rpc_tx_hsh = node.sendrawtransaction(txFS["hex"])
  File "/work/test/functional/test_framework/coverage.py", line 48, in __call__
    return_val = self.auth_service_proxy_instance.__call__(*args, **kwargs)
  File "/work/test/functional/test_framework/authproxy.py", line 159, in __call__
    raise JSONRPCException(response['error'], status)
test_framework.authproxy.JSONRPCException: txn-mempool-conflict (-26)
2021-01-22T09:06:22.165000Z TestFramework (INFO): Stopping nodes
2021-01-22T09:06:22.373000Z TestFramework (WARNING): Not cleaning up dir /work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_  _20210122_090318/mempool_unbroadcast_396
2021-01-22T09:06:22.373000Z TestFramework (ERROR): Test failed. Test logging available at /work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_  _20210122_090318/mempool_unbroadcast_396/test_framework.log
2021-01-22T09:06:22.373000Z TestFramework (ERROR): Hint: Call /work/test/functional/combine_logs.py '/work/abc-ci-builds/build-diff/test/tmp/test_runner_₿₵_  _20210122_090318/mempool_unbroadcast_396' to consolidate all logs

Each failure log is accessible here:
Bitcoin ABC functional tests with the next upgrade activated: mempool_unbroadcast.py

This revision was landed with ongoing or failed builds.Jan 22 2021, 09:42
This revision was automatically updated to reflect the committed changes.