Page MenuHomePhabricator

[electrum] provide a test framework for Electrum ABC functional tests
ClosedPublic

Authored by PiRK on Jun 25 2025, 15:23.

Details

Summary

Remove the dependency on Fulcrum and pytest-docker, and get all the benefits from the monorepo by testing with an up-to-date Chronik Electrum server.

The tests are now run on CI when electrum or chronik-http code is modified.

Potential future improvements that are out of scope for this diff:

  • reuse more features from the node's test framework (assert_equal, wait_until...)
  • provide more feedback on test failure (path the node's debug log and stderr...)
  • provide less feedback on test success: don't log Electrum ABC's verbose output unconditionally, write it to a file

Certificate and key generated with:
Self-signed certificate generated with:

openssl req -newkey rsa:2048 -sha256 -nodes -x509 -days 36500 -subj '/O=BitcoinABC' -keyout server.key -out server.crt

Depends on D18295

Test Plan

Test all the ways to run the test suite or an individual test module

./contrib/teamcity/build-configurations.py electrum-functional-tests
mkdir build && cd build
cmake .. -GNinja -DBUILD_CHRONIK=1
ninja check-electrum-functional
cd ../electrum
BITCOIND='/home/pierre/dev/bitcoin-abc/build/src/bitcoind' python -m electrumabc.tests.regtest.test_rpc_misc
BITCOIND='/home/pierre/dev/bitcoin-abc/build/src/bitcoind' python test_runner.py functional

Diff Detail

Repository
rABC Bitcoin ABC
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Tail of the build log:

  _descriptor.EnumValueDescriptor(
/usr/local/lib/python3.11/dist-packages/keepkeylib/types_pb2.py:322: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/usr/local/lib/python3.11/dist-packages/keepkeylib/types_pb2.py:326: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/usr/local/lib/python3.11/dist-packages/keepkeylib/types_pb2.py:330: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/usr/local/lib/python3.11/dist-packages/keepkeylib/types_pb2.py:334: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/usr/local/lib/python3.11/dist-packages/keepkeylib/types_pb2.py:338: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/usr/local/lib/python3.11/dist-packages/keepkeylib/types_pb2.py:342: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/usr/local/lib/python3.11/dist-packages/keepkeylib/types_pb2.py:346: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/usr/local/lib/python3.11/dist-packages/keepkeylib/types_pb2.py:350: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/usr/local/lib/python3.11/dist-packages/keepkeylib/types_pb2.py:354: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/usr/local/lib/python3.11/dist-packages/keepkeylib/types_pb2.py:358: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/usr/local/lib/python3.11/dist-packages/keepkeylib/types_pb2.py:362: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/usr/local/lib/python3.11/dist-packages/keepkeylib/types_pb2.py:366: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/usr/local/lib/python3.11/dist-packages/keepkeylib/types_pb2.py:370: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/usr/local/lib/python3.11/dist-packages/keepkeylib/types_pb2.py:374: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/usr/local/lib/python3.11/dist-packages/keepkeylib/types_pb2.py:378: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/usr/local/lib/python3.11/dist-packages/keepkeylib/types_pb2.py:236: DeprecationWarning: Call to deprecated create function EnumDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _BUTTONREQUESTTYPE = _descriptor.EnumDescriptor(
/usr/local/lib/python3.11/dist-packages/keepkeylib/types_pb2.py:397: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
/usr/local/lib/python3.11/dist-packages/keepkeylib/types_pb2.py:401: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
  _descriptor.EnumValueDescriptor(
..
======================================================================
ERROR: setUpClass (electrumabc.tests.regtest.test_rpc_misc.TestRPCMisc)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/work/electrum/electrumabc/tests/regtest/framework.py", line 49, in setUpClass
    cls.node = TestNode(
               ^^^^^^^^^
  File "/work/electrum/electrumabc/tests/regtest/../../../../test/functional/test_framework/test_node.py", line 117, in __init__
    if not os.path.isfile(self.binary):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen genericpath>", line 30, in isfile
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

----------------------------------------------------------------------
Ran 342 tests in 28.322s

FAILED (errors=1, skipped=4)
Testing `setup.py --version`: OK

/tmp/electrum_func_test_3gwybymz
ninja: build stopped: cannot make progress due to previous errors.
Build electrum-tests failed with exit code 1
  • Use setUp rather than setUpClass, so every test case has its own state (fresh bitcoind, fresh chain, fresh ElectrumABC daemon)
  • Mine 100 blocks so funds are immediately available to the node
  • wrap the generatetoaddress call (see D12521)

there is still some work left to do, but now the tests pass

Tail of the build log:

Traceback (most recent call last):
  File "/work/electrum/electrumabc/tests/regtest/framework.py", line 47, in setUp
    self.node = TestNode(
                ^^^^^^^^^
  File "/work/electrum/electrumabc/tests/regtest/../../../../test/functional/test_framework/test_node.py", line 117, in __init__
    if not os.path.isfile(self.binary):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen genericpath>", line 30, in isfile
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

======================================================================
ERROR: test_getunusedaddress (electrumabc.tests.regtest.test_rpc_misc.TestRPCMisc.test_getunusedaddress)
Verify the `getunusedaddress` RPC
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/work/electrum/electrumabc/tests/regtest/framework.py", line 47, in setUp
    self.node = TestNode(
                ^^^^^^^^^
  File "/work/electrum/electrumabc/tests/regtest/../../../../test/functional/test_framework/test_node.py", line 117, in __init__
    if not os.path.isfile(self.binary):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen genericpath>", line 30, in isfile
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

======================================================================
ERROR: test_payto_broadcast_getaddresshistory (electrumabc.tests.regtest.test_rpc_misc.TestRPCMisc.test_payto_broadcast_getaddresshistory)
Use the payto and broadcast commands and check the resulting  wallet UTXOs
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/work/electrum/electrumabc/tests/regtest/framework.py", line 47, in setUp
    self.node = TestNode(
                ^^^^^^^^^
  File "/work/electrum/electrumabc/tests/regtest/../../../../test/functional/test_framework/test_node.py", line 117, in __init__
    if not os.path.isfile(self.binary):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen genericpath>", line 30, in isfile
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

======================================================================
ERROR: test_signtransaction (electrumabc.tests.regtest.test_rpc_misc.TestRPCMisc.test_signtransaction)
Create an unsigned transaction, sign and broadcast it
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/work/electrum/electrumabc/tests/regtest/framework.py", line 47, in setUp
    self.node = TestNode(
                ^^^^^^^^^
  File "/work/electrum/electrumabc/tests/regtest/../../../../test/functional/test_framework/test_node.py", line 117, in __init__
    if not os.path.isfile(self.binary):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen genericpath>", line 30, in isfile
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

----------------------------------------------------------------------
Ran 349 tests in 30.153s

FAILED (errors=7, skipped=4)
Testing `setup.py --version`: OK

ninja: build stopped: cannot make progress due to previous errors.
Build electrum-tests failed with exit code 1

make test_runner.py and cmake able to run the tests

electrum/test_runner.py
109 โ†—(On Diff #54595)

probably not needed, remove (we properly pass top_level_dir where needed above)

PiRK edited the test plan for this revision. (Show Details)

add buildbot command

use the framework for all tests, drop the previous frameworks code, update documentation, run these integration tests on CI
It is almost ready now, but i'm still investigating a test failure in the reorg test

electrum/CONTRIBUTING.md
125 โ†—(On Diff #54604)

typo

The build failed due to an unexpected infrastructure outage. The administrators have been notified to investigate. Sorry for the inconvenience.
The build failed due to an unexpected infrastructure outage. The administrators have been notified to investigate. Sorry for the inconvenience.
PiRK edited the summary of this revision. (Show Details)

rebase

@bot check-electrum-functional

The build failed due to an unexpected infrastructure outage. The administrators have been notified to investigate. Sorry for the inconvenience.

@bot electrum-functional-tests

Tail of the build log:

Build electrum-functional-tests completed successfully

fix failing test by replacing a bare assert with a wait_until
I think the transaction notification can come before the header notif in some cases, so the assumption that when the transaction has the expected height the server also has the correct height is wrong. It eventually gets to the expected height.

PiRK published this revision for review.Jun 27 2025, 19:10
Fabien added a subscriber: Fabien.

Looks good !

electrum/test_runner.py
97 โ†—(On Diff #54618)

you should print this to stderr

This revision is now accepted and ready to land.Jun 27 2025, 20:40

print error to stderr
In the future it would be an improvement to use a proper logger.

@bot electrum-functional-tests

Tail of the build log:

Build electrum-functional-tests completed successfully

@bot electrum-functional-tests