[electrum] be selective about ignoring errors
ClosedPublic
Actions

Authored by PiRK on Jul 12 2023, 16:06.

Details

Reviewers

Fabien

Group Reviewers

Restricted Project

Commits

rABCd0466eaf73c4: [electrum] be selective about ignoring errors

Summary

Only ignore a whitelist of acceptable errors instead of ignoring all errors during test discovery. Ignoring all errors could lead to tests just being skipped and the test suite succeeding, which could go unnoticed for a long time.

This will help us detect some unexpected errors, and provide us with a useful error stack trace to debug it.

For instance, intentionnaly raising an error in the main context of a module that is imported by an __init__.py produces this output:

$ python electrum/test_runner.py
Testing `setup.py --version`: OK

........................................................................................................................................................................................................................s.........s.........s..................................E......
======================================================================
ERROR: electrumabc_gui.qt (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: electrumabc_gui.qt
Traceback (most recent call last):
  File "/usr/lib/python3.10/unittest/loader.py", line 470, in _find_test_path
    package = self._get_module_from_name(name)
  File "/usr/lib/python3.10/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/pierre/dev/bitcoin-abc/electrum/electrumabc_gui/qt/__init__.py", line 86, in <module>
    from .exception_window import ExceptionHook
  File "/home/pierre/dev/bitcoin-abc/electrum/electrumabc_gui/qt/exception_window.py", line 46, in <module>
    from .main_window import ElectrumWindow
  File "/home/pierre/dev/bitcoin-abc/electrum/electrumabc_gui/qt/main_window.py", line 88, in <module>
    from . import address_dialog, external_plugins_window, qrwindow
  File "/home/pierre/dev/bitcoin-abc/electrum/electrumabc_gui/qt/address_dialog.py", line 33, in <module>
    from .history_list import HistoryList
  File "/home/pierre/dev/bitcoin-abc/electrum/electrumabc_gui/qt/history_list.py", line 46, in <module>
    raise ImportError("spam spam spam spam spam")
ImportError: spam spam spam spam spam

----------------------------------------------------------------------
Ran 278 tests in 9.094s

FAILED (errors=1, skipped=3)

Use addTest instead of addTests, because the former can take a TestCase (which _FailedTest is a subclass of) or a TestSuite, whereas the latter expected an iterable object (i.e a TestSuite or a list of TestCase)

Test Plan

python test_runner.py

Diff Detail

Repository

rABC Bitcoin ABC

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

PiRK created this revision.Jul 12 2023, 16:06

Herald added a reviewer: Restricted Project. · View Herald TranscriptJul 12 2023, 16:06

PiRK requested review of this revision.Jul 12 2023, 16:06

PiRK edited the summary of this revision. (Show Details)Jul 12 2023, 16:07

PiRK added a parent revision: D14261: [electrum] don't run the unit tests twice.

PiRK edited the summary of this revision. (Show Details)

Harbormaster completed remote builds in B24473: Diff 41419.Jul 12 2023, 16:07

Build Bitcoin ABC Diffs / Diff Testing (electrum-tests) failed.

Tail of the build log:

    module = self._get_module_from_name(name)
  File "/usr/lib/python3.9/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/work/electrum/electrumabc/tests/regtest/test_rpc_misc.py", line 4, in <module>
    from jsonrpcclient import request
ModuleNotFoundError: No module named 'jsonrpcclient'


======================================================================
ERROR: electrumabc.tests.regtest.test_rpc_payment_request (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: electrumabc.tests.regtest.test_rpc_payment_request
Traceback (most recent call last):
  File "/usr/lib/python3.9/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.9/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/work/electrum/electrumabc/tests/regtest/test_rpc_payment_request.py", line 4, in <module>
    from jsonrpcclient import request
ModuleNotFoundError: No module named 'jsonrpcclient'


======================================================================
ERROR: electrumabc_gui.qt (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: electrumabc_gui.qt
Traceback (most recent call last):
  File "/work/electrum/electrumabc_gui/qt/__init__.py", line 39, in <module>
    from PyQt5 import QtCore, QtWidgets
ModuleNotFoundError: No module named 'PyQt5'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/unittest/loader.py", line 470, in _find_test_path
    package = self._get_module_from_name(name)
  File "/usr/lib/python3.9/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/work/electrum/electrumabc_gui/qt/__init__.py", line 64, in <module>
    sys.exit(msg)
SystemExit: 

Error: Could not import PyQt5.
You may try:

    python3 -m pip install --user -I pyqt5

Or, if on Linux Ubuntu, Debian, etc:

    sudo apt-get install python3-pyqt5




----------------------------------------------------------------------
Ran 271 tests in 19.088s

FAILED (errors=3, skipped=7)
ninja: build stopped: cannot make progress due to previous errors.
Build electrum-tests failed with exit code 1

needs more know failure or alternatively installing more test dependencies on CI

skip also electrumabc.tests.regtest.*

If there are more of these in the future and the error is not resolved, we can make KNOWN_FAILURES regular expressions and match the module names instead exact string comparisons in the future.

Harbormaster completed remote builds in B24474: Diff 41420.Jul 12 2023, 16:21

Build Bitcoin ABC Diffs / Diff Testing (electrum-tests) failed.

Tail of the build log:

[ecc] info: libsecp256k1 library not available, falling back to python-ecdsa. This means signing operations will be slower. Try running:

  $  contrib/make_secp

(You need to be running from the git sources for contrib/make_secp to be available)
......................................................................................................127.0.0.1 - - [12/Jul/2023 16:20:59] "GET /invoice HTTP/1.1" 503 -
.127.0.0.1 - - [12/Jul/2023 16:21:00] "GET / HTTP/1.1" 200 -
.127.0.0.1 - - [12/Jul/2023 16:21:00] "GET /invoice HTTP/1.1" 200 -
127.0.0.1 - - [12/Jul/2023 16:21:00] "POST /pay HTTP/1.1" 200 -
.127.0.0.1 - - [12/Jul/2023 16:21:00] "GET / HTTP/1.1" 200 -
.s............../work/electrum/electrumabc_plugins/fusion/plugin.py:426: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 56308)>
  if is_tor_port(host, port):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/work/electrum/electrumabc_plugins/fusion/plugin.py:426: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 43480)>
  if is_tor_port(host, port):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
....Traceback (most recent call last):
  File "/work/electrum/electrumabc_plugins/trezor/trezor.py", line 29, in <module>
    import trezorlib
ModuleNotFoundError: No module named 'trezorlib'
..............................................................................................s.........sss..................................E..s.s.
======================================================================
ERROR: electrumabc_gui.qt (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: electrumabc_gui.qt
Traceback (most recent call last):
  File "/work/electrum/electrumabc_gui/qt/__init__.py", line 39, in <module>
    from PyQt5 import QtCore, QtWidgets
ModuleNotFoundError: No module named 'PyQt5'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/unittest/loader.py", line 470, in _find_test_path
    package = self._get_module_from_name(name)
  File "/usr/lib/python3.9/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/work/electrum/electrumabc_gui/qt/__init__.py", line 64, in <module>
    sys.exit(msg)
SystemExit: 

Error: Could not import PyQt5.
You may try:

    python3 -m pip install --user -I pyqt5

Or, if on Linux Ubuntu, Debian, etc:

    sudo apt-get install python3-pyqt5




----------------------------------------------------------------------
Ran 269 tests in 11.680s

FAILED (errors=1, skipped=7)
------------ electrumabc_gui.qt
ninja: build stopped: cannot make progress due to previous errors.
Build electrum-tests failed with exit code 1

Still more exceptions are needed. Will try to use pattern matching to avoid repeating too many similar module names.

use regular expressions for known errors, to avoid having to update the list frequently as regtests are added or PyQt is imported in more place in the GUI package.

Harbormaster completed remote builds in B24484: Diff 41434.Jul 13 2023, 12:14

Fabien accepted this revision.Jul 13 2023, 16:25

This revision is now accepted and ready to land.Jul 13 2023, 16:25

Closed by commit rABCd0466eaf73c4: [electrum] be selective about ignoring errors (authored by PiRK). · Explain WhyJul 17 2023, 08:11

This revision was automatically updated to reflect the committed changes.

PiRK added a commit: rABCd0466eaf73c4: [electrum] be selective about ignoring errors.