It seems like unittest.defaultTestLoader.discover imports some of the python files (scripts outside packages, __init__.py files...), so all the code that is not in a function or inside a if __name__ == "__main__" context will be executed. This explains why failure to import PyQt5 was reported as a failed test before D14135, and why the tests seemed to be run twice.

Depends on D14260

python test_runner.py
Check that the test results are printed only once