Page MenuHomePhabricator

[lint] remove executable flag and shebang on non-script python files
ClosedPublic

Authored by PiRK on Feb 14 2024, 12:30.

Details

Summary

Python files that are imported as modules or always executed using the interpreter explicitely (python test.py) should not have a shebang and should not be executable.
Fix the current inconsistencies.

This is mostly a scripted diff, except for the PythonShebangLinter.php changes:

find test/functional/ -name '*py' -not -name 'test_runner.py' | xargs  sed -i '1{/^#!/d;}'
find test/functional/ -name '*py' -not -name 'test_runner.py' | xargs chmod 644

find electrum -name '*py' -not -path 'electrum/scripts/*' -not -path 'electrum/contrib/*' -not -name 'test_runner.py' | xargs  sed -i '1{/^#!/d;}'
find electrum -name '*py' -not -path 'electrum/scripts/*' -not -path 'electrum/contrib/*' -not -name 'test_runner.py' | xargs chmod 644

find chronik/chronik-plugin-impl -name '*py'  | xargs  sed -i '1{/^#!/d;}'

chmod 755 contrib/testgen/base58.py
sed -i '1{/^#!/d;}' contrib/buildbot/shieldio.py
sed -i '1{/^#!/d;}' contrib/buildbot/test/abcbot_fixture.py
sed -i '1{/^#!/d;}' electrum/scripts/util.py

And arc lint to remove a few blank lines at the beginning of files.

Note that the sed command removes the first line of a file if it starts with #! (shebang lines).

Depends on D15449

Test Plan

ninja check-functional-extend check-electrum

arc lint --everything

Add a few shebangs to non executable python files and check that it raises an error.

Diff Detail

Repository
rABC Bitcoin ABC
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Tail of the build log:

-- Performing Test have_C__Wshadow - Success
-- Performing Test have_C__Wundef
-- Performing Test have_C__Wundef - Success
-- Performing Test have_C__Wno_unused_function
-- Performing Test have_C__Wno_unused_function - Success
-- Performing Test have_C__Wno_overlength_strings
-- Performing Test have_C__Wno_overlength_strings - Success
-- Performing Test have_C__std_c89
-- Performing Test have_C__std_c89 - Success
-- Performing Test have_C__Wno_long_long
-- Performing Test have_C__Wno_long_long - Success
-- Performing Test have_C__Wno_duplicated_branches
-- Performing Test have_C__Wno_duplicated_branches - Success
-- Performing Test USE_ASM_X86_64
-- Performing Test USE_ASM_X86_64 - Success
-- Found Event component event: /usr/lib/x86_64-linux-gnu/libevent.so
-- Found Event: /usr/include (found suitable version "2.1.12-stable", minimum required is "2.0.22") found components: event 
-- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.74.0/BoostConfig.cmake (found suitable version "1.74.0", minimum required is "1.64")  
-- Found Event component pthreads: /usr/lib/x86_64-linux-gnu/libevent_pthreads.so
-- Found Event: /usr/include (found suitable version "2.1.12-stable", minimum required is "2.0.22") found components: pthreads 
-- Found MiniUPnPc component miniupnpc: /usr/lib/x86_64-linux-gnu/libminiupnpc.so
-- Found MiniUPnPc: /usr/include/miniupnpc (found suitable version "2.2.1", minimum required is "1.9")  
-- Found NATPMP component natpmp: /usr/lib/x86_64-linux-gnu/libnatpmp.so
-- Found NATPMP: /usr/include   
-- Performing Test fuzz_target_builds_without_main_fuzz
-- Performing Test fuzz_target_builds_without_main_fuzz - Failed
-- Found BerkeleyDB component CXX: /usr/lib/x86_64-linux-gnu/libdb_cxx-5.3.so
-- Found BerkeleyDB: /usr/include (found suitable version "5.3.28", minimum required is "5.3") found components: CXX 
-- Found SQLite3: /usr/include (found suitable version "3.34.1", minimum required is "3.7.17") 
-- Found ZeroMQ component zmq: /usr/lib/x86_64-linux-gnu/libzmq.so
-- Found ZeroMQ: /usr/include (found suitable version "4.3.4", minimum required is "4.1.5")  
-- Could NOT find Protobuf (missing: Protobuf_DIR)
-- Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.so;-pthread (found version "3.12.4") 
-- Found OpenSSL: /usr/lib/x86_64-linux-gnu/libcrypto.so (found version "1.1.1w")  
-- Looking for EVP_MD_CTX_new
-- Looking for EVP_MD_CTX_new - found
-- Found QREncode component qrencode: /usr/lib/x86_64-linux-gnu/libqrencode.so
-- Found QREncode: /usr/include   
-- Configuring native build in /work/abc-ci-builds/electrum-tests/native
-- Configuring done
-- Generating done
-- Build files have been written to: /work/abc-ci-builds/electrum-tests
[1/2] link test_runner.py
[2/2] Run Electrum ABC unit tests...
FAILED: electrum/CMakeFiles/check-electrum 
cd /work/abc-ci-builds/electrum-tests/electrum && /usr/bin/python3.9 ./test_runner.py
Traceback (most recent call last):
  File "/work/abc-ci-builds/electrum-tests/electrum/./test_runner.py", line 51, in <module>
    test_setup()
  File "/work/abc-ci-builds/electrum-tests/electrum/./test_runner.py", line 22, in test_setup
    ret = subprocess.run(
  File "/usr/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.9/subprocess.py", line 1823, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
PermissionError: [Errno 13] Permission denied: '/work/electrum/setup.py'
ninja: build stopped: cannot make progress due to previous errors.
Build electrum-tests failed with exit code 1

The fact that the changes are not displayed makes it hard to review. Except for the linter (php), all changes are only file permissions and removal of blank lines and shebangs.

Tail of the build log:

wallet_listtransactions.py --descriptors   | ✓ Passed  | 3 s
wallet_multiwallet.py                      | ✓ Passed  | 41 s
wallet_multiwallet.py --descriptors        | ✓ Passed  | 41 s
wallet_multiwallet.py --usecli             | ✓ Passed  | 9 s
wallet_reorgsrestore.py                    | ✓ Passed  | 3 s
wallet_resendwallettransactions.py         | ✓ Passed  | 2 s
wallet_send.py                             | ✓ Passed  | 9 s
wallet_startup.py                          | ✓ Passed  | 2 s
wallet_timelock.py                         | ✓ Passed  | 1 s
wallet_txn_clone.py                        | ✓ Passed  | 1 s
wallet_txn_clone.py --mineblock            | ✓ Passed  | 2 s
wallet_txn_doublespend.py                  | ✓ Passed  | 1 s
wallet_txn_doublespend.py --mineblock      | ✓ Passed  | 2 s
wallet_watchonly.py                        | ✓ Passed  | 1 s
wallet_watchonly.py --usecli               | ✓ Passed  | 1 s
chronik_avalanche.py                       | ○ Skipped | 0 s
chronik_block.py                           | ○ Skipped | 0 s
chronik_block_info.py                      | ○ Skipped | 0 s
chronik_block_txs.py                       | ○ Skipped | 0 s
chronik_blockchain_info.py                 | ○ Skipped | 0 s
chronik_blocks.py                          | ○ Skipped | 0 s
chronik_chronik_info.py                    | ○ Skipped | 0 s
chronik_disable_token_index.py             | ○ Skipped | 0 s
chronik_disallow_prune.py                  | ○ Skipped | 0 s
chronik_mempool_conflicts.py               | ○ Skipped | 0 s
chronik_pause.py                           | ○ Skipped | 0 s
chronik_plugins_setup.py                   | ○ Skipped | 0 s
chronik_raw_tx.py                          | ○ Skipped | 0 s
chronik_resync.py                          | ○ Skipped | 0 s
chronik_script_confirmed_txs.py            | ○ Skipped | 0 s
chronik_script_history.py                  | ○ Skipped | 0 s
chronik_script_unconfirmed_txs.py          | ○ Skipped | 0 s
chronik_script_utxos.py                    | ○ Skipped | 0 s
chronik_serve.py                           | ○ Skipped | 0 s
chronik_spent_by.py                        | ○ Skipped | 0 s
chronik_token_alp.py                       | ○ Skipped | 0 s
chronik_token_broadcast_txs.py             | ○ Skipped | 0 s
chronik_token_burn.py                      | ○ Skipped | 0 s
chronik_token_id_group.py                  | ○ Skipped | 0 s
chronik_token_parse_failure.py             | ○ Skipped | 0 s
chronik_token_script_group.py              | ○ Skipped | 0 s
chronik_token_slp_fungible.py              | ○ Skipped | 0 s
chronik_token_slp_mint_vault.py            | ○ Skipped | 0 s
chronik_token_slp_nft1.py                  | ○ Skipped | 0 s
chronik_tx.py                              | ○ Skipped | 0 s
chronik_tx_removal_order.py                | ○ Skipped | 0 s
chronik_ws.py                              | ○ Skipped | 0 s
chronik_ws_ping.py                         | ○ Skipped | 0 s
chronik_ws_script.py                       | ○ Skipped | 0 s
feature_bind_port_discover.py              | ○ Skipped | 0 s
feature_bind_port_externalip.py            | ○ Skipped | 0 s
interface_usdt_net.py                      | ○ Skipped | 0 s
interface_usdt_utxocache.py                | ○ Skipped | 0 s
interface_usdt_validation.py               | ○ Skipped | 0 s

ALL                                        | ✓ Passed  | 1341 s (accumulated) 
Runtime: 268 s

ninja: build stopped: cannot make progress due to previous errors.
Build build-diff failed with exit code 1
PiRK edited the summary of this revision. (Show Details)
PiRK edited the test plan for this revision. (Show Details)

rebase onto D15449, keep electrum/test_runner.py executable

PiRK edited the test plan for this revision. (Show Details)

rebase and reverse the test order to skip some potentially expensive pattern matching on executable python scripts

PiRK published this revision for review.Feb 14 2024, 14:43
The build failed due to an unexpected infrastructure outage. The administrators have been notified to investigate. Sorry for the inconvenience.
The build failed due to an unexpected infrastructure outage. The administrators have been notified to investigate. Sorry for the inconvenience.
The build failed due to an unexpected infrastructure outage. The administrators have been notified to investigate. Sorry for the inconvenience.
The build failed due to an unexpected infrastructure outage. The administrators have been notified to investigate. Sorry for the inconvenience.
The build failed due to an unexpected infrastructure outage. The administrators have been notified to investigate. Sorry for the inconvenience.
The build failed due to an unexpected infrastructure outage. The administrators have been notified to investigate. Sorry for the inconvenience.
The build failed due to an unexpected infrastructure outage. The administrators have been notified to investigate. Sorry for the inconvenience.
The build failed due to an unexpected infrastructure outage. The administrators have been notified to investigate. Sorry for the inconvenience.
The build failed due to an unexpected infrastructure outage. The administrators have been notified to investigate. Sorry for the inconvenience.
The build failed due to an unexpected infrastructure outage. The administrators have been notified to investigate. Sorry for the inconvenience.
PiRK planned changes to this revision.Feb 14 2024, 16:14

checking the infra outage, it could very well be related

Fabien added a subscriber: Fabien.

linter looks OK, I trust the remaining files are correct

This revision is now accepted and ready to land.Feb 15 2024, 13:49