qa: Warn when specified test is not found
ClosedPublic
Actions

Authored by Fabien on Jun 19 2019, 12:26.

Details

Reviewers

deadalnix
jasonbcox

Group Reviewers

Restricted Project

Commits

rSTAGING7999d34b16ec: qa: Warn when specified test is not found
rABC7999d34b16ec: qa: Warn when specified test is not found

Summary

This will warn on typos on tests provided manually on the command line.

Also, this comes with the new feature that the same test can be
specified multiple times on the command line.

Backport of core PR10374
https://github.com/bitcoin/bitcoin/pull/10374/files

Depends on D3367

Test Plan

Sanity check:

test_runner.py

Should be equivalent:

test_runner.py rpc_uptime
test_runner.py rpc_uptime.py

Should warn for test not found:

test_runner.py rpc_emitpu

Displays:

WARNING! Test 'rpc_emitpu.py' not found in full test list.

Should run twice:

test_runner.py rpc_uptime rpc_uptime
test_runner.py rpc_uptim* rpc_uptime

Should run once:

test_runner.py rpc_uptim* rpc_uptime --exclude=rpc_uptime
test_runner.py rpc_uptim* rpc_uptime --exclude=rpc_uptime.py

Should run nothing and display No valid test scripts specified. [...]:

test_runner.py rpc_uptim* rpc_uptime --exclude=rpc_uptime,rpc_uptime

Should run the test and warns that the excluded test is not found:

test_runner.py rpc_uptime --exclude=rpc_users

Displays:

WARNING! Test 'rpc_users.py' not found in current test list.

Diff Detail

Repository

rABC Bitcoin ABC

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

Fabien created this revision.Jun 19 2019, 12:26

Herald added a reviewer: Restricted Project. · View Herald TranscriptJun 19 2019, 12:26

Herald added a reviewer: deadalnix. · View Herald Transcript

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing started.

Harbormaster completed remote builds in B6401: Diff 9532.Jun 19 2019, 12:32

Build Bitcoin-ABC / Diffs / Bitcoin-ABC Diff Testing passed.

jasonbcox accepted this revision.Jun 19 2019, 20:26

jasonbcox added a subscriber: jasonbcox.

jasonbcox added inline comments.

test/functional/test_runner.py
260 ↗	(On Diff #9532)	I find the exclude behavior to be unintuitive. If you think it's worth taking ownership on this section of test_runner and this behavior is not depended on anywhere, this line could be replaced with something like: while exclude_test in test_list: test_list.remove(exclude_test) But considering it's a backport, I'll leave it up to you.

This revision is now accepted and ready to land.Jun 19 2019, 20:26

Fabien added inline comments.Jun 24 2019, 13:34

test/functional/test_runner.py
260 ↗	(On Diff #9532)	You're not the only one, see the comment (and the answer): https://github.com/bitcoin/bitcoin/pull/10374#discussion_r115612544 ; in short: this is not a bug, this is a feature. I will first backport the test framework completely then this can be updated. I personally don't see a case where one would run a test multiple times but excluding a single occurrence, that sounds absurd.