This prevents running the tests in parallel but make the CI report
useful info in case of failure and allow for tracking the flaky tests.
Details
- Reviewers
deadalnix jasonbcox - Group Reviewers
Restricted Project - Commits
- rSTAGING41d6f76ea618: Make the CI record and track the unit tests
rABC41d6f76ea618: Make the CI record and track the unit tests
See result on CI.
Running the build-diff configuration should create 2 different junit files, with different test suite names.
In the CI UI, the Tests tab should display each test case individually, associated with its test suite.
Diff Detail
- Repository
- rABC Bitcoin ABC
- Branch
- junit_unit_tests
- Lint
Lint Passed - Unit
No Test Coverage - Build Status
Buildable 9125 Build 16209: Default Diff Build & Tests Build 16208: arc lint + arc unit
Event Timeline
Increase log level from warning to message.
Make a test to fail intentionally to see how the CI reacts.
Snippet of first build failure:
[21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:03]W: [Step 1/1] Error: Specified -walletdir "wallets" is a relative path [21:09:05]W: [Step 1/1] [21:09:05]W: [Step 1/1] [1;31;49m*** 1 failure is detected in the test module "Bitcoin ABC unit tests" [21:09:05] : [Step 1/1] *** Output of /tmp/sanitizer_logs/*.log.* *** [21:09:05]W: [Step 1/1] [0;39;49m++ print_sanitizers_log [21:09:05]W: [Step 1/1] ++ for log in "${SAN_LOG_DIR}"/*.log.* [21:09:05]W: [Step 1/1] ++ echo '*** Output of /tmp/sanitizer_logs/*.log.* ***' [21:09:05]W: [Step 1/1] ++ cat '/tmp/sanitizer_logs/*.log.*' [21:09:05]W: [Step 1/1] cat: '/tmp/sanitizer_logs/*.log.*': No such file or directory [21:09:05]W: [Step 1/1] Process exited with code 1 [21:09:06]E: [Step 1/1] Process exited with code 1 (Step: Command Line)
Each failure log is accessible here:
Bitcoin_ABC_unit_tests: activation_tests.isphononenabled
Undo failure in tests.
Pass the args through environment to see how Teamcity handles skipped tests.
Snippet of first build failure:
[09:38:07]i: [wallet_multiwallet.py --usecli] [Test Output] 2020-01-24T09:37:07.890000Z TestFramework (INFO): Initializing test directory /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_multiwallet_435 2020-01-24T09:37:14.190000Z TestFramework (INFO): Check for per-wallet settxfee call 2020-01-24T09:37:14.235000Z TestFramework (INFO): Test dynamic wallet loading 2020-01-24T09:37:14.817000Z TestFramework (INFO): Load first wallet 2020-01-24T09:37:14.838000Z TestFramework (INFO): Load second wallet 2020-01-24T09:37:14.858000Z TestFramework (INFO): Load remaining wallets 2020-01-24T09:37:15.018000Z TestFramework (INFO): Test dynamic wallet creation. 2020-01-24T09:37:15.180000Z TestFramework (INFO): Test dynamic wallet unloading 2020-01-24T09:37:15.410000Z TestFramework (INFO): Test wallet backup 2020-01-24T09:37:17.388000Z TestFramework (INFO): Stopping nodes 2020-01-24T09:37:17.607000Z TestFramework (INFO): Cleaning up /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_multiwallet_435 on exit 2020-01-24T09:37:17.607000Z TestFramework (INFO): Tests successful [09:38:07]i: [bitcoin_abc_tests] wallet_resendwallettransactions.py [09:38:07]i: [wallet_resendwallettransactions.py] [Test Output] 2020-01-24T09:38:03.926000Z TestFramework (INFO): Initializing test directory /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_resendwallettransactions_522 2020-01-24T09:38:04.833000Z TestFramework (INFO): Stopping nodes 2020-01-24T09:38:05.137000Z TestFramework (INFO): Cleaning up /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_resendwallettransactions_522 on exit 2020-01-24T09:38:05.138000Z TestFramework (INFO): Tests successful [09:38:07]i: [bitcoin_abc_tests] wallet_txn_clone.py [09:38:07]i: [wallet_txn_clone.py] [Test Output] 2020-01-24T09:37:57.627000Z TestFramework (INFO): Initializing test directory /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_txn_clone_498 2020-01-24T09:38:00.025000Z TestFramework (INFO): Stopping nodes 2020-01-24T09:38:00.378000Z TestFramework (INFO): Cleaning up /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_txn_clone_498 on exit 2020-01-24T09:38:00.378000Z TestFramework (INFO): Tests successful [09:38:07]i: [bitcoin_abc_tests] wallet_txn_clone.py --mineblock [09:38:07]i: [wallet_txn_clone.py --mineblock] [Test Output] 2020-01-24T09:37:48.534000Z TestFramework (INFO): Initializing test directory /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_txn_clone_478 2020-01-24T09:37:50.742000Z TestFramework (INFO): Stopping nodes 2020-01-24T09:37:51.353000Z TestFramework (INFO): Cleaning up /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_txn_clone_478 on exit 2020-01-24T09:37:51.353000Z TestFramework (INFO): Tests successful [09:38:07]i: [bitcoin_abc_tests] wallet_txn_doublespend.py [09:38:07]i: [wallet_txn_doublespend.py] [Test Output] 2020-01-24T09:37:48.994000Z TestFramework (INFO): Initializing test directory /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_txn_doublespend_479 2020-01-24T09:37:51.351000Z TestFramework (INFO): Stopping nodes 2020-01-24T09:37:51.957000Z TestFramework (INFO): Cleaning up /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_txn_doublespend_479 on exit 2020-01-24T09:37:51.958000Z TestFramework (INFO): Tests successful [09:38:07]i: [bitcoin_abc_tests] wallet_txn_doublespend.py --mineblock [09:38:07]i: [wallet_txn_doublespend.py --mineblock] [Test Output] 2020-01-24T09:37:50.297000Z TestFramework (INFO): Initializing test directory /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_txn_doublespend_480 2020-01-24T09:37:51.904000Z TestFramework (INFO): Stopping nodes 2020-01-24T09:37:52.459000Z TestFramework (INFO): Cleaning up /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_txn_doublespend_480 on exit 2020-01-24T09:37:52.459000Z TestFramework (INFO): Tests successful [09:38:07]i: [bitcoin_abc_tests] wallet_zapwallettxes.py [09:38:07]i: [wallet_zapwallettxes.py] [Test Output] 2020-01-24T09:37:35.246000Z TestFramework (INFO): Initializing test directory /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_zapwallettxes_457 2020-01-24T09:37:35.613000Z TestFramework (INFO): Mining blocks... 2020-01-24T09:37:37.188000Z TestFramework (INFO): Stopping nodes 2020-01-24T09:37:37.441000Z TestFramework (INFO): Cleaning up /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_zapwallettxes_457 on exit 2020-01-24T09:37:37.441000Z TestFramework (INFO): Tests successful [09:38:07]E: Ant JUnit report watcher [09:38:07] : [Ant JUnit report watcher] 105 reports found for paths: [09:38:07] : [Ant JUnit report watcher] +:build/test_bitcoin.xml [09:38:07] : [Ant JUnit report watcher] +:build/**/junit_results*.xml [09:38:07] : [Ant JUnit report watcher] +:build/**/Master_Test_Suite*.xml [09:38:07]E: [Ant JUnit report watcher] Parsing errors [09:38:07]E: [Parsing errors] Failed to parse 2 reports [09:38:07]E: [Parsing errors] build/src/test/Master_Test_Suite.xml: Content is not allowed in trailing section. [09:38:07]E: [Parsing errors] build/src/test/Master_Test_Suite_2.xml: Content is not allowed in trailing section. [09:38:10]E: [Ant JUnit report watcher] Failed to parse xml reports
contrib/teamcity/build-configurations.sh | ||
---|---|---|
118 ↗ | (On Diff #15767) | Maintaining this list is a bit iffy, but I don't think the alternative of a new target that's ninja check-everything-except-bitcoin makes much sense either. I suppose if someone adds a new target, it's also there job to make sure it gets run on CI. |
src/test/test_bitcoin_main.cpp | ||
59 ↗ | (On Diff #15767) | This can be more informative: " with the next upgrade activated: " + nextUpgradeFlag |
contrib/teamcity/build-configurations.sh | ||
---|---|---|
118 ↗ | (On Diff #15767) | Ideally I would like to do something like https://reviews.bitcoinabc.org/D5042?id=15766, but I'm facing limitations both from boost and Teamcity. |
src/test/test_bitcoin_main.cpp | ||
59 ↗ | (On Diff #15767) | It can even be a custom argument in the spirit of D4943, but I needed something "quick and dirty" for challenging Teamcity first. |
Cleanup, add an option to set the test suite name in the spirit of D4943.
Now ready for review.
Snippet of first build failure:
[11:21:53] : [Step 1/1] [73/109] leveldb: testing env_test [11:21:53] : [Step 1/1] [74/109] bitcoin-seeder: testing seeder_tests [11:21:53] : [Step 1/1] [75/109] Running bitcoin-seeder test suite [11:21:53] : [Step 1/1] PASSED: bitcoin-seeder test suite [11:21:53] : [Step 1/1] [76/109] Test Bitcoin RPC authentication... [11:21:53] : [Step 1/1] ... [11:21:53] : [Step 1/1] ---------------------------------------------------------------------- [11:21:53] : [Step 1/1] Ran 3 tests in 0.003s [11:21:53] : [Step 1/1] [11:21:53] : [Step 1/1] OK [11:21:53] : [Step 1/1] [77/109] cd /home/teamcity/buildAgent/work/c4a5708f2bae7929/contrib/devtools/chainparams && /usr/bin/python3 ./test_make_chainparams.py [11:21:53] : [Step 1/1] ..... [11:21:53] : [Step 1/1] ---------------------------------------------------------------------- [11:21:53] : [Step 1/1] Ran 5 tests in 0.001s [11:21:53] : [Step 1/1] [11:21:53] : [Step 1/1] OK [11:21:53] : [Step 1/1] [78/109] Building CXX object src/leveldb/CMakeFiles/bloom_test.dir/util/bloom_test.cc.o [11:21:53] : [Step 1/1] [79/109] Building CXX object src/leveldb/CMakeFiles/autocompact_test.dir/db/autocompact_test.cc.o [11:21:53] : [Step 1/1] [80/109] Linking CXX executable src/leveldb/bloom_test [11:21:53] : [Step 1/1] [81/109] leveldb: testing bloom_test [11:21:53] : [Step 1/1] [82/109] leveldb: testing fault_injection_test [11:21:53] : [Step 1/1] [83/109] Linking CXX executable src/leveldb/autocompact_test [11:21:53] : [Step 1/1] [84/109] leveldb: testing corruption_test [11:21:53] : [Step 1/1] [85/109] leveldb: testing skiplist_test [11:21:54] : [Step 1/1] [86/109] leveldb: testing arena_test [11:21:54] : [Step 1/1] [87/109] leveldb: testing table_test [11:21:54] : [Step 1/1] [88/109] Automatic MOC for target test_bitcoin-qt [11:21:54] : [Step 1/1] [89/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/util.cpp.o [11:21:54] : [Step 1/1] [90/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/bitcoinaddressvalidatortests.cpp.o [11:21:54] : [Step 1/1] [91/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/compattests.cpp.o [11:21:54] : [Step 1/1] [92/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/guiutiltests.cpp.o [11:21:54] : [Step 1/1] [93/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/apptests.cpp.o [11:21:54] : [Step 1/1] [94/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/uritests.cpp.o [11:21:54] : [Step 1/1] [95/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/test_main.cpp.o [11:21:54] : [Step 1/1] [96/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/rpcnestedtests.cpp.o [11:21:54] : [Step 1/1] [97/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/addressbooktests.cpp.o [11:21:54] : [Step 1/1] [98/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/__/__/wallet/test/wallet_test_fixture.cpp.o [11:21:54] : [Step 1/1] [99/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/wallettests.cpp.o [11:21:54] : [Step 1/1] [100/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/test_bitcoin-qt_autogen/mocs_compilation.cpp.o [11:21:55] : [Step 1/1] [101/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/paymentservertests.cpp.o [11:21:55] : [Step 1/1] [102/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/__/__/test/test_bitcoin.cpp.o [11:21:56] : [Step 1/1] [103/109] leveldb: testing issue178_test [11:21:58] : [Step 1/1] [104/109] Linking CXX executable src/qt/test/test_bitcoin-qt [11:21:59] : [Step 1/1] [105/109] bitcoin-qt: testing test_bitcoin-qt [11:21:59] : [Step 1/1] [106/109] Running bitcoin-qt test suite [11:21:59] : [Step 1/1] PASSED: bitcoin-qt test suite [11:22:15] : [Step 1/1] [107/109] leveldb: testing autocompact_test [11:22:49] : [Step 1/1] [108/109] leveldb: testing db_test [11:22:49] : [Step 1/1] [109/109] Running leveldb test suite [11:22:49] : [Step 1/1] PASSED: leveldb test suite [11:22:49]W: [Step 1/1] + check-secp256k1 check-univalue [11:22:49]W: [Step 1/1] ./contrib/teamcity/build-configurations.sh: line 143: check-secp256k1: command not found [11:22:49] : [Step 1/1] *** Output of /tmp/sanitizer_logs/*.log.* *** [11:22:49]W: [Step 1/1] ++ print_sanitizers_log [11:22:49]W: [Step 1/1] ++ for log in "${SAN_LOG_DIR}"/*.log.* [11:22:49]W: [Step 1/1] ++ echo '*** Output of /tmp/sanitizer_logs/*.log.* ***' [11:22:49]W: [Step 1/1] ++ cat '/tmp/sanitizer_logs/*.log.*' [11:22:49]W: [Step 1/1] cat: '/tmp/sanitizer_logs/*.log.*': No such file or directory [11:22:49]W: [Step 1/1] Process exited with code 1 [11:22:50]E: [Step 1/1] Process exited with code 1 (Step: Command Line)
contrib/teamcity/build-configurations.sh | ||
---|---|---|
57 ↗ | (On Diff #16071) | Nit: It may be best for readability to put each arg (especially --) on it's own line: ./src/test/test_bitcoin \ --logger=HRF:JUNIT,${UNIT_TESTS_JUNIT_LOG_LEVEL},${TEST_BITCOIN_JUNIT} \ -- \ -testsuitename="${TEST_BITCOIN_SUITE_NAME}" \ "${@:2}" |
contrib/teamcity/build-configurations.sh | ||
---|---|---|
88 ↗ | (On Diff #16071) | Aren't the sanitizer tests reducing coverage for all tools and libs that are compiled with cmake? |
contrib/teamcity/build-configurations.sh | ||
---|---|---|
45 ↗ | (On Diff #16071) | Apparently this log level produces strange results in ASAN (I haven't tried UBSAN or TSAN yet): https://build.bitcoinabc.org/viewLog.html?buildId=28319&buildTypeId=BitcoinABC_BitcoinAbcStaging&tab=buildLog&state=60&_focus=1070#_state=60 This isn't an actual error and appears to be expected: https://stackoverflow.com/a/54951825/6312728 It's unfortunate that it throws garbage into the logs. Please take a look and see if there's an easy way to mitigate this. |
Maintain coverage for the sanitizer tests.
Do not track autotool, as it makes little sense. Revert to make check.
Address feedbacks.
contrib/teamcity/build-configurations.sh | ||
---|---|---|
45 ↗ | (On Diff #16071) | message is not the cause. This seems to be a bug in boost, when there are multiple loggers enabled (independently of the log level). |
This is a bit of a concern, because now a dev adding a new test suite needs to update this to make sure it runs. Failing to do so will just result in not running the test suite, silently. You need to figure this out.
@deadalnix I agree this diff has a few downsides. I'll see how to fix this, I have a few ideas.
@markblundeberg The message log level only applies to the Junit file (well, boost bug excepted). I found it to be the sweet spot since it produces more output than the stdout log by default, but still does not produce an output for each assertion (that makes the report file reach ~1.1GB !).
contrib/teamcity/build-configurations.sh | ||
---|---|---|
45 ↗ | (On Diff #16071) |