Page MenuHomePhabricator
Differential D5042

Make the CI record and track the unit tests
ClosedPublic
Actions

Authored by Fabien on Jan 22 2020, 15:23.

Details

Reviewers
deadalnix
jasonbcox
Group Reviewers
Restricted Project
Commits
rSTAGING41d6f76ea618: Make the CI record and track the unit tests
rABC41d6f76ea618: Make the CI record and track the unit tests
Summary

This prevents running the tests in parallel but make the CI report
useful info in case of failure and allow for tracking the flaky tests.

Test Plan

See result on CI.
Running the build-diff configuration should create 2 different junit files, with different test suite names.
In the CI UI, the Tests tab should display each test case individually, associated with its test suite.

Diff Detail

Repository
rABC Bitcoin ABC
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Fabien created this revision.Jan 22 2020, 15:23
Herald added a reviewer: Restricted Project. · View Herald TranscriptJan 22 2020, 15:23
Herald added a reviewer: deadalnix. · View Herald Transcript
teamcity added a comment.Jan 22 2020, 15:23

Build Bitcoin-ABC / Diffs / Diff Testing started.

teamcity added a comment.Jan 22 2020, 15:30

Build Bitcoin-ABC / Diffs / Diff Testing passed.

Harbormaster completed remote builds in B9105: Diff 15738.Jan 22 2020, 15:30
Fabien updated this revision to Diff 15745.Jan 22 2020, 20:53

Increase log level from warning to message.
Make a test to fail intentionally to see how the CI reacts.

teamcity added a comment.Jan 22 2020, 20:57

Build Bitcoin-ABC / Diffs / Diff Testing started.

teamcity added a comment.Jan 22 2020, 21:09
Build Bitcoin-ABC / Diffs / Diff Testing failed.

Snippet of first build failure:

[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:03]W:	 [Step 1/1] Error: Specified -walletdir "wallets" is a relative path
[21:09:05]W:	 [Step 1/1] 
[21:09:05]W:	 [Step 1/1]  [1;31;49m*** 1 failure is detected in the test module "Bitcoin ABC unit tests"
[21:09:05] :	 [Step 1/1] *** Output of /tmp/sanitizer_logs/*.log.* ***
[21:09:05]W:	 [Step 1/1]  [0;39;49m++ print_sanitizers_log
[21:09:05]W:	 [Step 1/1] ++ for log in "${SAN_LOG_DIR}"/*.log.*
[21:09:05]W:	 [Step 1/1] ++ echo '*** Output of /tmp/sanitizer_logs/*.log.* ***'
[21:09:05]W:	 [Step 1/1] ++ cat '/tmp/sanitizer_logs/*.log.*'
[21:09:05]W:	 [Step 1/1] cat: '/tmp/sanitizer_logs/*.log.*': No such file or directory
[21:09:05]W:	 [Step 1/1] Process exited with code 1
[21:09:06]E:	 [Step 1/1] Process exited with code 1 (Step: Command Line)

Each failure log is accessible here:
Bitcoin_ABC_unit_tests: activation_tests.isphononenabled

Harbormaster failed remote builds in B9108: Diff 15745!Jan 22 2020, 21:09
Fabien updated this revision to Diff 15766.Jan 24 2020, 09:33

Undo failure in tests.
Pass the args through environment to see how Teamcity handles skipped tests.

teamcity added a comment.Jan 24 2020, 09:33

Build Bitcoin-ABC / Diffs / Diff Testing started.

Harbormaster failed remote builds in B9125: Diff 15766!Jan 24 2020, 09:38
teamcity added a comment.Jan 24 2020, 09:38
Build Bitcoin-ABC / Diffs / Diff Testing failed.

Snippet of first build failure:

[09:38:07]i:			 [wallet_multiwallet.py --usecli] [Test Output]
2020-01-24T09:37:07.890000Z TestFramework (INFO): Initializing test directory /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_multiwallet_435
2020-01-24T09:37:14.190000Z TestFramework (INFO): Check for per-wallet settxfee call
2020-01-24T09:37:14.235000Z TestFramework (INFO): Test dynamic wallet loading
2020-01-24T09:37:14.817000Z TestFramework (INFO): Load first wallet
2020-01-24T09:37:14.838000Z TestFramework (INFO): Load second wallet
2020-01-24T09:37:14.858000Z TestFramework (INFO): Load remaining wallets
2020-01-24T09:37:15.018000Z TestFramework (INFO): Test dynamic wallet creation.
2020-01-24T09:37:15.180000Z TestFramework (INFO): Test dynamic wallet unloading
2020-01-24T09:37:15.410000Z TestFramework (INFO): Test wallet backup
2020-01-24T09:37:17.388000Z TestFramework (INFO): Stopping nodes
2020-01-24T09:37:17.607000Z TestFramework (INFO): Cleaning up /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_multiwallet_435 on exit
2020-01-24T09:37:17.607000Z TestFramework (INFO): Tests successful
[09:38:07]i:		 [bitcoin_abc_tests] wallet_resendwallettransactions.py
[09:38:07]i:			 [wallet_resendwallettransactions.py] [Test Output]
2020-01-24T09:38:03.926000Z TestFramework (INFO): Initializing test directory /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_resendwallettransactions_522
2020-01-24T09:38:04.833000Z TestFramework (INFO): Stopping nodes
2020-01-24T09:38:05.137000Z TestFramework (INFO): Cleaning up /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_resendwallettransactions_522 on exit
2020-01-24T09:38:05.138000Z TestFramework (INFO): Tests successful
[09:38:07]i:		 [bitcoin_abc_tests] wallet_txn_clone.py
[09:38:07]i:			 [wallet_txn_clone.py] [Test Output]
2020-01-24T09:37:57.627000Z TestFramework (INFO): Initializing test directory /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_txn_clone_498
2020-01-24T09:38:00.025000Z TestFramework (INFO): Stopping nodes
2020-01-24T09:38:00.378000Z TestFramework (INFO): Cleaning up /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_txn_clone_498 on exit
2020-01-24T09:38:00.378000Z TestFramework (INFO): Tests successful
[09:38:07]i:		 [bitcoin_abc_tests] wallet_txn_clone.py --mineblock
[09:38:07]i:			 [wallet_txn_clone.py --mineblock] [Test Output]
2020-01-24T09:37:48.534000Z TestFramework (INFO): Initializing test directory /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_txn_clone_478
2020-01-24T09:37:50.742000Z TestFramework (INFO): Stopping nodes
2020-01-24T09:37:51.353000Z TestFramework (INFO): Cleaning up /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_txn_clone_478 on exit
2020-01-24T09:37:51.353000Z TestFramework (INFO): Tests successful
[09:38:07]i:		 [bitcoin_abc_tests] wallet_txn_doublespend.py
[09:38:07]i:			 [wallet_txn_doublespend.py] [Test Output]
2020-01-24T09:37:48.994000Z TestFramework (INFO): Initializing test directory /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_txn_doublespend_479
2020-01-24T09:37:51.351000Z TestFramework (INFO): Stopping nodes
2020-01-24T09:37:51.957000Z TestFramework (INFO): Cleaning up /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_txn_doublespend_479 on exit
2020-01-24T09:37:51.958000Z TestFramework (INFO): Tests successful
[09:38:07]i:		 [bitcoin_abc_tests] wallet_txn_doublespend.py --mineblock
[09:38:07]i:			 [wallet_txn_doublespend.py --mineblock] [Test Output]
2020-01-24T09:37:50.297000Z TestFramework (INFO): Initializing test directory /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_txn_doublespend_480
2020-01-24T09:37:51.904000Z TestFramework (INFO): Stopping nodes
2020-01-24T09:37:52.459000Z TestFramework (INFO): Cleaning up /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_txn_doublespend_480 on exit
2020-01-24T09:37:52.459000Z TestFramework (INFO): Tests successful
[09:38:07]i:		 [bitcoin_abc_tests] wallet_zapwallettxes.py
[09:38:07]i:			 [wallet_zapwallettxes.py] [Test Output]
2020-01-24T09:37:35.246000Z TestFramework (INFO): Initializing test directory /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_zapwallettxes_457
2020-01-24T09:37:35.613000Z TestFramework (INFO): Mining blocks...
2020-01-24T09:37:37.188000Z TestFramework (INFO): Stopping nodes
2020-01-24T09:37:37.441000Z TestFramework (INFO): Cleaning up /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/test/tmp/bitcoin_test_runner_20200124_093621/wallet_zapwallettxes_457 on exit
2020-01-24T09:37:37.441000Z TestFramework (INFO): Tests successful
[09:38:07]E: Ant JUnit report watcher
[09:38:07] :	 [Ant JUnit report watcher] 105 reports found for paths:
[09:38:07] :	 [Ant JUnit report watcher] +:build/test_bitcoin.xml
[09:38:07] :	 [Ant JUnit report watcher] +:build/**/junit_results*.xml
[09:38:07] :	 [Ant JUnit report watcher] +:build/**/Master_Test_Suite*.xml
[09:38:07]E:	 [Ant JUnit report watcher] Parsing errors
[09:38:07]E:		 [Parsing errors] Failed to parse 2 reports
[09:38:07]E:		 [Parsing errors] build/src/test/Master_Test_Suite.xml: Content is not allowed in trailing section.
[09:38:07]E:		 [Parsing errors] build/src/test/Master_Test_Suite_2.xml: Content is not allowed in trailing section.
[09:38:10]E:	 [Ant JUnit report watcher] Failed to parse xml reports
Fabien updated this revision to Diff 15767.Jan 24 2020, 09:56

This was as bad as expected, revert to a more demonstrative (working) version.

teamcity added a comment.Jan 24 2020, 09:56

Build Bitcoin-ABC / Diffs / Diff Testing started.

teamcity added a comment.Jan 24 2020, 10:03

Build Bitcoin-ABC / Diffs / Diff Testing passed.

Harbormaster completed remote builds in B9126: Diff 15767.Jan 24 2020, 10:03
jasonbcox requested changes to this revision.Jan 24 2020, 18:07
jasonbcox added a subscriber: jasonbcox.
jasonbcox added inline comments.
contrib/teamcity/build-configurations.sh
118 ↗(On Diff #15767)

Maintaining this list is a bit iffy, but I don't think the alternative of a new target that's ninja check-everything-except-bitcoin makes much sense either. I suppose if someone adds a new target, it's also there job to make sure it gets run on CI.

src/test/test_bitcoin_main.cpp
59 ↗(On Diff #15767)

This can be more informative: " with the next upgrade activated: " + nextUpgradeFlag

This revision now requires changes to proceed.Jan 24 2020, 18:07
Fabien added inline comments.Jan 24 2020, 21:01
contrib/teamcity/build-configurations.sh
118 ↗(On Diff #15767)

Ideally I would like to do something like https://reviews.bitcoinabc.org/D5042?id=15766, but I'm facing limitations both from boost and Teamcity.

src/test/test_bitcoin_main.cpp
59 ↗(On Diff #15767)

It can even be a custom argument in the spirit of D4943, but I needed something "quick and dirty" for challenging Teamcity first.

jasonbcox accepted this revision.Jan 24 2020, 21:12
This revision is now accepted and ready to land.Jan 24 2020, 21:12
Fabien planned changes to this revision.Feb 6 2020, 11:14
Fabien retitled this revision from [WIP] Make the CI record and track the unit tests to Make the CI record and track the unit tests.Feb 6 2020, 11:16
Fabien edited the test plan for this revision. (Show Details)
Fabien updated this revision to Diff 16070.Feb 6 2020, 11:19

Cleanup, add an option to set the test suite name in the spirit of D4943.
Now ready for review.

This revision is now accepted and ready to land.Feb 6 2020, 11:19
teamcity added a comment.Feb 6 2020, 11:19

Build Bitcoin-ABC / Diffs / Diff Testing started.

teamcity added a comment.Feb 6 2020, 11:22
Build Bitcoin-ABC / Diffs / Diff Testing failed.

Snippet of first build failure:

[11:21:53] :	 [Step 1/1] [73/109] leveldb: testing env_test
[11:21:53] :	 [Step 1/1] [74/109] bitcoin-seeder: testing seeder_tests
[11:21:53] :	 [Step 1/1] [75/109] Running bitcoin-seeder test suite
[11:21:53] :	 [Step 1/1] PASSED: bitcoin-seeder test suite
[11:21:53] :	 [Step 1/1] [76/109] Test Bitcoin RPC authentication...
[11:21:53] :	 [Step 1/1] ...
[11:21:53] :	 [Step 1/1] ----------------------------------------------------------------------
[11:21:53] :	 [Step 1/1] Ran 3 tests in 0.003s
[11:21:53] :	 [Step 1/1] 
[11:21:53] :	 [Step 1/1] OK
[11:21:53] :	 [Step 1/1] [77/109] cd /home/teamcity/buildAgent/work/c4a5708f2bae7929/contrib/devtools/chainparams && /usr/bin/python3 ./test_make_chainparams.py
[11:21:53] :	 [Step 1/1] .....
[11:21:53] :	 [Step 1/1] ----------------------------------------------------------------------
[11:21:53] :	 [Step 1/1] Ran 5 tests in 0.001s
[11:21:53] :	 [Step 1/1] 
[11:21:53] :	 [Step 1/1] OK
[11:21:53] :	 [Step 1/1] [78/109] Building CXX object src/leveldb/CMakeFiles/bloom_test.dir/util/bloom_test.cc.o
[11:21:53] :	 [Step 1/1] [79/109] Building CXX object src/leveldb/CMakeFiles/autocompact_test.dir/db/autocompact_test.cc.o
[11:21:53] :	 [Step 1/1] [80/109] Linking CXX executable src/leveldb/bloom_test
[11:21:53] :	 [Step 1/1] [81/109] leveldb: testing bloom_test
[11:21:53] :	 [Step 1/1] [82/109] leveldb: testing fault_injection_test
[11:21:53] :	 [Step 1/1] [83/109] Linking CXX executable src/leveldb/autocompact_test
[11:21:53] :	 [Step 1/1] [84/109] leveldb: testing corruption_test
[11:21:53] :	 [Step 1/1] [85/109] leveldb: testing skiplist_test
[11:21:54] :	 [Step 1/1] [86/109] leveldb: testing arena_test
[11:21:54] :	 [Step 1/1] [87/109] leveldb: testing table_test
[11:21:54] :	 [Step 1/1] [88/109] Automatic MOC for target test_bitcoin-qt
[11:21:54] :	 [Step 1/1] [89/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/util.cpp.o
[11:21:54] :	 [Step 1/1] [90/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/bitcoinaddressvalidatortests.cpp.o
[11:21:54] :	 [Step 1/1] [91/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/compattests.cpp.o
[11:21:54] :	 [Step 1/1] [92/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/guiutiltests.cpp.o
[11:21:54] :	 [Step 1/1] [93/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/apptests.cpp.o
[11:21:54] :	 [Step 1/1] [94/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/uritests.cpp.o
[11:21:54] :	 [Step 1/1] [95/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/test_main.cpp.o
[11:21:54] :	 [Step 1/1] [96/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/rpcnestedtests.cpp.o
[11:21:54] :	 [Step 1/1] [97/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/addressbooktests.cpp.o
[11:21:54] :	 [Step 1/1] [98/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/__/__/wallet/test/wallet_test_fixture.cpp.o
[11:21:54] :	 [Step 1/1] [99/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/wallettests.cpp.o
[11:21:54] :	 [Step 1/1] [100/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/test_bitcoin-qt_autogen/mocs_compilation.cpp.o
[11:21:55] :	 [Step 1/1] [101/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/paymentservertests.cpp.o
[11:21:55] :	 [Step 1/1] [102/109] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/__/__/test/test_bitcoin.cpp.o
[11:21:56] :	 [Step 1/1] [103/109] leveldb: testing issue178_test
[11:21:58] :	 [Step 1/1] [104/109] Linking CXX executable src/qt/test/test_bitcoin-qt
[11:21:59] :	 [Step 1/1] [105/109] bitcoin-qt: testing test_bitcoin-qt
[11:21:59] :	 [Step 1/1] [106/109] Running bitcoin-qt test suite
[11:21:59] :	 [Step 1/1] PASSED: bitcoin-qt test suite
[11:22:15] :	 [Step 1/1] [107/109] leveldb: testing autocompact_test
[11:22:49] :	 [Step 1/1] [108/109] leveldb: testing db_test
[11:22:49] :	 [Step 1/1] [109/109] Running leveldb test suite
[11:22:49] :	 [Step 1/1] PASSED: leveldb test suite
[11:22:49]W:	 [Step 1/1] + check-secp256k1 check-univalue
[11:22:49]W:	 [Step 1/1] ./contrib/teamcity/build-configurations.sh: line 143: check-secp256k1: command not found
[11:22:49] :	 [Step 1/1] *** Output of /tmp/sanitizer_logs/*.log.* ***
[11:22:49]W:	 [Step 1/1] ++ print_sanitizers_log
[11:22:49]W:	 [Step 1/1] ++ for log in "${SAN_LOG_DIR}"/*.log.*
[11:22:49]W:	 [Step 1/1] ++ echo '*** Output of /tmp/sanitizer_logs/*.log.* ***'
[11:22:49]W:	 [Step 1/1] ++ cat '/tmp/sanitizer_logs/*.log.*'
[11:22:49]W:	 [Step 1/1] cat: '/tmp/sanitizer_logs/*.log.*': No such file or directory
[11:22:49]W:	 [Step 1/1] Process exited with code 1
[11:22:50]E:	 [Step 1/1] Process exited with code 1 (Step: Command Line)
Harbormaster failed remote builds in B9297: Diff 16070!Feb 6 2020, 11:22
Fabien updated this revision to Diff 16071.Feb 6 2020, 11:26

Fix missing line continuation.

teamcity added a comment.Feb 6 2020, 11:26

Build Bitcoin-ABC / Diffs / Diff Testing started.

teamcity added a comment.Feb 6 2020, 11:33

Build Bitcoin-ABC / Diffs / Diff Testing passed.

Harbormaster completed remote builds in B9298: Diff 16071.Feb 6 2020, 11:33
jasonbcox added inline comments.Feb 6 2020, 18:53
contrib/teamcity/build-configurations.sh
57 ↗(On Diff #16071)

Nit: It may be best for readability to put each arg (especially --) on it's own line:

./src/test/test_bitcoin \
  --logger=HRF:JUNIT,${UNIT_TESTS_JUNIT_LOG_LEVEL},${TEST_BITCOIN_JUNIT} \
  -- \
  -testsuitename="${TEST_BITCOIN_SUITE_NAME}" \
  "${@:2}"
teamcity added a comment.Feb 6 2020, 18:53

Build Bitcoin-ABC / Diffs / Diff Testing started.

teamcity added a comment.Feb 6 2020, 19:03

Build Bitcoin-ABC / Diffs / Diff Testing passed.

jasonbcox requested changes to this revision.Feb 6 2020, 19:05
jasonbcox added inline comments.
contrib/teamcity/build-configurations.sh
88 ↗(On Diff #16071)

Aren't the sanitizer tests reducing coverage for all tools and libs that are compiled with cmake?

This revision now requires changes to proceed.Feb 6 2020, 19:05
jasonbcox added inline comments.Feb 6 2020, 19:09
contrib/teamcity/build-configurations.sh
45 ↗(On Diff #16071)

Apparently this log level produces strange results in ASAN (I haven't tried UBSAN or TSAN yet): https://build.bitcoinabc.org/viewLog.html?buildId=28319&buildTypeId=BitcoinABC_BitcoinAbcStaging&tab=buildLog&state=60&_focus=1070#_state=60

This isn't an actual error and appears to be expected: https://stackoverflow.com/a/54951825/6312728 It's unfortunate that it throws garbage into the logs. Please take a look and see if there's an easy way to mitigate this.

Fabien updated this revision to Diff 16084.Feb 6 2020, 20:25

Maintain coverage for the sanitizer tests.
Do not track autotool, as it makes little sense. Revert to make check.
Address feedbacks.

teamcity added a comment.Feb 6 2020, 20:26

Build Bitcoin-ABC / Diffs / Diff Testing started.

Fabien added inline comments.Feb 6 2020, 20:28
contrib/teamcity/build-configurations.sh
45 ↗(On Diff #16071)

message is not the cause. This seems to be a bug in boost, when there are multiple loggers enabled (independently of the log level).
Unfortunately I have no solution at the moment apart trying to remove the BOOST_TEST_CONTEXT occurrence, and I don't want to do this.

teamcity added a comment.Feb 6 2020, 20:44

Build Bitcoin-ABC / Diffs / Diff Testing passed.

Harbormaster completed remote builds in B9308: Diff 16084.Feb 6 2020, 20:44
deadalnix accepted this revision.Feb 7 2020, 02:14

This is a bit of a concern, because now a dev adding a new test suite needs to update this to make sure it runs. Failing to do so will just result in not running the test suite, silently. You need to figure this out.

markblundeberg added a subscriber: markblundeberg.Feb 7 2020, 02:38
In D5042#122099, @Fabien wrote:

Increase log level from warning to message.
Make a test to fail intentionally to see how the CI reacts.

Why message level, by the way?

Fabien added a comment.Feb 7 2020, 07:16

@deadalnix I agree this diff has a few downsides. I'll see how to fix this, I have a few ideas.

@markblundeberg The message log level only applies to the Junit file (well, boost bug excepted). I found it to be the sweet spot since it produces more output than the stdout log by default, but still does not produce an output for each assertion (that makes the report file reach ~1.1GB !).

Fabien added inline comments.Feb 7 2020, 08:43
contrib/teamcity/build-configurations.sh
45 ↗(On Diff #16071)

See https://github.com/boostorg/test/issues/251

jasonbcox accepted this revision.Feb 7 2020, 19:16
This revision is now accepted and ready to land.Feb 7 2020, 19:16
Closed by commit rABC41d6f76ea618: Make the CI record and track the unit tests (authored by Fabien). · Explain WhyFeb 7 2020, 19:21
This revision was automatically updated to reflect the committed changes.
Fabien added a commit: rABC41d6f76ea618: Make the CI record and track the unit tests.
Fabien mentioned this in D5213: [CI] Deduplicate test_bitcoin run for the TSAN build.Feb 7 2020, 21:12
Fabien mentioned this in rABCfc405beaa159: [CI] Deduplicate test_bitcoin run for the TSAN build.Feb 7 2020, 21:53
markblundeberg mentioned this in D5232: util_tests.test_LockDirectory: avoid exit in forked process.Feb 9 2020, 15:14
markblundeberg mentioned this in D3898: Use best-fit strategy in Arena, now O(log(n)) instead O(n).Feb 15 2020, 11:35
Fabien added a commit: rSTAGING41d6f76ea618: Make the CI record and track the unit tests.Apr 7 2020, 22:54
Fabien mentioned this in rSTAGINGfc405beaa159: [CI] Deduplicate test_bitcoin run for the TSAN build.

Revision Contents
Changeset List

PathSize
contrib/
teamcity/
101 lines
src/
test/
12 lines

Diff 16148

View Options

contrib/teamcity/build-configurations.sh

Loading...
View Options

src/test/test_bitcoin_main.cpp

Loading...