Page MenuHomePhabricator

[CMAKE] Generate junit output for boost unit tests
ClosedPublic

Authored by Fabien on Jul 22 2020, 13:50.

Details

Reviewers
deadalnix
Group Reviewers
Restricted Project
Commits
rABC98b979b0acf1: [CMAKE] Generate junit output for boost unit tests
Summary

This diff makes the individual boost tests output a junit report XML,
and merge them in order to produce a test suite junit report. The final
report is located under a test/junit dir under the build directory.

This requires a few behavior changes and implies some limitations:

  • A new option ENABLE_JUNIT_REPORT (disabled by default) let the boost tests generate a Junit report, and proceed incremental merge of the results to a test suite report.
  • A test failure will stop the build process, and the result won't appear in the Junit report by default. One need to run ninja with the -k0 flag to change this behavior and allow the test suite to run to completion.
  • The test suite duration reported is not the actual duration, because it can't account for the tests running parallel. The reported duration is equivalent to running the tests serially.
  • The incremental nature of the report makes it impossible to clear the content by running the tests. Each test will override its previous run but not impact the other tests, independently of what is being run. To clear the report, ninja clean should be used.
Test Plan
cmake -GNinja ..
ninja check

Check no report is generated.

cmake -GNinja .. -DENABLE_JUNIT_REPORT=ON
ninja check

Check the report are generated.

ninja clean

Check the test/junit directory no longer exist.

Edit a test from the bitcoin test suite to make it fail:

ninja -k0 check

Ensure all the tests including the failure are correctly reported.
Check the return code is non zero.

Event Timeline

Fabien requested review of this revision.Jul 22 2020, 13:50
deadalnix requested changes to this revision.Jul 22 2020, 14:17
deadalnix added a subscriber: deadalnix.

The idea is there, but the way it is deployed is unsound.

cmake/modules/TestSuite.cmake
32 ↗(On Diff #22422)

You do not need a new folder, especially since the logs are not done this way. Just use a naming convention. If you are worried, pass al the filename to the merge script explicitly (in fact, it's probably not a bad idea to do so so that it doesn't pickup results from deleted tests).

46 ↗(On Diff #22422)

There can be leftover crap in there. Pass all the files. if you do so, you don't need a folder.

cmake/utils/junit-reports-merge.py
121 ↗(On Diff #22422)

This is a bad idea as return code are typically truncated in a system dependent manner, often over 8bits. This means 256 failure will be success on many systems.

This revision now requires changes to proceed.Jul 22 2020, 14:17

Actually pass the list of tests rather than a directory.
Return a constant error code.

deadalnix requested changes to this revision.Jul 22 2020, 21:42
deadalnix added inline comments.
cmake/modules/TestSuite.cmake
116 ↗(On Diff #22424)

Use similar naming as is used for logs.

This revision now requires changes to proceed.Jul 22 2020, 21:42
cmake/modules/TestSuite.cmake
116 ↗(On Diff #22424)

The actual log naming is the buggy one as it allow for name collision, I will fix it in another diff.

Fabien edited the summary of this revision. (Show Details)

Rebase on top of D7024.

Snippet of first build failure:

[383/423] Generating forms/ui_sendcoinsentry.h
[384/423] Generating forms/ui_debugwindow.h
[385/423] Generating temp_bitcoin_locale.qrc
[386/423] Generating qrc_bitcoin.cpp
[387/423] Linking CXX executable src/bitcoind
[388/423] Building CXX object src/bench/CMakeFiles/bitcoin-bench.dir/prevector.cpp.o
[389/423] Generating qrc_bitcoin_locale.cpp
[390/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/bitcoinunits.cpp.o
[391/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/csvmodelwriter.cpp.o
[392/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/bitcoinamountfield.cpp.o
[393/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/bitcoinaddressvalidator.cpp.o
[394/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/notificator.cpp.o
[395/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/bitcoingui.cpp.o
[396/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/intro.cpp.o
[397/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/bitcoin.cpp.o
[398/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/modaloverlay.cpp.o
[399/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/networkstyle.cpp.o
[400/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/guiutil.cpp.o
[401/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/platformstyle.cpp.o
[402/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/optionsdialog.cpp.o
[403/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/optionsmodel.cpp.o
[404/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/qvalidatedlineedit.cpp.o
[405/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/qvaluecombobox.cpp.o
[406/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/peertablemodel.cpp.o
[407/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/trafficgraphwidget.cpp.o
[408/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/splashscreen.cpp.o
[409/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/rpcconsole.cpp.o
[410/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/utilitydialog.cpp.o
[411/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/bitcoin-qt-base_autogen/mocs_compilation.cpp.o
[412/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/qrc_bitcoin.cpp.o
[413/423] Building CXX object src/bench/CMakeFiles/bitcoin-bench.dir/checkblock.cpp.o
[414/423] Building CXX object src/bench/CMakeFiles/bitcoin-bench.dir/duplicate_inputs.cpp.o
[415/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/clientmodel.cpp.o
[416/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/bantablemodel.cpp.o
[417/423] Linking CXX executable src/bench/bitcoin-bench
[418/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/qrc_bitcoin_locale.cpp.o
[419/423] Linking CXX static library src/qt/libbitcoin-qt-base.a
[420/423] Automatic MOC for target bitcoin-qt
[421/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt.dir/bitcoin-qt_autogen/mocs_compilation.cpp.o
[422/423] Building CXX object src/qt/CMakeFiles/bitcoin-qt.dir/main.cpp.o
[423/423] Linking CXX executable src/qt/bitcoin-qt
[1/18] Automatic MOC for target bitcoin-qt-protobuf
[2/17] Automatic MOC for target bitcoin-qt-base
[3/15] Automatic MOC for target test_bitcoin-qt
[4/15] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/bitcoinaddressvalidatortests.cpp.o
[5/15] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/compattests.cpp.o
[6/15] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/rpcnestedtests.cpp.o
[7/15] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/test_main.cpp.o
[8/15] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/uritests.cpp.o
[9/15] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/util.cpp.o
[10/15] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/apptests.cpp.o
[11/15] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/guiutiltests.cpp.o
[12/15] Building CXX object src/qt/test/CMakeFiles/test_bitcoin-qt.dir/test_bitcoin-qt_autogen/mocs_compilation.cpp.o
[13/15] Linking CXX executable src/qt/test/test_bitcoin-qt
[14/15] bitcoin-qt: testing test_bitcoin-qt
FAILED: src/qt/test/CMakeFiles/check-bitcoin-qt-test_bitcoin-qt 
cd /work/abc-ci-builds/build-without-wallet/src/qt/test && /usr/bin/cmake -E env /work/cmake/utils/test_wrapper.sh "<<<<<<<" HEAD bitcoin-qt-test_bitcoin-qt.log /work/abc-ci-builds/build-without-wallet/src/qt/test/test_bitcoin-qt ======= test_bitcoin-qt.log /work/abc-ci-builds/build-without-wallet/src/qt/test/test_bitcoin-qt ">>>>>>>" d35c6cc715... [CMAKE] Generate junit output for boost unit tests
/work/cmake/utils/test_wrapper.sh: 28: /work/cmake/utils/test_wrapper.sh: HEAD: not found
ninja: build stopped: subcommand failed.
Build build-without-wallet failed with exit code 1

Snippet of first build failure:

[463/471] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/rpcconsole.cpp.o
[464/471] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/signverifymessagedialog.cpp.o
[465/471] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/walletmodel.cpp.o
[466/471] Building CXX object src/qt/CMakeFiles/bitcoin-qt-base.dir/sendcoinsdialog.cpp.o
[467/471] Linking CXX static library src/qt/libbitcoin-qt-base.a
[468/471] Automatic MOC for target bitcoin-qt
[469/471] Building CXX object src/qt/CMakeFiles/bitcoin-qt.dir/bitcoin-qt_autogen/mocs_compilation.cpp.o
[470/471] Building CXX object src/qt/CMakeFiles/bitcoin-qt.dir/main.cpp.o
[471/471] Linking CXX executable src/qt/bitcoin-qt
[1/292] Generating data/base58_encode_decode.json.h
[2/292] Generating data/key_io_invalid.json.h
[3/292] Generating data/blockfilters.json.h
[4/292] Generating data/key_io_valid.json.h
[5/292] Generating data/tx_invalid.json.h
[6/292] Generating data/tx_valid.json.h
[7/292] Building C object src/secp256k1/CMakeFiles/secp256k1-exhaustive_tests.dir/src/tests_exhaustive.c.o
[8/292] Building CXX object src/pow/test/CMakeFiles/test-pow.dir/eda_tests.cpp.o
[9/292] Building C object src/secp256k1/CMakeFiles/secp256k1-tests.dir/src/tests.c.o
[10/292] Building CXX object src/avalanche/test/CMakeFiles/test-avalanche.dir/util.cpp.o
[11/292] Building CXX object src/avalanche/test/CMakeFiles/test-avalanche.dir/peermanager_tests.cpp.o
[12/292] Building CXX object src/pow/test/CMakeFiles/test-pow.dir/fixture.cpp.o
[13/292] Building CXX object src/pow/test/CMakeFiles/test-pow.dir/daa_tests.cpp.o
[14/292] Building CXX object src/avalanche/test/CMakeFiles/test-avalanche.dir/processor_tests.cpp.o
[15/292] Building CXX object src/seeder/test/CMakeFiles/test-seeder.dir/p2p_messaging_tests.cpp.o
[16/292] Automatic MOC for target bitcoin-qt-protobuf
[17/291] Linking C executable src/secp256k1/secp256k1-exhaustive_tests
[18/291] Linking C executable src/secp256k1/secp256k1-tests
[19/291] Test Bitcoin RPC authentication...
...
----------------------------------------------------------------------
Ran 3 tests in 0.004s

OK
[20/291] Automatic MOC for target bitcoin-qt-base
[21/289] secp256k1: testing secp256k1-exhaustive_tests
FAILED: src/secp256k1/CMakeFiles/check-secp256k1-secp256k1-exhaustive_tests 
cd /work/abc-ci-builds/build-clang-10/src/secp256k1 && /usr/bin/cmake -E env /work/cmake/utils/test_wrapper.sh "<<<<<<<" HEAD secp256k1-secp256k1-exhaustive_tests.log /work/abc-ci-builds/build-clang-10/src/secp256k1/secp256k1-exhaustive_tests ======= secp256k1-exhaustive_tests.log /work/abc-ci-builds/build-clang-10/src/secp256k1/secp256k1-exhaustive_tests ">>>>>>>" d35c6cc715... [CMAKE] Generate junit output for boost unit tests
/work/cmake/utils/test_wrapper.sh: 28: /work/cmake/utils/test_wrapper.sh: HEAD: not found
[22/289] secp256k1: testing secp256k1-tests
FAILED: src/secp256k1/CMakeFiles/check-secp256k1-secp256k1-tests 
cd /work/abc-ci-builds/build-clang-10/src/secp256k1 && /usr/bin/cmake -E env /work/cmake/utils/test_wrapper.sh "<<<<<<<" HEAD secp256k1-secp256k1-tests.log /work/abc-ci-builds/build-clang-10/src/secp256k1/secp256k1-tests ======= secp256k1-tests.log /work/abc-ci-builds/build-clang-10/src/secp256k1/secp256k1-tests ">>>>>>>" d35c6cc715... [CMAKE] Generate junit output for boost unit tests
/work/cmake/utils/test_wrapper.sh: 28: /work/cmake/utils/test_wrapper.sh: HEAD: not found
[23/289] cd /work/contrib/devtools/chainparams && /usr/bin/python3.7 ./test_make_chainparams.py
.....
----------------------------------------------------------------------
Ran 5 tests in 0.001s

OK
[24/289] Generating data/script_tests.json.h
[25/289] Generating data/sighash.json.h
[26/289] Building CXX object src/avalanche/test/CMakeFiles/test-avalanche.dir/fixture.cpp.o
[27/289] Building CXX object src/seeder/test/CMakeFiles/test-seeder.dir/fixture.cpp.o
[28/289] Building CXX object src/seeder/test/CMakeFiles/test-seeder.dir/write_name_tests.cpp.o
[29/289] Building CXX object src/seeder/test/CMakeFiles/test-seeder.dir/parse_name_tests.cpp.o
[30/289] Building CXX object src/avalanche/test/CMakeFiles/test-avalanche.dir/proof_tests.cpp.o
[31/289] Linking CXX executable src/pow/test/test-pow
[32/289] Automatic MOC for target test_bitcoin-qt
[33/289] Test Bitcoin utilities...
ninja: build stopped: subcommand failed.
Build build-clang-10 failed with exit code 1
Fabien planned changes to this revision.Jul 23 2020, 06:49

Snippet of first build failure:

Assertion occurred in a following context:
    ../../src/test/sigcheckcount_tests.cpp:339
Assertion occurred in a following context:
    ../../src/test/sigcheckcount_tests.cpp:349
Assertion occurred in a following context:
    ../../src/test/sigcheckcount_tests.cpp:349
Assertion occurred in a following context:
    ../../src/test/sigcheckcount_tests.cpp:359
Assertion occurred in a following context:
    ../../src/test/sigcheckcount_tests.cpp:359
*** No errors detected
+ ninja check-leveldb
[1/82] Building C object src/leveldb/CMakeFiles/c_test.dir/db/c_test.c.o
[2/82] Building CXX object src/leveldb/CMakeFiles/hash_test.dir/util/hash_test.cc.o
[3/82] Building CXX object src/leveldb/CMakeFiles/env_posix_test.dir/util/env_posix_test.cc.o
[4/82] Building CXX object src/leveldb/CMakeFiles/env_test.dir/util/env_test.cc.o
[5/82] Building CXX object src/leveldb/CMakeFiles/arena_test.dir/util/arena_test.cc.o
[6/82] Building CXX object src/leveldb/CMakeFiles/bloom_test.dir/util/bloom_test.cc.o
[7/82] Building CXX object src/leveldb/CMakeFiles/dbformat_test.dir/db/dbformat_test.cc.o
[8/82] Building CXX object src/leveldb/CMakeFiles/leveldb_test_base.dir/util/testutil.cc.o
[9/82] Building CXX object src/leveldb/CMakeFiles/filename_test.dir/db/filename_test.cc.o
[10/82] Building CXX object src/leveldb/CMakeFiles/skiplist_test.dir/db/skiplist_test.cc.o
[11/82] Building CXX object src/leveldb/CMakeFiles/leveldb_test_base.dir/util/testharness.cc.o
[12/82] Building CXX object src/leveldb/CMakeFiles/crc32c_test.dir/util/crc32c_test.cc.o
[13/82] Linking CXX static library src/leveldb/libleveldb_test_base.a
[14/82] Building CXX object src/leveldb/CMakeFiles/coding_test.dir/util/coding_test.cc.o
[15/82] Linking CXX executable src/leveldb/c_test
[16/82] Linking CXX executable src/leveldb/hash_test
[17/82] Linking CXX executable src/leveldb/bloom_test
[18/82] Linking CXX executable src/leveldb/coding_test
[19/82] Linking CXX executable src/leveldb/dbformat_test
[20/82] Linking CXX executable src/leveldb/skiplist_test
[21/82] Linking CXX executable src/leveldb/filename_test
[22/82] Linking CXX executable src/leveldb/env_test
[23/82] Linking CXX executable src/leveldb/env_posix_test
[24/82] Linking CXX executable src/leveldb/arena_test
[25/82] Linking CXX executable src/leveldb/crc32c_test
[26/82] Building CXX object src/leveldb/CMakeFiles/cache_test.dir/util/cache_test.cc.o
[27/82] Linking CXX executable src/leveldb/cache_test
[28/82] Building CXX object src/leveldb/CMakeFiles/write_batch_test.dir/db/write_batch_test.cc.o
[29/82] Linking CXX executable src/leveldb/write_batch_test
[30/82] leveldb: testing env_posix_test
FAILED: src/leveldb/CMakeFiles/check-leveldb-env_posix_test 
cd /work/abc-ci-builds/build-diff/src/leveldb && /usr/bin/cmake -E env /work/cmake/utils/test_wrapper.sh "<<<<<<<" HEAD leveldb-env_posix_test.log /work/abc-ci-builds/build-diff/src/leveldb/env_posix_test ======= env_posix_test.log /work/abc-ci-builds/build-diff/src/leveldb/env_posix_test ">>>>>>>" d35c6cc715... [CMAKE] Generate junit output for boost unit tests
/work/cmake/utils/test_wrapper.sh: 28: /work/cmake/utils/test_wrapper.sh: HEAD: not found
[31/82] Building CXX object src/leveldb/CMakeFiles/autocompact_test.dir/db/autocompact_test.cc.o
[32/82] Building CXX object src/leveldb/CMakeFiles/issue178_test.dir/issues/issue178_test.cc.o
[33/82] Building CXX object src/leveldb/CMakeFiles/version_edit_test.dir/db/version_edit_test.cc.o
[34/82] Building CXX object src/leveldb/CMakeFiles/issue200_test.dir/issues/issue200_test.cc.o
[35/82] Building CXX object src/leveldb/CMakeFiles/memenv_test.dir/helpers/memenv/memenv_test.cc.o
[36/82] Building CXX object src/leveldb/CMakeFiles/version_set_test.dir/db/version_set_test.cc.o
[37/82] Building CXX object src/leveldb/CMakeFiles/filter_block_test.dir/table/filter_block_test.cc.o
[38/82] Building CXX object src/leveldb/CMakeFiles/table_test.dir/table/table_test.cc.o
[39/82] Building CXX object src/leveldb/CMakeFiles/corruption_test.dir/db/corruption_test.cc.o
[40/82] Building CXX object src/leveldb/CMakeFiles/fault_injection_test.dir/db/fault_injection_test.cc.o
[41/82] Building CXX object src/leveldb/CMakeFiles/log_test.dir/db/log_test.cc.o
[42/82] Building CXX object src/leveldb/CMakeFiles/recovery_test.dir/db/recovery_test.cc.o
[43/82] Building CXX object src/leveldb/CMakeFiles/db_test.dir/db/db_test.cc.o
ninja: build stopped: subcommand failed.
Build build-diff failed with exit code 1
Fabien edited the test plan for this revision. (Show Details)

Fix bad rebase.

Address some feedaback from D7004 https://reviews.bitcoinabc.org/D7004#inline-41952:

  • Move the report directory to test/junit
  • Clean the report dir with ninja clean
Fabien edited the test plan for this revision. (Show Details)
deadalnix requested changes to this revision.Jul 24 2020, 14:12
deadalnix added inline comments.
cmake/modules/TestSuite.cmake
46 ↗(On Diff #22453)

Now that this is here, it can be removed from src/test/CMakeFiles.txt

118 ↗(On Diff #22453)

Why do all the test are prefixed with junit ? It's be preferable that each test has its output next to its log and use a similar name.

This revision now requires changes to proceed.Jul 24 2020, 14:12
Fabien planned changes to this revision.Jul 24 2020, 14:31
Fabien added inline comments.
cmake/modules/TestSuite.cmake
118 ↗(On Diff #22453)

Just an attempt to make it clear, as an XML file can be anything. I can update that.

Remove junit_ prefix everywhere

cmake/modules/TestSuite.cmake
82 ↗(On Diff #22500)

Doesn't this break ninja check-foo-bar functionality to run a test individually?

deadalnix requested changes to this revision.Jul 25 2020, 16:04
This revision now requires changes to proceed.Jul 25 2020, 16:04
Fabien planned changes to this revision.Jul 27 2020, 20:07
Fabien added inline comments.
cmake/modules/TestSuite.cmake
82 ↗(On Diff #22500)

It does indeed, and fixing this requires some fundamental changes to this design. I'll submit another version soon.

Fabien edited the test plan for this revision. (Show Details)

Change the behavior to make it possible to run a single test target without dropping the return code.

Ths summary has been updated to reflect the new behavior, which differs from the previous solution.

Accepting because I think this is good now. Make sure you look at the comments and see if they can be addressed somehow.

CMakeLists.txt
41

Maybe this option should be declared in TestSuite.cmake ? If that works - which i'm not sure it does, tbh, I think that'd be better.

cmake/modules/TestSuite.cmake
88

Isn't test/tmp already in this list? It seems like it should be unconditional. cmake/ninja doesn't complain when something doesn't exists and is in the delete list. Less branching in the behavior is always better.

This revision is now accepted and ready to land.Jul 27 2020, 22:49
CMakeLists.txt
41

Yes that will work with no issue. I was hesitant between the 2 places because I'll need it also for functional tests and maybe others that may not use the module, but it can still be moved when needed.

cmake/modules/TestSuite.cmake
88

Good catch it is from the test/CMakeLists.txt, I'll deduplicate and make it global.

This comment was removed by Fabien.