While this is not doign much, this will allows us to start boltin on it.
Details
Details
- Reviewers
Fabien - Group Reviewers
Restricted Project - Commits
- rSTAGING01d2a4a9c83f: [avalanche] Start event loop at node startup
rABC01d2a4a9c83f: [avalanche] Start event loop at node startup
Ran the extended test suite.
Ran under ubsan.
I do not have a good way to run tsan and asan locally. Having a doc on how to do so and get them green would be beneficial.
Diff Detail
Diff Detail
- Repository
- rABC Bitcoin ABC
- Branch
- avainstance
- Lint
Lint Passed - Unit
No Test Coverage - Build Status
Buildable 9686 Build 17269: Default Diff Build & Tests Build 17268: arc lint + arc unit
Event Timeline
Comment Actions
Build Bitcoin-ABC / Diffs / Diff Testing failed.
Snippet of first build failure:
[08:32:57] : [Step 1/1] #0 0x7f4dd8e1a6bf (/lib/x86_64-linux-gnu/libpthread.so.0+0xa6bf) [08:32:57] : [Step 1/1] #1 0x562089e8630f in __gthread_mutex_lock(pthread_mutex_t*) /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/x86_64-linux-gnu/c++/8/bits/gthr-default.h:748:12 [08:32:57] : [Step 1/1] #2 0x562089ef33e4 in std::mutex::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/std_mutex.h:103:17 [08:32:57] : [Step 1/1] #3 0x562089ef3343 in std::unique_lock<std::mutex>::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/std_mutex.h:267:17 [08:32:57] : [Step 1/1] #4 0x562089ef2df8 in UniqueLock<AnnotatedMixin<std::mutex>, std::unique_lock<std::mutex> >::Enter(char const*, char const*, int) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/./sync.h:124:19 [08:32:57] : [Step 1/1] #5 0x562089e98450 in UniqueLock<AnnotatedMixin<std::mutex>, std::unique_lock<std::mutex> >::UniqueLock(AnnotatedMixin<std::mutex>&, char const*, char const*, int, bool) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/./sync.h:146:13 [08:32:57] : [Step 1/1] #6 0x56208a851e10 in AvalancheProcessor::stopEventLoop() /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/avalanche.cpp:325:5 [08:32:57] : [Step 1/1] #7 0x562089e4a9e3 in Shutdown(InitInterfaces&) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/init.cpp:200:18 [08:32:57] : [Step 1/1] #8 0x562089e3be4a in AppInit(int, char**) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:214:5 [08:32:57] : [Step 1/1] #9 0x562089e3a358 in main /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:229:13 [08:32:57] : [Step 1/1] #10 0x7f4dd8c4809a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a) [08:32:57] : [Step 1/1] #11 0x562089d60389 in _start (/home/teamcity/buildAgent/work/c4a5708f2bae7929/build/src/bitcoind+0x1b8389) [08:32:57] : [Step 1/1] [08:32:57] : [Step 1/1] AddressSanitizer can not provide additional info. [08:32:57] : [Step 1/1] SUMMARY: AddressSanitizer: SEGV (/lib/x86_64-linux-gnu/libpthread.so.0+0xa6bf) [08:32:57] : [Step 1/1] ==29188==ABORTING [08:32:57] : [Step 1/1] *** Output of /tmp/sanitizer_logs/ubsan.log.31363 *** [08:32:57] : [Step 1/1] ================================================================= [08:32:57] : [Step 1/1] ==31363==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000478 (pc 0x7f4802b2a6c0 bp 0x7fff998866b0 sp 0x7fff99886688 T0) [08:32:57] : [Step 1/1] ==31363==The signal is caused by a READ memory access. [08:32:57] : [Step 1/1] ==31363==Hint: address points to the zero page. [08:32:57] : [Step 1/1] #0 0x7f4802b2a6bf (/lib/x86_64-linux-gnu/libpthread.so.0+0xa6bf) [08:32:57] : [Step 1/1] #1 0x560f5a0ff30f in __gthread_mutex_lock(pthread_mutex_t*) /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/x86_64-linux-gnu/c++/8/bits/gthr-default.h:748:12 [08:32:57] : [Step 1/1] #2 0x560f5a16c3e4 in std::mutex::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/std_mutex.h:103:17 [08:32:57] : [Step 1/1] #3 0x560f5a16c343 in std::unique_lock<std::mutex>::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/std_mutex.h:267:17 [08:32:57] : [Step 1/1] #4 0x560f5a16bdf8 in UniqueLock<AnnotatedMixin<std::mutex>, std::unique_lock<std::mutex> >::Enter(char const*, char const*, int) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/./sync.h:124:19 [08:32:57] : [Step 1/1] #5 0x560f5a111450 in UniqueLock<AnnotatedMixin<std::mutex>, std::unique_lock<std::mutex> >::UniqueLock(AnnotatedMixin<std::mutex>&, char const*, char const*, int, bool) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/./sync.h:146:13 [08:32:57] : [Step 1/1] #6 0x560f5aacae10 in AvalancheProcessor::stopEventLoop() /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/avalanche.cpp:325:5 [08:32:57] : [Step 1/1] #7 0x560f5a0c39e3 in Shutdown(InitInterfaces&) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/init.cpp:200:18 [08:32:57] : [Step 1/1] #8 0x560f5a0b4e4a in AppInit(int, char**) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:214:5 [08:32:57] : [Step 1/1] #9 0x560f5a0b3358 in main /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:229:13 [08:32:57] : [Step 1/1] #10 0x7f480295809a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a) [08:32:57] : [Step 1/1] #11 0x560f59fd9389 in _start (/home/teamcity/buildAgent/work/c4a5708f2bae7929/build/src/bitcoind+0x1b8389) [08:32:57] : [Step 1/1] [08:32:57] : [Step 1/1] AddressSanitizer can not provide additional info. [08:32:57] : [Step 1/1] SUMMARY: AddressSanitizer: SEGV (/lib/x86_64-linux-gnu/libpthread.so.0+0xa6bf) [08:32:57] : [Step 1/1] ==31363==ABORTING [08:32:57] : [Step 1/1] *** Output of /tmp/sanitizer_logs/ubsan.log.31825 *** [08:32:57] : [Step 1/1] ================================================================= [08:32:57] : [Step 1/1] ==31825==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000478 (pc 0x7f71c2f026c0 bp 0x7ffd93c47d70 sp 0x7ffd93c47d48 T0) [08:32:57] : [Step 1/1] ==31825==The signal is caused by a READ memory access. [08:32:57] : [Step 1/1] ==31825==Hint: address points to the zero page. [08:32:57] : [Step 1/1] #0 0x7f71c2f026bf (/lib/x86_64-linux-gnu/libpthread.so.0+0xa6bf) [08:32:57] : [Step 1/1] #1 0x562bdca2a30f in __gthread_mutex_lock(pthread_mutex_t*) /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/x86_64-linux-gnu/c++/8/bits/gthr-default.h:748:12 [08:32:57] : [Step 1/1] #2 0x562bdca973e4 in std::mutex::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/std_mutex.h:103:17 [08:32:57] : [Step 1/1] #3 0x562bdca97343 in std::unique_lock<std::mutex>::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/std_mutex.h:267:17 [08:32:57] : [Step 1/1] #4 0x562bdca96df8 in UniqueLock<AnnotatedMixin<std::mutex>, std::unique_lock<std::mutex> >::Enter(char const*, char const*, int) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/./sync.h:124:19 [08:32:57] : [Step 1/1] #5 0x562bdca3c450 in UniqueLock<AnnotatedMixin<std::mutex>, std::unique_lock<std::mutex> >::UniqueLock(AnnotatedMixin<std::mutex>&, char const*, char const*, int, bool) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/./sync.h:146:13 [08:32:57] : [Step 1/1] #6 0x562bdd3f5e10 in AvalancheProcessor::stopEventLoop() /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/avalanche.cpp:325:5 [08:32:57] : [Step 1/1] #7 0x562bdc9ee9e3 in Shutdown(InitInterfaces&) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/init.cpp:200:18 [08:32:57] : [Step 1/1] #8 0x562bdc9dfe4a in AppInit(int, char**) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:214:5 [08:32:57] : [Step 1/1] #9 0x562bdc9de358 in main /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:229:13 [08:32:57] : [Step 1/1] #10 0x7f71c2d3009a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a) [08:32:57] : [Step 1/1] #11 0x562bdc904389 in _start (/home/teamcity/buildAgent/work/c4a5708f2bae7929/build/src/bitcoind+0x1b8389) [08:32:57] : [Step 1/1] [08:32:57] : [Step 1/1] AddressSanitizer can not provide additional info. [08:32:57] : [Step 1/1] SUMMARY: AddressSanitizer: SEGV (/lib/x86_64-linux-gnu/libpthread.so.0+0xa6bf) [08:32:57] : [Step 1/1] ==31825==ABORTING [08:32:57]W: [Step 1/1] Process exited with code 1 [08:32:57]E: [Step 1/1] Process exited with code 1 (Step: Command Line)
Each failure log is accessible here:
Bitcoin ABC functional tests: feature_logging.py
Bitcoin ABC functional tests: feature_uacomment.py
Bitcoin ABC functional tests: wallet_multiwallet.py
Bitcoin ABC functional tests: wallet_multiwallet.py --usecli
Comment Actions
Snippet of first build failure:
[08:53:09] : [Step 1/1] #7 AvalancheProcessor::stopEventLoop() /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/avalanche.cpp:325:5 (bitcoind+0x7fd215)
[08:53:09] : [Step 1/1] #8 Shutdown(InitInterfaces&) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/init.cpp:200:18 (bitcoind+0x12fef9)
[08:53:09] : [Step 1/1] #9 AppInit(int, char**) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:214:5 (bitcoind+0x122720)
[08:53:09] : [Step 1/1] #10 main /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:229:13 (bitcoind+0x120f6c)
[08:53:09] : [Step 1/1] #11 __libc_start_main <null> (libc.so.6+0x2409a)
[08:53:09] : [Step 1/1] #12 _start <null> (bitcoind+0x8f759)
[08:53:09] : [Step 1/1]
[08:53:09] : [Step 1/1] ThreadSanitizer can not provide additional info.
[08:53:09] : [Step 1/1] SUMMARY: ThreadSanitizer: SEGV (/lib/x86_64-linux-gnu/libpthread.so.0+0xa6bf)
[08:53:09]W: [Step 1/1] ++ for log in "${SAN_LOG_DIR}"/*.log.*
[08:53:09] : [Step 1/1] ==11440==ABORTING
[08:53:09]W: [Step 1/1] ++ echo '*** Output of /tmp/sanitizer_logs/ubsan.log.13289 ***'
[08:53:09] : [Step 1/1] *** Output of /tmp/sanitizer_logs/ubsan.log.13289 ***
[08:53:09]W: [Step 1/1] ++ cat /tmp/sanitizer_logs/ubsan.log.13289
[08:53:09] : [Step 1/1] ==13289==ERROR: ThreadSanitizer: SEGV on unknown address 0x000000000478 (pc 0x7ffa95c096c0 bp 0x7ffdc3bbfdc0 sp 0x7ffdc3bbfd48 T13289)
[08:53:09]W: [Step 1/1] ++ for log in "${SAN_LOG_DIR}"/*.log.*
[08:53:09] : [Step 1/1] ==13289==The signal is caused by a READ memory access.
[08:53:09]W: [Step 1/1] ++ echo '*** Output of /tmp/sanitizer_logs/ubsan.log.13806 ***'
[08:53:09] : [Step 1/1] ==13289==Hint: address points to the zero page.
[08:53:09]W: [Step 1/1] ++ cat /tmp/sanitizer_logs/ubsan.log.13806
[08:53:09] : [Step 1/1] #0 <null> <null> (libpthread.so.0+0xa6bf)
[08:53:09] : [Step 1/1] #1 pthread_mutex_lock <null> (bitcoind+0xb27f6)
[08:53:09] : [Step 1/1] #2 __gthread_mutex_lock(pthread_mutex_t*) /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/x86_64-linux-gnu/c++/8/bits/gthr-default.h:748:12 (bitcoind+0x15b5d3)
[08:53:09] : [Step 1/1] #3 std::mutex::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/std_mutex.h:103:17 (bitcoind+0x1a8a75)
[08:53:09] : [Step 1/1] #4 std::unique_lock<std::mutex>::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/std_mutex.h:267:17 (bitcoind+0x1a89de)
[08:53:09] : [Step 1/1] #5 UniqueLock<AnnotatedMixin<std::mutex>, std::unique_lock<std::mutex> >::Enter(char const*, char const*, int) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/./sync.h:124:19 (bitcoind+0x1a8557)
[08:53:09] : [Step 1/1] #6 UniqueLock<AnnotatedMixin<std::mutex>, std::unique_lock<std::mutex> >::UniqueLock(AnnotatedMixin<std::mutex>&, char const*, char const*, int, bool) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/./sync.h:146:13 (bitcoind+0x165fc1)
[08:53:09] : [Step 1/1] #7 AvalancheProcessor::stopEventLoop() /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/avalanche.cpp:325:5 (bitcoind+0x7fd215)
[08:53:09] : [Step 1/1] #8 Shutdown(InitInterfaces&) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/init.cpp:200:18 (bitcoind+0x12fef9)
[08:53:09] : [Step 1/1] #9 AppInit(int, char**) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:214:5 (bitcoind+0x122720)
[08:53:09] : [Step 1/1] #10 main /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:229:13 (bitcoind+0x120f6c)
[08:53:09] : [Step 1/1] #11 __libc_start_main <null> (libc.so.6+0x2409a)
[08:53:09] : [Step 1/1] #12 _start <null> (bitcoind+0x8f759)
[08:53:09] : [Step 1/1]
[08:53:09] : [Step 1/1] ThreadSanitizer can not provide additional info.
[08:53:09] : [Step 1/1] SUMMARY: ThreadSanitizer: SEGV (/lib/x86_64-linux-gnu/libpthread.so.0+0xa6bf)
[08:53:09] : [Step 1/1] ==13289==ABORTING
[08:53:09] : [Step 1/1] *** Output of /tmp/sanitizer_logs/ubsan.log.13806 ***
[08:53:09] : [Step 1/1] ==13806==ERROR: ThreadSanitizer: SEGV on unknown address 0x000000000478 (pc 0x7f07e87ee6c0 bp 0x7ffd48141500 sp 0x7ffd48141488 T13806)
[08:53:09] : [Step 1/1] ==13806==The signal is caused by a READ memory access.
[08:53:09] : [Step 1/1] ==13806==Hint: address points to the zero page.
[08:53:09] : [Step 1/1] #0 <null> <null> (libpthread.so.0+0xa6bf)
[08:53:09] : [Step 1/1] #1 pthread_mutex_lock <null> (bitcoind+0xb27f6)
[08:53:09] : [Step 1/1] #2 __gthread_mutex_lock(pthread_mutex_t*) /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/x86_64-linux-gnu/c++/8/bits/gthr-default.h:748:12 (bitcoind+0x15b5d3)
[08:53:09] : [Step 1/1] #3 std::mutex::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/std_mutex.h:103:17 (bitcoind+0x1a8a75)
[08:53:09] : [Step 1/1] #4 std::unique_lock<std::mutex>::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/std_mutex.h:267:17 (bitcoind+0x1a89de)
[08:53:09] : [Step 1/1] #5 UniqueLock<AnnotatedMixin<std::mutex>, std::unique_lock<std::mutex> >::Enter(char const*, char const*, int) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/./sync.h:124:19 (bitcoind+0x1a8557)
[08:53:09] : [Step 1/1] #6 UniqueLock<AnnotatedMixin<std::mutex>, std::unique_lock<std::mutex> >::UniqueLock(AnnotatedMixin<std::mutex>&, char const*, char const*, int, bool) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/./sync.h:146:13 (bitcoind+0x165fc1)
[08:53:09] : [Step 1/1] #7 AvalancheProcessor::stopEventLoop() /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/avalanche.cpp:325:5 (bitcoind+0x7fd215)
[08:53:09] : [Step 1/1] #8 Shutdown(InitInterfaces&) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/init.cpp:200:18 (bitcoind+0x12fef9)
[08:53:09] : [Step 1/1] #9 AppInit(int, char**) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:214:5 (bitcoind+0x122720)
[08:53:09] : [Step 1/1] #10 main /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:229:13 (bitcoind+0x120f6c)
[08:53:09] : [Step 1/1] #11 __libc_start_main <null> (libc.so.6+0x2409a)
[08:53:09] : [Step 1/1] #12 _start <null> (bitcoind+0x8f759)
[08:53:09] : [Step 1/1]
[08:53:09] : [Step 1/1] ThreadSanitizer can not provide additional info.
[08:53:09] : [Step 1/1] SUMMARY: ThreadSanitizer: SEGV (/lib/x86_64-linux-gnu/libpthread.so.0+0xa6bf)
[08:53:09] : [Step 1/1] ==13806==ABORTING
[08:53:09]W: [Step 1/1] Process exited with code 1
[08:53:09]E: [Step 1/1] Process exited with code 1 (Step: Command Line)Comment Actions
I have an update to the doc in D5395 to help running the sanitizers, please let me know if you have issues.
I started a run on CI and ASAN and TSAN are failing (apparently the same root cause).