Page MenuHomePhabricator

[avalanche] Start event loop at node startup
ClosedPublic

Authored by deadalnix on Mar 1 2020, 14:41.

Details

Summary

While this is not doign much, this will allows us to start boltin on it.

Test Plan

Ran the extended test suite.
Ran under ubsan.

I do not have a good way to run tsan and asan locally. Having a doc on how to do so and get them green would be beneficial.

Diff Detail

Repository
rABC Bitcoin ABC
Branch
avainstance
Lint
Lint Passed
Unit
No Test Coverage
Build Status
Buildable 9673
Build 17244: Default Diff Build & Tests
Build 17243: arc lint + arc unit

Event Timeline

Snippet of first build failure:

[08:32:57] :	 [Step 1/1]     #0 0x7f4dd8e1a6bf  (/lib/x86_64-linux-gnu/libpthread.so.0+0xa6bf)
[08:32:57] :	 [Step 1/1]     #1 0x562089e8630f in __gthread_mutex_lock(pthread_mutex_t*) /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/x86_64-linux-gnu/c++/8/bits/gthr-default.h:748:12
[08:32:57] :	 [Step 1/1]     #2 0x562089ef33e4 in std::mutex::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/std_mutex.h:103:17
[08:32:57] :	 [Step 1/1]     #3 0x562089ef3343 in std::unique_lock<std::mutex>::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/std_mutex.h:267:17
[08:32:57] :	 [Step 1/1]     #4 0x562089ef2df8 in UniqueLock<AnnotatedMixin<std::mutex>, std::unique_lock<std::mutex> >::Enter(char const*, char const*, int) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/./sync.h:124:19
[08:32:57] :	 [Step 1/1]     #5 0x562089e98450 in UniqueLock<AnnotatedMixin<std::mutex>, std::unique_lock<std::mutex> >::UniqueLock(AnnotatedMixin<std::mutex>&, char const*, char const*, int, bool) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/./sync.h:146:13
[08:32:57] :	 [Step 1/1]     #6 0x56208a851e10 in AvalancheProcessor::stopEventLoop() /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/avalanche.cpp:325:5
[08:32:57] :	 [Step 1/1]     #7 0x562089e4a9e3 in Shutdown(InitInterfaces&) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/init.cpp:200:18
[08:32:57] :	 [Step 1/1]     #8 0x562089e3be4a in AppInit(int, char**) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:214:5
[08:32:57] :	 [Step 1/1]     #9 0x562089e3a358 in main /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:229:13
[08:32:57] :	 [Step 1/1]     #10 0x7f4dd8c4809a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
[08:32:57] :	 [Step 1/1]     #11 0x562089d60389 in _start (/home/teamcity/buildAgent/work/c4a5708f2bae7929/build/src/bitcoind+0x1b8389)
[08:32:57] :	 [Step 1/1] 
[08:32:57] :	 [Step 1/1] AddressSanitizer can not provide additional info.
[08:32:57] :	 [Step 1/1] SUMMARY: AddressSanitizer: SEGV (/lib/x86_64-linux-gnu/libpthread.so.0+0xa6bf) 
[08:32:57] :	 [Step 1/1] ==29188==ABORTING
[08:32:57] :	 [Step 1/1] *** Output of /tmp/sanitizer_logs/ubsan.log.31363 ***
[08:32:57] :	 [Step 1/1] =================================================================
[08:32:57] :	 [Step 1/1] ==31363==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000478 (pc 0x7f4802b2a6c0 bp 0x7fff998866b0 sp 0x7fff99886688 T0)
[08:32:57] :	 [Step 1/1] ==31363==The signal is caused by a READ memory access.
[08:32:57] :	 [Step 1/1] ==31363==Hint: address points to the zero page.
[08:32:57] :	 [Step 1/1]     #0 0x7f4802b2a6bf  (/lib/x86_64-linux-gnu/libpthread.so.0+0xa6bf)
[08:32:57] :	 [Step 1/1]     #1 0x560f5a0ff30f in __gthread_mutex_lock(pthread_mutex_t*) /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/x86_64-linux-gnu/c++/8/bits/gthr-default.h:748:12
[08:32:57] :	 [Step 1/1]     #2 0x560f5a16c3e4 in std::mutex::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/std_mutex.h:103:17
[08:32:57] :	 [Step 1/1]     #3 0x560f5a16c343 in std::unique_lock<std::mutex>::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/std_mutex.h:267:17
[08:32:57] :	 [Step 1/1]     #4 0x560f5a16bdf8 in UniqueLock<AnnotatedMixin<std::mutex>, std::unique_lock<std::mutex> >::Enter(char const*, char const*, int) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/./sync.h:124:19
[08:32:57] :	 [Step 1/1]     #5 0x560f5a111450 in UniqueLock<AnnotatedMixin<std::mutex>, std::unique_lock<std::mutex> >::UniqueLock(AnnotatedMixin<std::mutex>&, char const*, char const*, int, bool) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/./sync.h:146:13
[08:32:57] :	 [Step 1/1]     #6 0x560f5aacae10 in AvalancheProcessor::stopEventLoop() /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/avalanche.cpp:325:5
[08:32:57] :	 [Step 1/1]     #7 0x560f5a0c39e3 in Shutdown(InitInterfaces&) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/init.cpp:200:18
[08:32:57] :	 [Step 1/1]     #8 0x560f5a0b4e4a in AppInit(int, char**) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:214:5
[08:32:57] :	 [Step 1/1]     #9 0x560f5a0b3358 in main /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:229:13
[08:32:57] :	 [Step 1/1]     #10 0x7f480295809a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
[08:32:57] :	 [Step 1/1]     #11 0x560f59fd9389 in _start (/home/teamcity/buildAgent/work/c4a5708f2bae7929/build/src/bitcoind+0x1b8389)
[08:32:57] :	 [Step 1/1] 
[08:32:57] :	 [Step 1/1] AddressSanitizer can not provide additional info.
[08:32:57] :	 [Step 1/1] SUMMARY: AddressSanitizer: SEGV (/lib/x86_64-linux-gnu/libpthread.so.0+0xa6bf) 
[08:32:57] :	 [Step 1/1] ==31363==ABORTING
[08:32:57] :	 [Step 1/1] *** Output of /tmp/sanitizer_logs/ubsan.log.31825 ***
[08:32:57] :	 [Step 1/1] =================================================================
[08:32:57] :	 [Step 1/1] ==31825==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000478 (pc 0x7f71c2f026c0 bp 0x7ffd93c47d70 sp 0x7ffd93c47d48 T0)
[08:32:57] :	 [Step 1/1] ==31825==The signal is caused by a READ memory access.
[08:32:57] :	 [Step 1/1] ==31825==Hint: address points to the zero page.
[08:32:57] :	 [Step 1/1]     #0 0x7f71c2f026bf  (/lib/x86_64-linux-gnu/libpthread.so.0+0xa6bf)
[08:32:57] :	 [Step 1/1]     #1 0x562bdca2a30f in __gthread_mutex_lock(pthread_mutex_t*) /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/x86_64-linux-gnu/c++/8/bits/gthr-default.h:748:12
[08:32:57] :	 [Step 1/1]     #2 0x562bdca973e4 in std::mutex::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/std_mutex.h:103:17
[08:32:57] :	 [Step 1/1]     #3 0x562bdca97343 in std::unique_lock<std::mutex>::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/std_mutex.h:267:17
[08:32:57] :	 [Step 1/1]     #4 0x562bdca96df8 in UniqueLock<AnnotatedMixin<std::mutex>, std::unique_lock<std::mutex> >::Enter(char const*, char const*, int) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/./sync.h:124:19
[08:32:57] :	 [Step 1/1]     #5 0x562bdca3c450 in UniqueLock<AnnotatedMixin<std::mutex>, std::unique_lock<std::mutex> >::UniqueLock(AnnotatedMixin<std::mutex>&, char const*, char const*, int, bool) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/./sync.h:146:13
[08:32:57] :	 [Step 1/1]     #6 0x562bdd3f5e10 in AvalancheProcessor::stopEventLoop() /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/avalanche.cpp:325:5
[08:32:57] :	 [Step 1/1]     #7 0x562bdc9ee9e3 in Shutdown(InitInterfaces&) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/init.cpp:200:18
[08:32:57] :	 [Step 1/1]     #8 0x562bdc9dfe4a in AppInit(int, char**) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:214:5
[08:32:57] :	 [Step 1/1]     #9 0x562bdc9de358 in main /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:229:13
[08:32:57] :	 [Step 1/1]     #10 0x7f71c2d3009a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
[08:32:57] :	 [Step 1/1]     #11 0x562bdc904389 in _start (/home/teamcity/buildAgent/work/c4a5708f2bae7929/build/src/bitcoind+0x1b8389)
[08:32:57] :	 [Step 1/1] 
[08:32:57] :	 [Step 1/1] AddressSanitizer can not provide additional info.
[08:32:57] :	 [Step 1/1] SUMMARY: AddressSanitizer: SEGV (/lib/x86_64-linux-gnu/libpthread.so.0+0xa6bf) 
[08:32:57] :	 [Step 1/1] ==31825==ABORTING
[08:32:57]W:	 [Step 1/1] Process exited with code 1
[08:32:57]E:	 [Step 1/1] Process exited with code 1 (Step: Command Line)

Each failure log is accessible here:
Bitcoin ABC functional tests: feature_logging.py
Bitcoin ABC functional tests: feature_uacomment.py
Bitcoin ABC functional tests: wallet_multiwallet.py
Bitcoin ABC functional tests: wallet_multiwallet.py --usecli

Snippet of first build failure:

[08:53:09] :	 [Step 1/1]     #7 AvalancheProcessor::stopEventLoop() /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/avalanche.cpp:325:5 (bitcoind+0x7fd215)
[08:53:09] :	 [Step 1/1]     #8 Shutdown(InitInterfaces&) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/init.cpp:200:18 (bitcoind+0x12fef9)
[08:53:09] :	 [Step 1/1]     #9 AppInit(int, char**) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:214:5 (bitcoind+0x122720)
[08:53:09] :	 [Step 1/1]     #10 main /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:229:13 (bitcoind+0x120f6c)
[08:53:09] :	 [Step 1/1]     #11 __libc_start_main <null> (libc.so.6+0x2409a)
[08:53:09] :	 [Step 1/1]     #12 _start <null> (bitcoind+0x8f759)
[08:53:09] :	 [Step 1/1] 
[08:53:09] :	 [Step 1/1] ThreadSanitizer can not provide additional info.
[08:53:09] :	 [Step 1/1] SUMMARY: ThreadSanitizer: SEGV (/lib/x86_64-linux-gnu/libpthread.so.0+0xa6bf) 
[08:53:09]W:	 [Step 1/1] ++ for log in "${SAN_LOG_DIR}"/*.log.*
[08:53:09] :	 [Step 1/1] ==11440==ABORTING
[08:53:09]W:	 [Step 1/1] ++ echo '*** Output of /tmp/sanitizer_logs/ubsan.log.13289 ***'
[08:53:09] :	 [Step 1/1] *** Output of /tmp/sanitizer_logs/ubsan.log.13289 ***
[08:53:09]W:	 [Step 1/1] ++ cat /tmp/sanitizer_logs/ubsan.log.13289
[08:53:09] :	 [Step 1/1] ==13289==ERROR: ThreadSanitizer: SEGV on unknown address 0x000000000478 (pc 0x7ffa95c096c0 bp 0x7ffdc3bbfdc0 sp 0x7ffdc3bbfd48 T13289)
[08:53:09]W:	 [Step 1/1] ++ for log in "${SAN_LOG_DIR}"/*.log.*
[08:53:09] :	 [Step 1/1] ==13289==The signal is caused by a READ memory access.
[08:53:09]W:	 [Step 1/1] ++ echo '*** Output of /tmp/sanitizer_logs/ubsan.log.13806 ***'
[08:53:09] :	 [Step 1/1] ==13289==Hint: address points to the zero page.
[08:53:09]W:	 [Step 1/1] ++ cat /tmp/sanitizer_logs/ubsan.log.13806
[08:53:09] :	 [Step 1/1]     #0 <null> <null> (libpthread.so.0+0xa6bf)
[08:53:09] :	 [Step 1/1]     #1 pthread_mutex_lock <null> (bitcoind+0xb27f6)
[08:53:09] :	 [Step 1/1]     #2 __gthread_mutex_lock(pthread_mutex_t*) /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/x86_64-linux-gnu/c++/8/bits/gthr-default.h:748:12 (bitcoind+0x15b5d3)
[08:53:09] :	 [Step 1/1]     #3 std::mutex::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/std_mutex.h:103:17 (bitcoind+0x1a8a75)
[08:53:09] :	 [Step 1/1]     #4 std::unique_lock<std::mutex>::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/std_mutex.h:267:17 (bitcoind+0x1a89de)
[08:53:09] :	 [Step 1/1]     #5 UniqueLock<AnnotatedMixin<std::mutex>, std::unique_lock<std::mutex> >::Enter(char const*, char const*, int) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/./sync.h:124:19 (bitcoind+0x1a8557)
[08:53:09] :	 [Step 1/1]     #6 UniqueLock<AnnotatedMixin<std::mutex>, std::unique_lock<std::mutex> >::UniqueLock(AnnotatedMixin<std::mutex>&, char const*, char const*, int, bool) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/./sync.h:146:13 (bitcoind+0x165fc1)
[08:53:09] :	 [Step 1/1]     #7 AvalancheProcessor::stopEventLoop() /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/avalanche.cpp:325:5 (bitcoind+0x7fd215)
[08:53:09] :	 [Step 1/1]     #8 Shutdown(InitInterfaces&) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/init.cpp:200:18 (bitcoind+0x12fef9)
[08:53:09] :	 [Step 1/1]     #9 AppInit(int, char**) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:214:5 (bitcoind+0x122720)
[08:53:09] :	 [Step 1/1]     #10 main /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:229:13 (bitcoind+0x120f6c)
[08:53:09] :	 [Step 1/1]     #11 __libc_start_main <null> (libc.so.6+0x2409a)
[08:53:09] :	 [Step 1/1]     #12 _start <null> (bitcoind+0x8f759)
[08:53:09] :	 [Step 1/1] 
[08:53:09] :	 [Step 1/1] ThreadSanitizer can not provide additional info.
[08:53:09] :	 [Step 1/1] SUMMARY: ThreadSanitizer: SEGV (/lib/x86_64-linux-gnu/libpthread.so.0+0xa6bf) 
[08:53:09] :	 [Step 1/1] ==13289==ABORTING
[08:53:09] :	 [Step 1/1] *** Output of /tmp/sanitizer_logs/ubsan.log.13806 ***
[08:53:09] :	 [Step 1/1] ==13806==ERROR: ThreadSanitizer: SEGV on unknown address 0x000000000478 (pc 0x7f07e87ee6c0 bp 0x7ffd48141500 sp 0x7ffd48141488 T13806)
[08:53:09] :	 [Step 1/1] ==13806==The signal is caused by a READ memory access.
[08:53:09] :	 [Step 1/1] ==13806==Hint: address points to the zero page.
[08:53:09] :	 [Step 1/1]     #0 <null> <null> (libpthread.so.0+0xa6bf)
[08:53:09] :	 [Step 1/1]     #1 pthread_mutex_lock <null> (bitcoind+0xb27f6)
[08:53:09] :	 [Step 1/1]     #2 __gthread_mutex_lock(pthread_mutex_t*) /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/x86_64-linux-gnu/c++/8/bits/gthr-default.h:748:12 (bitcoind+0x15b5d3)
[08:53:09] :	 [Step 1/1]     #3 std::mutex::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/std_mutex.h:103:17 (bitcoind+0x1a8a75)
[08:53:09] :	 [Step 1/1]     #4 std::unique_lock<std::mutex>::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/std_mutex.h:267:17 (bitcoind+0x1a89de)
[08:53:09] :	 [Step 1/1]     #5 UniqueLock<AnnotatedMixin<std::mutex>, std::unique_lock<std::mutex> >::Enter(char const*, char const*, int) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/./sync.h:124:19 (bitcoind+0x1a8557)
[08:53:09] :	 [Step 1/1]     #6 UniqueLock<AnnotatedMixin<std::mutex>, std::unique_lock<std::mutex> >::UniqueLock(AnnotatedMixin<std::mutex>&, char const*, char const*, int, bool) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/./sync.h:146:13 (bitcoind+0x165fc1)
[08:53:09] :	 [Step 1/1]     #7 AvalancheProcessor::stopEventLoop() /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/avalanche.cpp:325:5 (bitcoind+0x7fd215)
[08:53:09] :	 [Step 1/1]     #8 Shutdown(InitInterfaces&) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/init.cpp:200:18 (bitcoind+0x12fef9)
[08:53:09] :	 [Step 1/1]     #9 AppInit(int, char**) /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:214:5 (bitcoind+0x122720)
[08:53:09] :	 [Step 1/1]     #10 main /home/teamcity/buildAgent/work/c4a5708f2bae7929/build/../src/bitcoind.cpp:229:13 (bitcoind+0x120f6c)
[08:53:09] :	 [Step 1/1]     #11 __libc_start_main <null> (libc.so.6+0x2409a)
[08:53:09] :	 [Step 1/1]     #12 _start <null> (bitcoind+0x8f759)
[08:53:09] :	 [Step 1/1] 
[08:53:09] :	 [Step 1/1] ThreadSanitizer can not provide additional info.
[08:53:09] :	 [Step 1/1] SUMMARY: ThreadSanitizer: SEGV (/lib/x86_64-linux-gnu/libpthread.so.0+0xa6bf) 
[08:53:09] :	 [Step 1/1] ==13806==ABORTING
[08:53:09]W:	 [Step 1/1] Process exited with code 1
[08:53:09]E:	 [Step 1/1] Process exited with code 1 (Step: Command Line)
Fabien requested changes to this revision.Mar 2 2020, 16:21
Fabien added a subscriber: Fabien.

I have an update to the doc in D5395 to help running the sanitizers, please let me know if you have issues.

I started a run on CI and ASAN and TSAN are failing (apparently the same root cause).

This revision now requires changes to proceed.Mar 2 2020, 16:21

Fix Shutdown routine when it gets called before the node completely started up.

typos...

src/avalanche.cpp
26 ↗(On Diff #16699)

s/gloabl/global

src/init.cpp
195 ↗(On Diff #16699)

s/that/this

2106 ↗(On Diff #16699)

s/Setp/Setup

Please fix the typos before landing.

src/init.cpp
2106 ↗(On Diff #16699)

I think it's Step :)

This revision is now accepted and ready to land.Mar 3 2020, 10:25