Details

Reviewers

Group Reviewers

Restricted Project

Commits

rABCccd03b6cf337: add a wallet dependency to sqlite3

Summary

PR description:

Add sqlite as an alternative wallet database and use it for new descriptor wallets

This PR adds a new class SQLiteDatabase which is a subclass of WalletDatabase. This provides access to a SQLite database that is used to store the wallet records. To keep compatibility with BDB and to complexity of the change down, we don't make use of many SQLite's features. We use it strictly as a key-value store. We create a table main which has two columns, key and value both with the type blob.

For new descriptor wallets, we will create a SQLiteDatabase instead of a BerkeleyDatabase. There is no requirement that all SQLite wallets are descriptor wallets, nor is there a requirement that all descriptor wallets be SQLite wallets. This allows for existing descriptor wallets to work as well as keeping open the option to migrate existing wallets to SQLite.

We keep the name wallet.dat for SQLite wallets. We are able to determine which database type to use by searching for specific magic bytes in the wallet.dat file. SQLite begins it's files with a null terminated string SQLite format 3. BDB has 0x00053162 at byte 12 (note that the byte order of this integer depends on the system endianness). So when we see that there is a wallet.dat file that we want to open, we check for the magic bytes to determine which database system to use.
I decided to keep the wallet.dat naming to keep things like backup script to continue to function as they won't need to be modified to look for a different file name. It also simplifies a couple of things in the implementation and the tests as wallet.dat is something that is specifically being looked for. If we don't want this behavior, then I do have another branch which creates wallet.sqlite files instead, but I find that this direction is easier.

This is a backport of core#19077 [1 & 2 & 24/26]
https://github.com/bitcoin/bitcoin/pull/19077/commits/54729f3f4e6765dfded590af5fb28c88331685f8
https://github.com/bitcoin/bitcoin/pull/19077/commits/e87df8258090138d5c22ac46b8602b618620e8a1
https://github.com/bitcoin/bitcoin/pull/19077/commits/6c6639ac9f6e1677da066cf809f9e3fa4d2e7c32

The last commit is a partial backport. All the necessary documentation to build Bitcoin ABC after adding the new dependency are included, but not general documentation (files.md)

Test Plan

cmake .. -GNinja
ninja

Diff Detail

Repository

rABC Bitcoin ABC

Lint

Lint Passed

Unit

No Test Coverage

Build Status

Buildable 17268
Build 34364: Build Diff	build-debug · lint-circular-dependencies · build-without-wallet · build-diff · build-clang-tidy · build-clang
Build 34363: arc lint + arc unit

Event Timeline

PiRK created this revision.Nov 12 2021, 15:58

Herald added a reviewer: Restricted Project. · View Herald TranscriptNov 12 2021, 15:58

teamcity edited the summary of this revision. (Show Details)Nov 12 2021, 15:59

@bot build-osx build-win64 build-linux-arm build-linux-aarch64

teamcity awarded a token.Nov 12 2021, 16:08

Build Bitcoin ABC Diffs / Diff Testing (build-linux-aarch64) failed.

Tail of the build log:

-- Performing Test have_C__std_c89
-- Performing Test have_C__std_c89 - Success
-- Performing Test have_C__Wno_long_long
-- Performing Test have_C__Wno_long_long - Success
-- Performing Test have_C__Wno_duplicated_branches
-- Performing Test have_C__Wno_duplicated_branches - Success
-- Found OpenSSL: /work/depends/aarch64-linux-gnu/lib/libcrypto.a (found version "1.0.1k") found components: Crypto 
-- Found Event component event: /work/depends/aarch64-linux-gnu/lib/libevent.a
-- Found Event: /work/depends/aarch64-linux-gnu/include (found suitable version "99.99.99", minimum required is "2.0.22") found components: event 
-- Found Boost 1.70.0 at /work/depends/aarch64-linux-gnu/lib/cmake/Boost-1.70.0
--   Requested configuration: QUIET REQUIRED COMPONENTS filesystem
-- Found boost_headers 1.70.0 at /work/depends/aarch64-linux-gnu/lib/cmake/boost_headers-1.70.0
-- Found boost_filesystem 1.70.0 at /work/depends/aarch64-linux-gnu/lib/cmake/boost_filesystem-1.70.0
--   libboost_filesystem-mt-a64.a
-- Adding boost_filesystem dependencies: headers
-- Found Boost: /work/depends/aarch64-linux-gnu/lib/cmake/Boost-1.70.0/BoostConfig.cmake (found suitable version "1.70.0", minimum required is "1.59") found components: filesystem 
-- Found Boost 1.70.0 at /work/depends/aarch64-linux-gnu/lib/cmake/Boost-1.70.0
--   Requested configuration: QUIET REQUIRED COMPONENTS thread
-- Found boost_thread 1.70.0 at /work/depends/aarch64-linux-gnu/lib/cmake/boost_thread-1.70.0
--   libboost_thread-mt-a64.a
-- Adding boost_thread dependencies: headers
-- Found Boost: /work/depends/aarch64-linux-gnu/lib/cmake/Boost-1.70.0/BoostConfig.cmake (found suitable version "1.70.0", minimum required is "1.59") found components: thread 
-- Found Event component pthreads: /work/depends/aarch64-linux-gnu/lib/libevent_pthreads.a
-- Found Event: /work/depends/aarch64-linux-gnu/include (found suitable version "99.99.99", minimum required is "2.0.22") found components: pthreads 
-- Found MiniUPnPc component miniupnpc: /work/depends/aarch64-linux-gnu/lib/libminiupnpc.a
-- Found MiniUPnPc: /work/depends/aarch64-linux-gnu/include/miniupnpc (found suitable version "2.0.20180203", minimum required is "1.9")  
-- Found Boost 1.70.0 at /work/depends/aarch64-linux-gnu/lib/cmake/Boost-1.70.0
--   Requested configuration: QUIET REQUIRED COMPONENTS unit_test_framework
-- Found boost_unit_test_framework 1.70.0 at /work/depends/aarch64-linux-gnu/lib/cmake/boost_unit_test_framework-1.70.0
--   libboost_unit_test_framework-mt-a64.a
-- Adding boost_unit_test_framework dependencies: headers
-- Found Boost: /work/depends/aarch64-linux-gnu/lib/cmake/Boost-1.70.0/BoostConfig.cmake (found suitable version "1.70.0", minimum required is "1.59") found components: unit_test_framework 
-- Performing Test BOOST_REQUIRES_TEST_DYN_LINK
-- Performing Test BOOST_REQUIRES_TEST_DYN_LINK - Failed
-- Found Boost 1.70.0 at /work/depends/aarch64-linux-gnu/lib/cmake/Boost-1.70.0
--   Requested configuration: QUIET REQUIRED COMPONENTS unit_test_framework
-- Found boost_unit_test_framework 1.70.0 at /work/depends/aarch64-linux-gnu/lib/cmake/boost_unit_test_framework-1.70.0
--   libboost_unit_test_framework-mt-a64.a
-- Adding boost_unit_test_framework dependencies: headers
-- Found Boost 1.70.0 at /work/depends/aarch64-linux-gnu/lib/cmake/Boost-1.70.0
--   Requested configuration: QUIET REQUIRED COMPONENTS unit_test_framework
-- Found boost_unit_test_framework 1.70.0 at /work/depends/aarch64-linux-gnu/lib/cmake/boost_unit_test_framework-1.70.0
--   libboost_unit_test_framework-mt-a64.a
-- Adding boost_unit_test_framework dependencies: headers
CMake Error at /usr/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:165 (message):
  Could NOT find BerkeleyDB (missing: BerkeleyDB_INCLUDE_DIR CXX) (Required
  is at least version "5.3")

      Reason given by package: if you don't want to build the wallet feature, this can be skipped by passing -DBUILD_BITCOIN_WALLET=OFF to the cmake command line

Call Stack (most recent call first):
  /usr/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:458 (_FPHSA_FAILURE_MESSAGE)
  cmake/modules/FindBerkeleyDB.cmake:162 (find_package_handle_standard_args)
  src/wallet/CMakeLists.txt:6 (find_package)


-- Configuring incomplete, errors occurred!
See also "/work/abc-ci-builds/build-linux-aarch64/CMakeFiles/CMakeOutput.log".
See also "/work/abc-ci-builds/build-linux-aarch64/CMakeFiles/CMakeError.log".
Build build-linux-aarch64 failed with exit code 1

Build Bitcoin ABC Diffs / Diff Testing (build-osx) failed.

Tail of the build log:

  }

  #endif

  

  __BEGIN_DECLS

  DB *dbopen(const char *, int, int, DBTYPE, const void *);

  

  #ifdef __DBINTERFACE_PRIVATE

  DB	*__bt_open(const char *, int, int, const BTREEINFO *, int);

  DB	*__hash_open(const char *, int, int, const HASHINFO *, int);

  DB	*__rec_open(const char *, int, int, const RECNOINFO *, int);

  void	 __dbpanic(DB *dbp);

  #endif

  __END_DECLS

  #endif /* !_DB_H_ */

  ", but required is at least "5.3" (found
  /work/depends/SDKs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers/usr/include,
  )

      Reason given by package: if you don't want to build the wallet feature, this can be skipped by passing -DBUILD_BITCOIN_WALLET=OFF to the cmake command line

Call Stack (most recent call first):
  /usr/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:456 (_FPHSA_FAILURE_MESSAGE)
  cmake/modules/FindBerkeleyDB.cmake:162 (find_package_handle_standard_args)
  src/wallet/CMakeLists.txt:6 (find_package)


-- Configuring incomplete, errors occurred!
See also "/work/abc-ci-builds/build-osx/CMakeFiles/CMakeOutput.log".
See also "/work/abc-ci-builds/build-osx/CMakeFiles/CMakeError.log".
CMake Warning:
  Value of BerkeleyDB_VERSION contained a newline; truncating


CMake Warning:
  Value of BerkeleyDB_VERSION_MAJOR contained a newline; truncating


CMake Warning:
  Value of BerkeleyDB_VERSION_MINOR contained a newline; truncating


CMake Warning:
  Value of BerkeleyDB_VERSION_PATCH contained a newline; truncating


Build build-osx failed with exit code 1

Build Bitcoin ABC Diffs / Diff Testing (build-win64) failed.

Tail of the build log:

-- Performing Test have_C__Wno_duplicated_branches
-- Performing Test have_C__Wno_duplicated_branches - Success
-- Performing Test USE_ASM_X86_64
-- Performing Test USE_ASM_X86_64 - Success
-- Found OpenSSL: /work/depends/x86_64-w64-mingw32/lib/libcrypto.a (found version "1.0.1k") found components: Crypto 
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29") 
-- Found Event component event: /work/depends/x86_64-w64-mingw32/lib/libevent.a
-- Found Event: /work/depends/x86_64-w64-mingw32/include (found suitable version "99.99.99", minimum required is "2.0.22") found components: event 
-- Found Boost 1.70.0 at /work/depends/x86_64-w64-mingw32/lib/cmake/Boost-1.70.0
--   Requested configuration: QUIET REQUIRED COMPONENTS filesystem
-- Found boost_headers 1.70.0 at /work/depends/x86_64-w64-mingw32/lib/cmake/boost_headers-1.70.0
-- Found boost_filesystem 1.70.0 at /work/depends/x86_64-w64-mingw32/lib/cmake/boost_filesystem-1.70.0
--   libboost_filesystem-mt-s-x64.a
-- Adding boost_filesystem dependencies: headers
-- Found Boost: /work/depends/x86_64-w64-mingw32/lib/cmake/Boost-1.70.0/BoostConfig.cmake (found suitable version "1.70.0", minimum required is "1.59") found components: filesystem 
-- Found Boost 1.70.0 at /work/depends/x86_64-w64-mingw32/lib/cmake/Boost-1.70.0
--   Requested configuration: QUIET REQUIRED COMPONENTS thread
-- Found boost_thread 1.70.0 at /work/depends/x86_64-w64-mingw32/lib/cmake/boost_thread-1.70.0
--   libboost_thread-mt-s-x64.a
-- Adding boost_thread dependencies: chrono;headers
-- Found boost_chrono 1.70.0 at /work/depends/x86_64-w64-mingw32/lib/cmake/boost_chrono-1.70.0
--   libboost_chrono-mt-s-x64.a
-- Adding boost_chrono dependencies: headers
-- Found Boost: /work/depends/x86_64-w64-mingw32/lib/cmake/Boost-1.70.0/BoostConfig.cmake (found suitable version "1.70.0", minimum required is "1.59") found components: thread 
-- Found MiniUPnPc component miniupnpc: /work/depends/x86_64-w64-mingw32/lib/libminiupnpc.a
-- Found MiniUPnPc: /work/depends/x86_64-w64-mingw32/include/miniupnpc (found suitable version "2.0.20180203", minimum required is "1.9")  
-- Found Boost 1.70.0 at /work/depends/x86_64-w64-mingw32/lib/cmake/Boost-1.70.0
--   Requested configuration: QUIET REQUIRED COMPONENTS unit_test_framework
-- Found boost_unit_test_framework 1.70.0 at /work/depends/x86_64-w64-mingw32/lib/cmake/boost_unit_test_framework-1.70.0
--   libboost_unit_test_framework-mt-s-x64.a
-- Adding boost_unit_test_framework dependencies: headers
-- Found Boost: /work/depends/x86_64-w64-mingw32/lib/cmake/Boost-1.70.0/BoostConfig.cmake (found suitable version "1.70.0", minimum required is "1.59") found components: unit_test_framework 
-- Performing Test BOOST_REQUIRES_TEST_DYN_LINK
-- Performing Test BOOST_REQUIRES_TEST_DYN_LINK - Failed
-- Found Boost 1.70.0 at /work/depends/x86_64-w64-mingw32/lib/cmake/Boost-1.70.0
--   Requested configuration: QUIET REQUIRED COMPONENTS unit_test_framework
-- Found boost_unit_test_framework 1.70.0 at /work/depends/x86_64-w64-mingw32/lib/cmake/boost_unit_test_framework-1.70.0
--   libboost_unit_test_framework-mt-s-x64.a
-- Adding boost_unit_test_framework dependencies: headers
-- Found Boost 1.70.0 at /work/depends/x86_64-w64-mingw32/lib/cmake/Boost-1.70.0
--   Requested configuration: QUIET REQUIRED COMPONENTS unit_test_framework
-- Found boost_unit_test_framework 1.70.0 at /work/depends/x86_64-w64-mingw32/lib/cmake/boost_unit_test_framework-1.70.0
--   libboost_unit_test_framework-mt-s-x64.a
-- Adding boost_unit_test_framework dependencies: headers
CMake Error at /usr/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:165 (message):
  Could NOT find BerkeleyDB (missing: BerkeleyDB_INCLUDE_DIR CXX) (Required
  is at least version "5.3")

      Reason given by package: if you don't want to build the wallet feature, this can be skipped by passing -DBUILD_BITCOIN_WALLET=OFF to the cmake command line

Call Stack (most recent call first):
  /usr/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:458 (_FPHSA_FAILURE_MESSAGE)
  cmake/modules/FindBerkeleyDB.cmake:162 (find_package_handle_standard_args)
  src/wallet/CMakeLists.txt:6 (find_package)


-- Configuring incomplete, errors occurred!
See also "/work/abc-ci-builds/build-win64/CMakeFiles/CMakeOutput.log".
See also "/work/abc-ci-builds/build-win64/CMakeFiles/CMakeError.log".
Build build-win64 failed with exit code 1

Build Bitcoin ABC Diffs / Diff Testing (build-linux-arm) failed.

Tail of the build log:

-- Performing Test have_C__Wno_long_long
-- Performing Test have_C__Wno_long_long - Success
-- Performing Test have_C__Wno_duplicated_branches
-- Performing Test have_C__Wno_duplicated_branches - Success
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/arm-linux-gnueabihf-gcc
-- Found OpenSSL: /work/depends/arm-linux-gnueabihf/lib/libcrypto.a (found version "1.0.1k") found components: Crypto 
-- Found Event component event: /work/depends/arm-linux-gnueabihf/lib/libevent.a
-- Found Event: /work/depends/arm-linux-gnueabihf/include (found suitable version "99.99.99", minimum required is "2.0.22") found components: event 
-- Found Boost 1.70.0 at /work/depends/arm-linux-gnueabihf/lib/cmake/Boost-1.70.0
--   Requested configuration: QUIET REQUIRED COMPONENTS filesystem
-- Found boost_headers 1.70.0 at /work/depends/arm-linux-gnueabihf/lib/cmake/boost_headers-1.70.0
-- Found boost_filesystem 1.70.0 at /work/depends/arm-linux-gnueabihf/lib/cmake/boost_filesystem-1.70.0
--   libboost_filesystem-mt-a32.a
-- Adding boost_filesystem dependencies: headers
-- Found Boost: /work/depends/arm-linux-gnueabihf/lib/cmake/Boost-1.70.0/BoostConfig.cmake (found suitable version "1.70.0", minimum required is "1.59") found components: filesystem 
-- Found Boost 1.70.0 at /work/depends/arm-linux-gnueabihf/lib/cmake/Boost-1.70.0
--   Requested configuration: QUIET REQUIRED COMPONENTS thread
-- Found boost_thread 1.70.0 at /work/depends/arm-linux-gnueabihf/lib/cmake/boost_thread-1.70.0
--   libboost_thread-mt-a32.a
-- Adding boost_thread dependencies: headers
-- Found Boost: /work/depends/arm-linux-gnueabihf/lib/cmake/Boost-1.70.0/BoostConfig.cmake (found suitable version "1.70.0", minimum required is "1.59") found components: thread 
-- Found Event component pthreads: /work/depends/arm-linux-gnueabihf/lib/libevent_pthreads.a
-- Found Event: /work/depends/arm-linux-gnueabihf/include (found suitable version "99.99.99", minimum required is "2.0.22") found components: pthreads 
-- Found MiniUPnPc component miniupnpc: /work/depends/arm-linux-gnueabihf/lib/libminiupnpc.a
-- Found MiniUPnPc: /work/depends/arm-linux-gnueabihf/include/miniupnpc (found suitable version "2.0.20180203", minimum required is "1.9")  
-- Found Boost 1.70.0 at /work/depends/arm-linux-gnueabihf/lib/cmake/Boost-1.70.0
--   Requested configuration: QUIET REQUIRED COMPONENTS unit_test_framework
-- Found boost_unit_test_framework 1.70.0 at /work/depends/arm-linux-gnueabihf/lib/cmake/boost_unit_test_framework-1.70.0
--   libboost_unit_test_framework-mt-a32.a
-- Adding boost_unit_test_framework dependencies: headers
-- Found Boost: /work/depends/arm-linux-gnueabihf/lib/cmake/Boost-1.70.0/BoostConfig.cmake (found suitable version "1.70.0", minimum required is "1.59") found components: unit_test_framework 
-- Performing Test BOOST_REQUIRES_TEST_DYN_LINK
-- Performing Test BOOST_REQUIRES_TEST_DYN_LINK - Failed
-- Found Boost 1.70.0 at /work/depends/arm-linux-gnueabihf/lib/cmake/Boost-1.70.0
--   Requested configuration: QUIET REQUIRED COMPONENTS unit_test_framework
-- Found boost_unit_test_framework 1.70.0 at /work/depends/arm-linux-gnueabihf/lib/cmake/boost_unit_test_framework-1.70.0
--   libboost_unit_test_framework-mt-a32.a
-- Adding boost_unit_test_framework dependencies: headers
-- Found Boost 1.70.0 at /work/depends/arm-linux-gnueabihf/lib/cmake/Boost-1.70.0
--   Requested configuration: QUIET REQUIRED COMPONENTS unit_test_framework
-- Found boost_unit_test_framework 1.70.0 at /work/depends/arm-linux-gnueabihf/lib/cmake/boost_unit_test_framework-1.70.0
--   libboost_unit_test_framework-mt-a32.a
-- Adding boost_unit_test_framework dependencies: headers
CMake Error at /usr/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:165 (message):
  Could NOT find BerkeleyDB (missing: BerkeleyDB_INCLUDE_DIR CXX) (Required
  is at least version "5.3")

      Reason given by package: if you don't want to build the wallet feature, this can be skipped by passing -DBUILD_BITCOIN_WALLET=OFF to the cmake command line

Call Stack (most recent call first):
  /usr/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:458 (_FPHSA_FAILURE_MESSAGE)
  cmake/modules/FindBerkeleyDB.cmake:162 (find_package_handle_standard_args)
  src/wallet/CMakeLists.txt:6 (find_package)


-- Configuring incomplete, errors occurred!
See also "/work/abc-ci-builds/build-linux-arm/CMakeFiles/CMakeOutput.log".
See also "/work/abc-ci-builds/build-linux-arm/CMakeFiles/CMakeError.log".
Build build-linux-arm failed with exit code 1

Harbormaster completed remote builds in B17247: Diff 30859.Nov 12 2021, 16:34

deduplicate find_package(BerkeleyDB)

@bot build-linux-aarch64

Build Bitcoin ABC Diffs / Diff Testing (build-linux-aarch64) failed.

Tail of the build log:

-- Performing Test have_C__std_c89
-- Performing Test have_C__std_c89 - Success
-- Performing Test have_C__Wno_long_long
-- Performing Test have_C__Wno_long_long - Success
-- Performing Test have_C__Wno_duplicated_branches
-- Performing Test have_C__Wno_duplicated_branches - Success
-- Found OpenSSL: /work/depends/aarch64-linux-gnu/lib/libcrypto.a (found version "1.0.1k") found components: Crypto 
-- Found Event component event: /work/depends/aarch64-linux-gnu/lib/libevent.a
-- Found Event: /work/depends/aarch64-linux-gnu/include (found suitable version "99.99.99", minimum required is "2.0.22") found components: event 
-- Found Boost 1.70.0 at /work/depends/aarch64-linux-gnu/lib/cmake/Boost-1.70.0
--   Requested configuration: QUIET REQUIRED COMPONENTS filesystem
-- Found boost_headers 1.70.0 at /work/depends/aarch64-linux-gnu/lib/cmake/boost_headers-1.70.0
-- Found boost_filesystem 1.70.0 at /work/depends/aarch64-linux-gnu/lib/cmake/boost_filesystem-1.70.0
--   libboost_filesystem-mt-a64.a
-- Adding boost_filesystem dependencies: headers
-- Found Boost: /work/depends/aarch64-linux-gnu/lib/cmake/Boost-1.70.0/BoostConfig.cmake (found suitable version "1.70.0", minimum required is "1.59") found components: filesystem 
-- Found Boost 1.70.0 at /work/depends/aarch64-linux-gnu/lib/cmake/Boost-1.70.0
--   Requested configuration: QUIET REQUIRED COMPONENTS thread
-- Found boost_thread 1.70.0 at /work/depends/aarch64-linux-gnu/lib/cmake/boost_thread-1.70.0
--   libboost_thread-mt-a64.a
-- Adding boost_thread dependencies: headers
-- Found Boost: /work/depends/aarch64-linux-gnu/lib/cmake/Boost-1.70.0/BoostConfig.cmake (found suitable version "1.70.0", minimum required is "1.59") found components: thread 
-- Found Event component pthreads: /work/depends/aarch64-linux-gnu/lib/libevent_pthreads.a
-- Found Event: /work/depends/aarch64-linux-gnu/include (found suitable version "99.99.99", minimum required is "2.0.22") found components: pthreads 
-- Found MiniUPnPc component miniupnpc: /work/depends/aarch64-linux-gnu/lib/libminiupnpc.a
-- Found MiniUPnPc: /work/depends/aarch64-linux-gnu/include/miniupnpc (found suitable version "2.0.20180203", minimum required is "1.9")  
-- Found Boost 1.70.0 at /work/depends/aarch64-linux-gnu/lib/cmake/Boost-1.70.0
--   Requested configuration: QUIET REQUIRED COMPONENTS unit_test_framework
-- Found boost_unit_test_framework 1.70.0 at /work/depends/aarch64-linux-gnu/lib/cmake/boost_unit_test_framework-1.70.0
--   libboost_unit_test_framework-mt-a64.a
-- Adding boost_unit_test_framework dependencies: headers
-- Found Boost: /work/depends/aarch64-linux-gnu/lib/cmake/Boost-1.70.0/BoostConfig.cmake (found suitable version "1.70.0", minimum required is "1.59") found components: unit_test_framework 
-- Performing Test BOOST_REQUIRES_TEST_DYN_LINK
-- Performing Test BOOST_REQUIRES_TEST_DYN_LINK - Failed
-- Found Boost 1.70.0 at /work/depends/aarch64-linux-gnu/lib/cmake/Boost-1.70.0
--   Requested configuration: QUIET REQUIRED COMPONENTS unit_test_framework
-- Found boost_unit_test_framework 1.70.0 at /work/depends/aarch64-linux-gnu/lib/cmake/boost_unit_test_framework-1.70.0
--   libboost_unit_test_framework-mt-a64.a
-- Adding boost_unit_test_framework dependencies: headers
-- Found Boost 1.70.0 at /work/depends/aarch64-linux-gnu/lib/cmake/Boost-1.70.0
--   Requested configuration: QUIET REQUIRED COMPONENTS unit_test_framework
-- Found boost_unit_test_framework 1.70.0 at /work/depends/aarch64-linux-gnu/lib/cmake/boost_unit_test_framework-1.70.0
--   libboost_unit_test_framework-mt-a64.a
-- Adding boost_unit_test_framework dependencies: headers
CMake Error at /usr/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:165 (message):
  Could NOT find BerkeleyDB (missing: BerkeleyDB_INCLUDE_DIR CXX) (Required
  is at least version "5.3")

      Reason given by package: if you don't want to build the wallet feature, this can be skipped by passing -DBUILD_BITCOIN_WALLET=OFF to the cmake command line

Call Stack (most recent call first):
  /usr/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:458 (_FPHSA_FAILURE_MESSAGE)
  cmake/modules/FindBerkeleyDB.cmake:162 (find_package_handle_standard_args)
  src/wallet/CMakeLists.txt:6 (find_package)


-- Configuring incomplete, errors occurred!
See also "/work/abc-ci-builds/build-linux-aarch64/CMakeFiles/CMakeOutput.log".
See also "/work/abc-ci-builds/build-linux-aarch64/CMakeFiles/CMakeError.log".
Build build-linux-aarch64 failed with exit code 1

Harbormaster completed remote builds in B17249: Diff 30861.Nov 12 2021, 17:05

add missing changes in depends/Makefile, and enable parralel build in sqlite.mk

@bot build-osx build-win64 build-linux-arm build-linux-aarch64

teamcity updated a token.Nov 12 2021, 21:23

Harbormaster completed remote builds in B17250: Diff 30862.Nov 12 2021, 21:33

add missing newline at end of file
add sqlite3 to target_link_libraries

Harbormaster completed remote builds in B17262: Diff 30879.Nov 16 2021, 16:13

include some documentation changes from commit 24/26 (the documentation that is necessary as of this commit for building Bitcoin ABC) and update the description accordingly

Harbormaster completed remote builds in B17265: Diff 30886.Nov 16 2021, 17:45

final rebase before submitting diff to review

PiRK published this revision for review.Nov 17 2021, 12:18

Harbormaster completed remote builds in B17268: Diff 30890.Nov 17 2021, 12:43

Fabien requested changes to this revision.Nov 17 2021, 15:09

Fabien added a subscriber: Fabien.

Fabien added inline comments.

doc/build-unix.md
107	You need one or the other, but not necessarily both
src/wallet/CMakeLists.txt
9	There is no such component
37	This is wrong and I'm a bit surprised this worked at all. Your find_package call above defines an imported target SQLite::SQLite3 which will bring the lib path and add the include dirs as needed. What you did just tell the linker to add a -lsqlite3 flag to the command line.

This revision now requires changes to proceed.Nov 17 2021, 15:09

PiRK added inline comments.Nov 17 2021, 16:58

doc/build-unix.md
107	At this point, I think both are needed. Legacy wallets keep using BDB, and Descriptor wallets will use sqlite3. I don't think there is a plan to add a switch to use only one of the two.
src/wallet/CMakeLists.txt
9	OK. Removing the last two words does not seem to break anything.
37	Ok, so this is something I don't understand. I initially didn't add sqlite3 here, and it all worked until the sqlite code started being used outside of sqlite.cpp. Then it broke at commit 18/26. The errors looked like this: `ld.lld: error: undefined symbol: sqlite3_libversion` Adding sqlite3 here fixed the errors. But I have 0 understanding of what linking is about, so I don't know why this works.

PiRK added inline comments.Nov 17 2021, 17:02

src/wallet/CMakeLists.txt
9	I can even remove the line entirely, as long as I leave `sqlite3` in the `target_link_libraries` call.

fix CMakeLists

PiRK added a child revision: D10542: Add SQLiteDatabase and SQLiteBatch dummy classes.Nov 25 2021, 12:35

Harbormaster completed remote builds in B17360: Diff 31018.Nov 25 2021, 13:03

Fabien accepted this revision.Nov 26 2021, 10:45

This revision is now accepted and ready to land.Nov 26 2021, 10:45

Closed by commit rABCccd03b6cf337: add a wallet dependency to sqlite3 (authored by PiRK). · Explain WhyNov 26 2021, 14:06

This revision was automatically updated to reflect the committed changes.

PiRK added a commit: rABCccd03b6cf337: add a wallet dependency to sqlite3.

@bot build-win64

teamcity updated a token.Nov 27 2021, 09:34

@bot build-gitian-win

Build Bitcoin ABC Diffs / Diff Testing (build-win64) failed.

Tail of the build log:

[778/784] Linking CXX executable src/test/test_bitcoin.exe
[779/784] Installing component bitcoin-test_bitcoin
-- Install configuration: "RelWithDebInfo"
-- Install component: "bitcoin-test_bitcoin"
-- Installing: /results/artifacts/bin/test_bitcoin.exe
[780/784] Installing component bitcoin-upgrade-activated-test_bitcoin
-- Install configuration: "RelWithDebInfo"
-- Install component: "bitcoin-upgrade-activated-test_bitcoin"
-- Installing: /results/artifacts/bin/test_bitcoin.exe
[781/784] Linking CXX executable src/qt/bitcoin-qt.exe
[781/784] Install the project...
-- Install configuration: "RelWithDebInfo"
-- Installing: /results/artifacts/lib/libbitcoinconsensus.dll.a
-- Installing: /results/artifacts/bin/libbitcoinconsensus.dll
-- Installing: /results/artifacts/include/bitcoinconsensus.h
-- Installing: /results/artifacts/bin/bitcoin-wallet.exe
-- Installing: /results/artifacts/bin/bitcoin-cli.exe
-- Installing: /results/artifacts/bin/bitcoin-tx.exe
-- Installing: /results/artifacts/bin/bitcoind.exe
-- Installing: /results/artifacts/bin/bitcoin-qt.exe
[783/784] Installing component bitcoin-qt-test_bitcoin-qt
-- Install configuration: "RelWithDebInfo"
-- Install component: "bitcoin-qt-test_bitcoin-qt"
-- Installing: /results/artifacts/bin/test_bitcoin-qt.exe
[1/9] Automatic MOC for target bitcoin-qt-protobuf
[2/8] Automatic MOC for target bitcoin-qt-base
[3/6] Automatic MOC for target bitcoin-qt
[3/4] Run CPack packaging tool...
CPack: Create package using NSIS
CPack: Install projects
CPack: - Install directory: /work/doc
CPack: - Install project: bitcoin-abc []
CPack: Create package
CPack: - package: /work/abc-ci-builds/build-win64/bitcoin-abc-0.24.7-x86_64-w64-mingw32.exe generated.
CPack: Create package using ZIP
CPack: Install projects
CPack: - Install directory: /work/doc
CPack: - Install project: bitcoin-abc []
CPack: Create package
CPack: - package: /work/abc-ci-builds/build-win64/bitcoin-abc-0.24.7-x86_64-w64-mingw32.zip generated.
wine: created the configuration directory '/root/.wine'
0009:err:file:init_redirects cannot open L"C:\\windows" (c000000f)
0012:err:ole:marshal_object couldn't get IPSFactory buffer for interface {00000131-0000-0000-c000-000000000046}
0012:err:ole:marshal_object couldn't get IPSFactory buffer for interface {6d5140c1-7436-11ce-8034-00aa006009fa}
0012:err:ole:StdMarshalImpl_MarshalInterface Failed to create ifstub, hres=0x80004002
0012:err:ole:CoMarshalInterface Failed to marshal the interface {6d5140c1-7436-11ce-8034-00aa006009fa}, 80004002
0012:err:ole:get_local_server_stream Failed: 80004002
000b:err:winediag:nodrv_CreateWindow Application tried to create a window, but no driver could be loaded.
000b:err:winediag:nodrv_CreateWindow Make sure that your X server is running and that $DISPLAY is set correctly.
0010:err:winediag:nodrv_CreateWindow Application tried to create a window, but no driver could be loaded.
0010:err:winediag:nodrv_CreateWindow Make sure that your X server is running and that $DISPLAY is set correctly.
Could not load wine-gecko. HTML rendering will be disabled.
0024:err:winediag:nodrv_CreateWindow Application tried to create a window, but no driver could be loaded.
0024:err:winediag:nodrv_CreateWindow Make sure that your X server is running and that $DISPLAY is set correctly.
Could not load wine-gecko. HTML rendering will be disabled.
wine: configuration in '/root/.wine' has been updated.
Running 524 test cases...

*** No errors detected
Build build-win64 completed successfully

@bot gitian-win

teamcity updated a token.Nov 27 2021, 09:47

Build Bitcoin ABC Diffs / Diff Testing (gitian-win) failed.

Tail of the build log:

popd

RESULTS_DIR=$(pwd)/results

echo "Building base image for: ${HASH}..."

# Note: ERROR_PREFIX needs to be broken up here because this script gets copied into the build log.
# Not doing this results in all errors being treated as infra failures.
ERROR_PREFIX=$(echo "[Infrastructure" "Error]")
export ERROR_PREFIX

~/infra/ci-wrappers/build-abc-base-image.sh -c ./bitcoin-abc \
  --cachetagname abc-base-image-"${CACHE_HASH}" \
  -t abc-base-image-"${HASH}" debian:buster

(cat <<EOF
results() {
  set +e
  shopt -s dotglob nullglob
  mv /work/abc-ci-builds/"gitian-win"/* /results
  chown -R ${ME} /work
  chown -R ${ME} /results
  chown -R ${ME} /root/.ccache
}
trap "results" EXIT
export TEAMCITY_VERSION="2019.2.4 (build 72059)"
export BASE_CACHE="/root/abc-depends/cache"
export SDK_ARCHIVE_DIR="/root/abc-depends/osx-sdk"
export SOURCES_PATH="/root/abc-depends/sources"
export SDK_DL_REMOTE="http://ci.fabien.cash"
mkdir -p "/root/abc-depends/cache" "/root/abc-depends/osx-sdk" "/root/abc-depends/sources"
./contrib/teamcity/build-configurations.py "gitian-win"
EOF
) > run-command.sh
chmod +x run-command.sh

~/infra/docker/docker-run.sh \
  -a "-v /home/teamcity/.ccache:/root/.ccache -v /home/teamcity/.abc-depends:/root/abc-depends -v "${RESULTS_DIR}":/results" \
  -c run-command.sh /work/run-command.sh abc-base-image-"${HASH}" ./run-command.sh

[09:52:46] :	 [Step 1/1] Starting: /home/teamcity/buildAgent/temp/agentTmp/custom_script4296348564427614066
[09:52:46] :	 [Step 1/1] in directory: /home/teamcity/buildAgent/work/jailed-build
[09:52:46]W:	 [Step 1/1] Traceback (most recent call last):
[09:52:46]W:	 [Step 1/1]   File "./contrib/teamcity/build-configurations.py", line 8, in <module>
[09:52:46]W:	 [Step 1/1]     from deepmerge import always_merger
[09:52:46]W:	 [Step 1/1] ModuleNotFoundError: No module named 'deepmerge'
[09:52:46]W:	 [Step 1/1] Process exited with code 1
[09:52:46]E:	 [Step 1/1] Process exited with code 1 (Step: Command Line)
[09:52:47]E:	 [Step 1/1] Step Command Line failed
[09:52:47]E: Ant JUnit report watcher
[09:52:47]E:	 [Ant JUnit report watcher] No reports found for paths:
[09:52:47]E:	 [Ant JUnit report watcher] +:results/test_bitcoin.xml
[09:52:47]E:	 [Ant JUnit report watcher] +:results/**/junit_results*.xml
[09:52:47] : Publishing internal artifacts (2s)
[09:52:50] :	 [Publishing internal artifacts] Publishing 1 file using [WebPublisher]
[09:52:50] :	 [Publishing internal artifacts] Publishing 1 file using [ArtifactsCachePublisher]
[09:52:47]W: Publishing artifacts (2s)
[09:52:47] :	 [Publishing artifacts] Collecting files to publish: [+:results/**/junit_results*.xml]
[09:52:47]W:	 [Publishing artifacts] Artifacts path 'results/**/junit_results*.xml' not found
[09:52:50] : Build finished

Build Bitcoin ABC Diffs / Diff Testing (build-gitian-win) failed.

Tail of the build log:

(cat <<EOF
results() {
  set +e
  shopt -s dotglob nullglob
  mv /work/abc-ci-builds/"build-gitian-win"/* /results
  chown -R ${ME} /work
  chown -R ${ME} /results
  chown -R ${ME} /root/.ccache
}
trap "results" EXIT
export TEAMCITY_VERSION="2019.2.4 (build 72059)"
export BASE_CACHE="/root/abc-depends/cache"
export SDK_ARCHIVE_DIR="/root/abc-depends/osx-sdk"
export SOURCES_PATH="/root/abc-depends/sources"
export SDK_DL_REMOTE="http://ci.fabien.cash"
mkdir -p "/root/abc-depends/cache" "/root/abc-depends/osx-sdk" "/root/abc-depends/sources"
./contrib/teamcity/build-configurations.py "build-gitian-win"
EOF
) > run-command.sh
chmod +x run-command.sh

~/infra/docker/docker-run.sh \
  -a "-v /home/teamcity/.ccache:/root/.ccache -v /home/teamcity/.abc-depends:/root/abc-depends -v "${RESULTS_DIR}":/results" \
  -c run-command.sh /work/run-command.sh abc-base-image-"${HASH}" ./run-command.sh

[09:52:35] :	 [Step 1/1] Starting: /home/teamcity/buildAgent/temp/agentTmp/custom_script7639818514997813846
[09:52:35] :	 [Step 1/1] in directory: /home/teamcity/buildAgent/work/jailed-build
[09:52:35] :	 [Step 1/1] ~/buildAgent/work/jailed-build/bitcoin-abc ~/buildAgent/work/jailed-build
[09:52:35] :	 [Step 1/1] ~/buildAgent/work/jailed-build
[09:52:35] :	 [Step 1/1] Building base image for: edd663b35...
[09:52:35] :	 [Step 1/1] ~/buildAgent/work/jailed-build/bitcoin-abc ~/buildAgent/work/jailed-build
[09:52:38] :	 [Step 1/1] ~/buildAgent/work/jailed-build
[09:52:38] :	 [Step 1/1] Tag name: abc-base-image-edd663b35
[09:52:50]W:	 [Step 1/1] Traceback (most recent call last):
[09:52:50]W:	 [Step 1/1]   File "./contrib/teamcity/build-configurations.py", line 573, in <module>
[09:52:50]W:	 [Step 1/1]     main()
[09:52:50]W:	 [Step 1/1]   File "./contrib/teamcity/build-configurations.py", line 562, in main
[09:52:50]W:	 [Step 1/1]     script_dir, config_path, args.build)
[09:52:50]W:	 [Step 1/1]   File "./contrib/teamcity/build-configurations.py", line 59, in __init__
[09:52:50]W:	 [Step 1/1]     self.load(build_name)
[09:52:50]W:	 [Step 1/1]   File "./contrib/teamcity/build-configurations.py", line 82, in load
[09:52:50]W:	 [Step 1/1]     self.name, list(config.keys())
[09:52:50]W:	 [Step 1/1] AssertionError: build-gitian-win is not a valid build identifier. Valid identifiers are ['templates', 'builds']
[09:52:50]W:	 [Step 1/1] mv: missing destination file operand after '/results'
[09:52:50]W:	 [Step 1/1] Try 'mv --help' for more information.
[09:52:54]W:	 [Step 1/1] Process exited with code 1
[09:52:54]E:	 [Step 1/1] Process exited with code 1 (Step: Command Line)
[09:52:55]E:	 [Step 1/1] Step Command Line failed
[09:52:55]E: Ant JUnit report watcher
[09:52:55]E:	 [Ant JUnit report watcher] No reports found for paths:
[09:52:55]E:	 [Ant JUnit report watcher] +:results/test_bitcoin.xml
[09:52:55]E:	 [Ant JUnit report watcher] +:results/**/junit_results*.xml
[09:52:55] : Publishing internal artifacts
[09:52:56] :	 [Publishing internal artifacts] Publishing 1 file using [WebPublisher]
[09:52:56] :	 [Publishing internal artifacts] Publishing 1 file using [ArtifactsCachePublisher]
[09:52:55]W: Publishing artifacts
[09:52:55] :	 [Publishing artifacts] Collecting files to publish: [+:results/**/junit_results*.xml]
[09:52:55]W:	 [Publishing artifacts] Artifacts path 'results/**/junit_results*.xml' not found
[09:52:56] : Build finished

PiRK added a reverting change: D10585: Revert "add a wallet dependency to sqlite3".Nov 29 2021, 09:12

add a wallet dependency to sqlite3
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 30890

contrib/debian/control

contrib/gitian-descriptors/gitian-linux.yml

contrib/gitian-descriptors/gitian-win.yml

contrib/utils/install-dependencies.sh

depends/Makefile

depends/README.md

depends/packages/packages.mk

depends/packages/sqlite.mk

doc/build-osx.md

doc/build-unix.md

doc/dependencies.md

src/wallet/CMakeLists.txt

add a wallet dependency to sqlite3ClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 30890

contrib/debian/control

contrib/gitian-descriptors/gitian-linux.yml

contrib/gitian-descriptors/gitian-win.yml

contrib/utils/install-dependencies.sh

depends/Makefile

depends/README.md

depends/packages/packages.mk

depends/packages/sqlite.mk

doc/build-osx.md

doc/build-unix.md

doc/dependencies.md

src/wallet/CMakeLists.txt

add a wallet dependency to sqlite3
ClosedPublic
Actions

Revision Contents
Changeset List