diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d8dc21c69..475d11610 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,290 +1,274 @@ Contributing to Bitcoin ABC =========================== The Bitcoin ABC project welcomes contributors! This guide is intended to help developers contribute effectively to Bitcoin ABC. Communicating with Developers ----------------------------- To get in contact with Bitcoin ABC developers, you can join the [eCash Development Telegram group](https://t.me/eCashDevelopment). The intent of this group is to facilitate development of Bitcoin ABC and other eCash node implementations. We welcome people who wish to participate. Acceptable use of this group includes the following: * Introducing yourself to other eCash developers. * Getting help with your development environment. * Discussing how to complete a patch. It is not for: * Market discussion * Non-constructive criticism Bitcoin ABC Development Philosophy ---------------------------------- Bitcoin ABC aims for fast iteration and continuous integration. This means that there should be quick turnaround for patches to be proposed, reviewed, and committed. Changes should not sit in a queue for long. Here are some tips to help keep the development working as intended. These are guidelines for the normal and expected development process. Developers can use their judgement to deviate from these guidelines when they have a good reason to do so. - Keep each change small and self-contained. - Reach out for a 1-on-1 review so things move quickly. - Land the Diff quickly after it is accepted. - Don't amend changes after the Diff accepted, new Diff for another fix. - Review Diffs from other developers as quickly as possible. - Large changes should be broken into logical chunks that are easy to review, and keep the code in a functional state. - Do not mix moving stuff around with changing stuff. Do changes with renames on their own. - Sometimes you want to replace one subsystem by another implementation, in which case it is not possible to do things incrementally. In such cases, you keep both implementations in the codebase for a while, as described [here](http://sevangelatos.com/john-carmack-on-parallel-implementations/) - There are no "development" branches, all Diffs apply to the master branch, and should always improve it (no regressions). - Don't break the build, it is important to keep master green as much as possible. If a Diff is landed, and breaks the build, fix it quickly. If it cannot be fixed quickly, it should be reverted, and re-applied later when it no longer breaks the build. - As soon as you see a bug, you fix it. Do not continue on. Fixing the bug becomes the top priority, more important than completing other tasks. - Automate as much as possible, and spend time on things only humans can do. Here are some handy links for development practices aligned with Bitcoin ABC: - [Developer Notes](doc/developer-notes.md) - [Statement of Bitcoin ABC Values and Visions](https://archive.md/ulgFI) - [How to Make Your Code Reviewer Fall in Love with You](https://mtlynch.io/code-review-love/) - [Large Diffs Are Hurting Your Ability To Ship](https://medium.com/@kurtisnusbaum/large-diffs-are-hurting-your-ability-to-ship-e0b2b41e8acf) - [Stacked Diffs: Keeping Phabricator Diffs Small](https://medium.com/@kurtisnusbaum/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6) - [Parallel Implementations](http://sevangelatos.com/john-carmack-on-parallel-implementations/) - [The Pragmatic Programmer: From Journeyman to Master](https://www.amazon.com/Pragmatic-Programmer-Journeyman-Master/dp/020161622X) - [Monorepo: Advantages of monolithic version control](https://danluu.com/monorepo/) - [Monorepo: Why Google Stores Billions of Lines of Code in a Single Repository](https://www.youtube.com/watch?v=W71BTkUbdqE) - [The importance of fixing bugs immediately](https://youtu.be/E2MIpi8pIvY?t=16m0s) - [Slow Deployment Causes Meetings](https://www.facebook.com/notes/kent-beck/slow-deployment-causes-meetings/1055427371156793/) - [Good Work, Great Work, and Right Work](https://forum.dlang.org/post/q7u6g1$94p$1@digitalmars.com) - [Accelerate: The Science of Lean Software and DevOps](https://www.amazon.com/Accelerate-Software-Performing-Technology-Organizations/dp/1942788339) - [Facebook Engineering Process with Kent Beck](https://softwareengineeringdaily.com/2019/08/28/facebook-engineering-process-with-kent-beck/) - [Trunk Based Development](https://trunkbaseddevelopment.com/) - [Step-by-step: Programming incrementally](https://ourmachinery.com/post/step-by-step-programming-incrementally/) - [Semantic Compression](https://caseymuratori.com/blog_0015) Getting set up with the Bitcoin ABC Repository ---------------------------------------------- 1. Create an account at [reviews.bitcoinabc.org](https://reviews.bitcoinabc.org/) 2. Install Git and Arcanist on your machine Git documentation can be found at [git-scm.com](https://git-scm.com/). For Arcanist documentation, you can read [Arcanist Quick Start](https://secure.phabricator.com/book/phabricator/article/arcanist_quick_start/) and the [Arcanist User Guide](https://secure.phabricator.com/book/phabricator/article/arcanist/). To install these packages on Debian or Ubuntu, type: `sudo apt-get install git arcanist` 3. If you do not already have an SSH key set up, follow these steps: Type: `ssh-keygen -t ed25519 -C "your_email@example.com"` Enter a file in which to save the key (/home/*username*/.ssh/id_ed25519): [Press enter] 4. Upload your SSH public key to - Go to: `https://reviews.bitcoinabc.org/settings/user/*username*/page/ssh/` - Under "SSH Key Actions", Select "Upload Public Key" Paste contents from: `/home/*username*/.ssh/id_ed25519.pub` 5. Clone the repository and install Arcanist certificate: ``` git clone ssh://vcs@reviews.bitcoinabc.org:2221/source/bitcoin-abc.git cd bitcoin-abc arc install-certificate ``` Note: Arcanist tooling will tend to fail if your remote origin is set to something other than the above. A common mistake is to clone from Github and then forget to update your remotes. Follow instructions provided by `arc install-certificate` to provide your API token. Contributing to the node software --------------------------------- During submission of patches, arcanist will automatically run `arc lint` to enforce Bitcoin ABC code formatting standards, and often suggests changes. If code formatting tools do not install automatically on your system, you will have to install the following: -On Ubuntu 20.04: +Install all the code formatting tools on Debian Bullseye (11) or Ubuntu 20.04: ``` -sudo apt-get install clang-format-11 clang-tidy-11 clang-tools-11 python3-isort python3-autopep8 flake8 php-codesniffer yamllint +sudo apt-get install clang-format-11 clang-tidy-11 clang-tools-11 python3-autopep8 python3-pip flake8 php-codesniffer shellcheck yamllint +pip3 install isort>=5.6.4 mypy>=0.780 +echo "export PATH=\"`python3 -m site --user-base`/bin:\$PATH\"" >> ~/.bashrc +source ~/.bashrc ``` If not available in the distribution, `clang-format-11` and `clang-tidy-11` can be installed from or . -On Debian (>= 10), the clang-11 family of tools is available from the https://apt.llvm.org/ repository: -``` -wget https://apt.llvm.org/llvm.sh -chmod +x llvm.sh -sudo ./llvm.sh 11 -``` - For example, for macOS: ``` curl -L https://github.com/llvm/llvm-project/releases/download/llvmorg-11.0.0/clang+llvm-11.0.0-x86_64-apple-darwin.tar.xz | tar -xJv ln -s $PWD/clang+llvm-11.0.0-x86_64-apple-darwin/bin/clang-format /usr/local/bin/clang-format ln -s $PWD/clang+llvm-11.0.0-x86_64-apple-darwin/bin/clang-tidy /usr/local/bin/clang-tidy ``` -If you are modifying a python script, you will need to install `mypy` and `isort`. -The minimum required version for `mypy` is 0.780, because the previous ones are -known to have issues with some python type annotations. -On Debian based systems, this can be installed via: -``` -sudo apt-get install python3-pip -pip3 install isort==5.6.4 mypy==0.780 -echo "export PATH=\"`python3 -m site --user-base`/bin:\$PATH\"" >> ~/.bashrc -source ~/.bashrc -``` - If you are modifying a shell script, you will need to install the `shellcheck` linter. A recent version is required and may not be packaged for your distribution. Standalone binaries are available for download on [the project's github release page](https://github.com/koalaman/shellcheck/releases). **Note**: In order for arcanist to detect the `shellcheck` executable, you need to make it available in your `PATH`; if another version is already installed, make sure the recent one is found first. Arcanist will tell you what version is expected and what is found when running `arc lint` against a shell script. - If you are running Debian 10, it is also available in the backports repository: ``` sudo apt-get -t buster-backports install shellcheck ``` If you are modifying Rust files, you will need to install a stable rust version, plus a nightly toolchain called "abc-nightly" for formatting: ```bash # Install latest stable Rust version curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s source ~/.cargo/env rustup install nightly-2022-06-29 rustup component add rustfmt --toolchain nightly-2022-06-29 # Name the nightly toolchain "abc-nightly" rustup toolchain link abc-nightly "$(rustc +nightly-2022-06-29 --print sysroot)" ``` Contributing to the web projects -------------------------------- To contribute to web projects, you will need `nodejs` > 16 and `npm` > 8.5.0. Follow these [installation instructions](https://github.com/nvm-sh/nvm#installing-and-updating) to install `nodejs` with node version manager. Then: ``` cd bitcoin-abc [sudo] nvm install 16 [sudo] npm install -g npm@latest [sudo] npm install -g prettier ``` To work on the extension, you will need `browserify` ``` [sudo] npm install -g browserify ``` Working with The Bitcoin ABC Repository --------------------------------------- A typical workflow would be: - Create a topic branch in Git for your changes git checkout -b 'my-topic-branch' - Make your changes, and commit them git commit -a -m 'my-commit' - Create a differential with Arcanist arc diff You should add suggested reviewers and a test plan to the commit message. Note that Arcanist is set up to look only at the most-recent commit message, So all you changes for this Diff should be in one Git commit. - For large changes, break them into several Diffs, as described in this [guide](https://medium.com/@kurtisnusbaum/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6). You must also include "Depends on Dxxx" in the Arcanist summary to indicate dependence on other Diffs. Note: the `arc land` procedure described in the guide above is obsolete. With the most recent version of arcanist, you may `arc land` the latest commit of your stacked diff after all parts are approved. - Log into Phabricator to see review and feedback. - Make changes as suggested by the reviewers. You can simply edit the files with my-topic-branch checked out, and then type `arc diff`. Arcanist will give you the option to add uncommited changes. Or, alternatively, you can commit the changes using `git commit -a --am` to add them to the last commit, or squash multiple commits by typing `git rebase -i master`. If you squash, make sure the commit message has the information needed for arcanist (such as the Diff number, reviewers, etc.). - Update your Diff by typing `arc diff` again. - When reviewers approve your Diff, it should be listed as "ready to Land" in Phabricator. When you want to commit your diff to the repository, check out type my-topic-branch in git, then type `arc land`. You have now successfully committed a change to the Bitcoin ABC repository. - When reviewing a Diff, apply the changeset on your local by using `arc patch D{NNNN}` - You will likely be re-writing git histories multiple times, which causes timestamp changes that require re-building a significant number of files. It's highly recommended to install `ccache` (re-run cmake if you install it later), as this will help cut your re-build times from several minutes to under a minute, in many cases. What to work on --------------- If you are looking for a useful task to contribute to the project, a good place to start is the list of tasks at . You could also try [backporting](doc/backporting.md) some code from Bitcoin Core. Copyright --------- By contributing to this repository, you agree to license your work under the MIT license unless specified otherwise in `contrib/debian/copyright` or at the top of the file itself. Any work contributed where you are not the original author must contain its license header with the original author(s) and source. Disclosure Policy ----------------- See [DISCLOSURE_POLICY](DISCLOSURE_POLICY.md). diff --git a/contrib/docker/bitcoin-abc-dev/Dockerfile b/contrib/docker/bitcoin-abc-dev/Dockerfile index 4273f8393..26e0aa7f8 100644 --- a/contrib/docker/bitcoin-abc-dev/Dockerfile +++ b/contrib/docker/bitcoin-abc-dev/Dockerfile @@ -1,16 +1,16 @@ FROM debian:buster # Expected to run from the project root COPY . /bitcoin-abc VOLUME /bitcoin-abc # Run the dependency installation script in its own directory so we can clean # it after it's done RUN mkdir -p /install WORKDIR /install -RUN /bitcoin-abc/contrib/utils/install-dependencies.sh +RUN /bitcoin-abc/contrib/utils/install-dependencies-buster.sh WORKDIR /bitcoin-abc RUN rm -rf /install CMD ["bash"] diff --git a/contrib/teamcity/setup-debian-buster.sh b/contrib/teamcity/setup-debian-bullseye.sh similarity index 95% copy from contrib/teamcity/setup-debian-buster.sh copy to contrib/teamcity/setup-debian-bullseye.sh index 2bc985115..1cbc6ba13 100755 --- a/contrib/teamcity/setup-debian-buster.sh +++ b/contrib/teamcity/setup-debian-bullseye.sh @@ -1,30 +1,30 @@ #!/usr/bin/env bash export LC_ALL=C.UTF-8 set -euxo pipefail TEAMCITY_DIR=$(dirname "$0") # Install all the build dependencies -"${TEAMCITY_DIR}"/../utils/install-dependencies.sh +"${TEAMCITY_DIR}"/../utils/install-dependencies-bullseye.sh # Python library for interacting with teamcity pip3 install teamcity-messages # Install Python dependencies for the build bot # Note: Path should be relative to TEAMCITY_DIR since the base image build # context may be different than the project root. pip3 install -r "${TEAMCITY_DIR}"/../buildbot/requirements.txt # Make sure our specific llvm and clang versions have highest priority update-alternatives --install /usr/bin/clang clang "$(command -v clang-11)" 100 update-alternatives --install /usr/bin/clang++ clang++ "$(command -v clang++-11)" 100 update-alternatives --install /usr/bin/llvm-symbolizer llvm-symbolizer "$(command -v llvm-symbolizer-11)" 100 # Set default git config so that any git operations requiring authoring, # rebasing, or cherry-picking of commits just work out of the box. git config --global user.name "abc-bot" git config --global user.email "no-email-abc-bot@bitcoinabc.org" # npm uses ssh to connect to github by default, use https instead git config --global url."https://github.com".insteadOf ssh://git@github.com diff --git a/contrib/teamcity/setup-debian-buster.sh b/contrib/teamcity/setup-debian-buster.sh index 2bc985115..1ec146eff 100755 --- a/contrib/teamcity/setup-debian-buster.sh +++ b/contrib/teamcity/setup-debian-buster.sh @@ -1,30 +1,30 @@ #!/usr/bin/env bash export LC_ALL=C.UTF-8 set -euxo pipefail TEAMCITY_DIR=$(dirname "$0") # Install all the build dependencies -"${TEAMCITY_DIR}"/../utils/install-dependencies.sh +"${TEAMCITY_DIR}"/../utils/install-dependencies-buster.sh # Python library for interacting with teamcity pip3 install teamcity-messages # Install Python dependencies for the build bot # Note: Path should be relative to TEAMCITY_DIR since the base image build # context may be different than the project root. pip3 install -r "${TEAMCITY_DIR}"/../buildbot/requirements.txt # Make sure our specific llvm and clang versions have highest priority update-alternatives --install /usr/bin/clang clang "$(command -v clang-11)" 100 update-alternatives --install /usr/bin/clang++ clang++ "$(command -v clang++-11)" 100 update-alternatives --install /usr/bin/llvm-symbolizer llvm-symbolizer "$(command -v llvm-symbolizer-11)" 100 # Set default git config so that any git operations requiring authoring, # rebasing, or cherry-picking of commits just work out of the box. git config --global user.name "abc-bot" git config --global user.email "no-email-abc-bot@bitcoinabc.org" # npm uses ssh to connect to github by default, use https instead git config --global url."https://github.com".insteadOf ssh://git@github.com diff --git a/contrib/utils/install-dependencies.sh b/contrib/utils/install-dependencies-bullseye.sh similarity index 78% copy from contrib/utils/install-dependencies.sh copy to contrib/utils/install-dependencies-bullseye.sh index 689a6c516..845087df5 100755 --- a/contrib/utils/install-dependencies.sh +++ b/contrib/utils/install-dependencies-bullseye.sh @@ -1,167 +1,152 @@ #!/usr/bin/env bash export LC_ALL=C.UTF-8 set -euxo pipefail dpkg --add-architecture i386 PACKAGES=( arcanist automake autotools-dev binutils bsdmainutils build-essential ccache + clang-11 + clang-format-11 + clang-tidy-11 + clang-tools-11 + cmake curl default-jdk devscripts doxygen dput flake8 + g++-9 g++-aarch64-linux-gnu g++-arm-linux-gnueabihf + gcc-9 gettext-base git golang g++-mingw-w64 gnupg graphviz gperf help2man imagemagick jq lcov less - lib32stdc++-8-dev + lib32stdc++-10-dev libboost-all-dev libbz2-dev libc6-dev:i386 libcap-dev libdb++-dev libdb-dev libevent-dev libjemalloc-dev libminiupnpc-dev libnatpmp-dev libprotobuf-dev libqrencode-dev libqt5core5a libqt5dbus5 libqt5gui5 librsvg2-bin libsqlite3-dev libssl-dev libtiff-tools libtinfo5 libtool libzmq3-dev lld make ninja-build nsis + pandoc php-codesniffer pkg-config protobuf-compiler python3 python3-autopep8 python3-pip python3-setuptools python3-yaml python3-zmq qemu-user-static qttools5-dev qttools5-dev-tools + shellcheck software-properties-common tar wget xvfb yamllint wine ) function join_by() { local IFS="$1" shift echo "$*" } apt-get update DEBIAN_FRONTEND=noninteractive apt-get install -y $(join_by ' ' "${PACKAGES[@]}") -BACKPORTS=( - cmake - shellcheck -) - -echo "deb http://deb.debian.org/debian buster-backports main" | tee -a /etc/apt/sources.list -apt-get update -DEBIAN_FRONTEND=noninteractive apt-get -t buster-backports install -y $(join_by ' ' "${BACKPORTS[@]}") - - -# Install llvm and clang -apt-key add "$(dirname "$0")"/llvm.pub -add-apt-repository "deb https://apt.llvm.org/buster/ llvm-toolchain-buster-8 main" -add-apt-repository "deb https://apt.llvm.org/buster/ llvm-toolchain-buster-11 main" -apt-get update - -LLVM_PACKAGES=( - clang-11 - clang-format-11 - clang-tidy-11 - clang-tools-11 -) -DEBIAN_FRONTEND=noninteractive apt-get install -y $(join_by ' ' "${LLVM_PACKAGES[@]}") +# Use gcc-9/g++-9 by default so it uses libstdc++-9. This prevents from pulling +# the new pthread_cond_clockwait symbol from GLIBC_30 and ensure we are testing +# under the same condition our release it built. +update-alternatives --install /usr/bin/gcc gcc "$(command -v gcc-9)" 100 +update-alternatives --install /usr/bin/g++ g++ "$(command -v g++-9)" 100 # Use the mingw posix variant update-alternatives --set x86_64-w64-mingw32-g++ $(command -v x86_64-w64-mingw32-g++-posix) update-alternatives --set x86_64-w64-mingw32-gcc $(command -v x86_64-w64-mingw32-gcc-posix) # Python library for merging nested structures pip3 install deepmerge # For running Python test suites pip3 install pytest # Up-to-date mypy and isort packages are required python linters pip3 install isort==5.6.4 mypy==0.780 echo "export PATH=\"$(python3 -m site --user-base)/bin:\$PATH\"" >> ~/.bashrc # shellcheck source=/dev/null source ~/.bashrc -# Install pandoc. The version from buster is outdated, so get a more recent one -# from github. -wget https://github.com/jgm/pandoc/releases/download/2.10.1/pandoc-2.10.1-1-amd64.deb -echo "4515d6fe2bf8b82765d8dfa1e1b63ccb0ff3332d60389f948672eaa37932e936 pandoc-2.10.1-1-amd64.deb" | sha256sum -c -DEBIAN_FRONTEND=noninteractive dpkg -i pandoc-2.10.1-1-amd64.deb - # Install npm v7.x and nodejs v15.x curl -sL https://deb.nodesource.com/setup_15.x | bash - apt-get install -y nodejs # Install Rust stable 1.61 and nightly from the 2022-06-29 curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain=1.61.0 RUST_HOME="${HOME}/.cargo/bin" RUST_NIGHTLY_DATE=2022-06-29 "${RUST_HOME}/rustup" install nightly-${RUST_NIGHTLY_DATE} "${RUST_HOME}/rustup" component add rustfmt --toolchain nightly-${RUST_NIGHTLY_DATE} # Name the nightly toolchain "abc-nightly" "${RUST_HOME}/rustup" toolchain link abc-nightly "$(${RUST_HOME}/rustc +nightly-${RUST_NIGHTLY_DATE} --print sysroot)" # Install required compile platform targets on stable "${RUST_HOME}/rustup" target add "i686-unknown-linux-gnu" \ "x86_64-unknown-linux-gnu" \ "aarch64-unknown-linux-gnu" \ "arm-unknown-linux-gnueabihf" \ "x86_64-apple-darwin" \ "x86_64-pc-windows-gnu" # Install corrosion from Github wget https://api.github.com/repos/corrosion-rs/corrosion/tarball/v0.2.1 -O corrosion.tar.gz echo "49fdaa6af103c5523cc940e73a23c67e5b25d4b74f4ee55a8b7a524a4f815517 corrosion.tar.gz" | sha256sum -c tar xzf corrosion.tar.gz CORROSION_SRC_FOLDER=corrosion-rs-corrosion-28fa50c CORROSION_BUILD_FOLDER=${CORROSION_SRC_FOLDER}-build cmake -S${CORROSION_SRC_FOLDER} -B${CORROSION_BUILD_FOLDER} -DCMAKE_BUILD_TYPE=Release cmake --build ${CORROSION_BUILD_FOLDER} --config Release cmake --install ${CORROSION_BUILD_FOLDER} --config Release diff --git a/contrib/utils/install-dependencies.sh b/contrib/utils/install-dependencies-buster.sh similarity index 100% rename from contrib/utils/install-dependencies.sh rename to contrib/utils/install-dependencies-buster.sh diff --git a/depends/README.md b/depends/README.md index 787fb1fd6..07b31d9d5 100644 --- a/depends/README.md +++ b/depends/README.md @@ -1,123 +1,123 @@ ### Usage To build dependencies for the current arch+OS: make To build for another arch/OS: make build- Where `` is one of the following: - linux64 - linux32 - linux-arm - linux-aarch64 - osx - win64 For example, building the dependencies for macOS: make build-osx Note that it will use all the CPU cores available on the machine by default. This behavior can be changed by setting the `JOBS` environment variable (see below). To use the dependencies for building Bitcoin ABC, you need to set the platform file to be used by `cmake`. The platform files are located under `cmake/platforms/`. For example, cross-building for macOS (run from the project root): mkdir build_osx cd build_osx cmake -GNinja .. -DCMAKE_TOOLCHAIN_FILE=../cmake/platforms/OSX.cmake ninja No other options are needed, the paths are automatically configured. ### Install the required dependencies: Ubuntu & Debian #### Common to all arch/OS sudo apt-get install build-essential autoconf automake cmake curl git libtool ninja-build patch pkg-config python3 #### For macOS cross compilation sudo apt-get install imagemagick libbz2-dev libcap-dev librsvg2-bin libtiff-tools libtinfo5 python3-setuptools #### For Win64 cross compilation - see [build-windows.md](../doc/build-windows.md#cross-compilation-for-ubuntu-and-windows-subsystem-for-linux) #### For linux cross compilation Common linux dependencies: sudo apt-get install gperf For linux 32 bits cross compilation: First add the i386 architecture to `dpkg`: sudo dpkg --add-architecture i386 sudo apt-get update Then install the dependencies: - sudo apt-get install lib32stdc++-8-dev libc6-dev:i386 + sudo apt-get install lib32stdc++-10-dev libc6-dev:i386 For linux ARM cross compilation: sudo apt-get install g++-arm-linux-gnueabihf For linux AARCH64 cross compilation: sudo apt-get install g++-aarch64-linux-gnu ### Dependency Options The following can be set when running make: `make FOO=bar` - `SOURCES_PATH`: Downloaded sources will be placed here - `BASE_CACHE`: Built packages will be placed here - `SDK_PATH`: Path where SDKs can be found (used by macOS) - `FALLBACK_DOWNLOAD_PATH`: If a source file can't be fetched, try here before giving up - `NO_QT`: Don't download/build/cache Qt and its dependencies - `NO_QR`: Don't download/build/cache packages needed for enabling qrencode - `NO_ZMQ`: Don't download/build/cache packages needed for enabling ZeroMQ - `NO_WALLET`: Don't download/build/cache libs needed to enable the wallet - `NO_BDB`: Don't download/build/cache BerkeleyDB - `NO_SQLITE`: Don't download/build/cache SQLite - `NO_UPNP`: Don't download/build/cache packages needed for enabling UPnP - `NO_NATPMP`: Don't download/build/cache packages needed for enabling NAT-PMP - `NO_JEMALLOC`: Don't download/build/cache jemalloc - `DEBUG`: Disable some optimizations and enable more runtime checking - `NO_PROTOBUF`: Don't download/build/cache protobuf (used for BIP70 support) - `HOST_ID_SALT`: Optional salt to use when generating host package ids - `BUILD_ID_SALT`: Optional salt to use when generating build package ids - `JOBS`: Number of jobs to use for each package build - `FORCE_USE_SYSTEM_CLANG`: (EXPERTS ONLY) When cross-compiling for macOS, use Clang found in the system's `$PATH` rather than the default prebuilt release of Clang from llvm.org. Clang 8 or later is required. If some packages are not built, for example by building the depends with `make NO_WALLET=1`, the appropriate options should be set when building Bitcoin ABC using these dependencies. In this example, `-DBUILD_BITCOIN_WALLET=OFF` should be passed to the `cmake` command line to ensure that the build will not fail due to missing dependencies. Additional targets: download: run 'make download' to fetch all sources without building them download-osx: run 'make download-osx' to fetch all sources needed for macOS builds download-win: run 'make download-win' to fetch all sources needed for win builds download-linux: run 'make download-linux' to fetch all sources needed for linux builds build-all: build the dependencies for all the arch/OS ### Other documentation - [description.md](description.md): General description of the depends system - [packages.md](packages.md): Steps for adding packages diff --git a/doc/build-unix.md b/doc/build-unix.md index 0d4cce80d..457a2544d 100644 --- a/doc/build-unix.md +++ b/doc/build-unix.md @@ -1,342 +1,338 @@ UNIX BUILD NOTES ==================== Some notes on how to build Bitcoin ABC in Unix. To Build --------------------- Before you start building, please make sure that your compiler supports C++17. It is recommended to create a build directory to build out-of-tree. ```bash mkdir build cd build cmake -GNinja .. ninja ninja install # optional ``` This will build bitcoin-qt as well. Dependencies --------------------- *Note: Bitcoin ABC provides a [Docker image with all the dependencies preinstalled](#build-using-a-docker-container).* These dependencies are required: Library | Purpose | Description ------------|------------------|---------------------- libssl | Crypto | Random Number Generation, Elliptic Curve Cryptography libboost | Utility | Library for threading, data structures, etc libevent | Networking | OS independent asynchronous networking Optional dependencies: Library | Purpose | Description ------------|------------------|---------------------- miniupnpc | UPnP Support | Firewall-jumping support libnatpmp | NAT-PMP Support | Firewall-jumping support libdb | Berkeley DB | Wallet storage (only needed when wallet enabled) libsqlite3 | SQLite 3 | Wallet storage (only needed when wallet enabled) jemalloc | Memory allocator | Library to enhance the memory allocation and improve performances qt | GUI | GUI toolkit (only needed when GUI enabled) protobuf | Payments in GUI | Data interchange format used for payment protocol (only needed when BIP70 enabled) libqrencode | QR codes in GUI | Optional for generating QR codes (only needed when GUI enabled) univalue | Utility | JSON parsing and encoding (bundled version will be used unless --with-system-univalue passed to configure) libzmq3 | ZMQ notification | Optional, allows generating ZMQ notifications (requires ZMQ version >= 4.1.5) For the versions used, see [dependencies.md](dependencies.md) Memory Requirements -------------------- C++ compilers are memory-hungry. It is recommended to have at least 1.5 GB of memory available when compiling Bitcoin ABC. On systems with less, gcc can be tuned to conserve memory with additional CXXFLAGS: cmake -GNinja .. -DCMAKE_CXX_FLAGS="--param ggc-min-expand=1 --param ggc-min-heapsize=32768" Dependency Build Instructions: Ubuntu & Debian ---------------------------------------------- Build requirements: - sudo apt-get install bsdmainutils build-essential libssl-dev libevent-dev lld ninja-build python3 + sudo apt-get install bsdmainutils build-essential cmake libssl-dev libevent-dev lld ninja-build python3 -**Installing cmake:** +**Installing cmake on older distribution (Debian < 11 or Ubuntu < 20.04):** On Debian Buster (10), `cmake` should be installed from the backports repository: echo "deb http://deb.debian.org/debian buster-backports main" | sudo tee -a /etc/apt/sources.list sudo apt-get update sudo apt-get -t buster-backports install cmake -On Ubuntu 20.04 and later: - - sudo apt-get install cmake - -On previous Ubuntu versions, the `cmake` package is too old and needs to be installed from the Kitware APT repository: +If the `cmake` packaged version is too old it can be installed from the Kitware APT repository: sudo apt-get install apt-transport-https ca-certificates gnupg software-properties-common wget wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | sudo apt-key add - Add the repository corresponding to your version (see [instructions from Kitware](https://apt.kitware.com)). For Ubuntu Bionic (18.04): sudo apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main' Then update the package list and install `cmake`: sudo apt update sudo apt install cmake Now, you can either build from self-compiled [depends](/depends/README.md) or install the required dependencies with the following instructions. Options when installing required Boost library files: 1. On at least Ubuntu 16.04+ and Debian 9+ there are generic names for the individual boost development packages, so the following can be used to only install necessary parts of boost: sudo apt-get install libboost-system-dev libboost-filesystem-dev libboost-test-dev libboost-thread-dev 2. If that doesn't work, you can install all boost development packages with: sudo apt-get install libboost-all-dev BerkeleyDB 5.3 or later and SQLite 3.7 or later are required for the wallet. These can be installed with: sudo apt-get install libdb-dev libdb++-dev libsqlite3-dev See the section "Disable-wallet mode" to build Bitcoin ABC without wallet. Port mapping dependencies MiniUPnPc and NAT-PMP (can be disabled by passing `-DENABLE_UPNP=OFF` and `-DENABLE_NATPMP=OFF` on the cmake command line): sudo apt-get install libminiupnpc-dev libnatpmp-dev ZMQ dependencies (provides ZMQ API, can be disabled by passing `-DBUILD_BITCOIN_ZMQ=OFF` on the cmake command line): sudo apt-get install libzmq3-dev jemalloc dependencies (provides the jemalloc library, can be disabled by passing `-DUSE_JEMALLOC=OFF` on the cmake command line): sudo apt-get install libjemalloc-dev Dependencies for the GUI: Ubuntu & Debian ----------------------------------------- If you want to build bitcoin-qt, make sure that the required packages for Qt development are installed. Qt 5 is necessary to build the GUI. To build without GUI pass `-DBUILD_BITCOIN_QT=OFF` on the cmake command line. To build with Qt 5 you need the following: sudo apt-get install libqt5gui5 libqt5core5a libqt5dbus5 qttools5-dev qttools5-dev-tools libprotobuf-dev protobuf-compiler libqrencode dependencies (can be disabled by passing `-DENABLE_QRCODE=OFF` on the cmake command line): sudo apt-get install libqrencode-dev Dependency Build Instructions: Fedora ------------------------------------- Build requirements: sudo dnf install boost-devel cmake gcc-c++ libdb-cxx-devel libdb-devel libevent-devel ninja-build openssl-devel python3 Port mapping dependencies MiniUPnPc and NAT-PMP (can be disabled by passing `-DENABLE_UPNP=OFF` and `-DENABLE_NATPMP=OFF` on the cmake command line): sudo dnf install miniupnpc-devel libnatpmp-devel ZMQ dependencies (can be disabled by passing `-DBUILD_BITCOIN_ZMQ=OFF` on the cmake command line): sudo dnf install zeromq-devel To build with Qt 5 you need the following: sudo dnf install qt5-qttools-devel qt5-qtbase-devel protobuf-devel libqrencode dependencies (can be disabled by passing `-DENABLE_QRCODE=OFF`): sudo dnf install qrencode-devel SQLite can be installed with: sudo dnf install sqlite-devel Notes ----- The release is built with GCC and then "strip bitcoind" to strip the debug symbols, which reduces the executable size by about 90%. miniupnpc --------- [miniupnpc](https://miniupnp.tuxfamily.org) may be used for UPnP port mapping. It can be downloaded from [here]( https://miniupnp.tuxfamily.org/files/). UPnP support is compiled in and turned off by default. See the cmake options for UPnP behavior desired: ENABLE_UPNP Enable UPnP support (miniupnp required, default ON) START_WITH_UPNP UPnP support turned on by default at runtime (default OFF) libnatpmp --------- [libnatpmp](https://miniupnp.tuxfamily.org/libnatpmp.html) may be used for NAT-PMP port mapping. It can be downloaded from [here](https://miniupnp.tuxfamily.org/files/). NAT-PMP support is compiled in and turned off by default. See the configure options for NAT-PMP behavior desired: ENABLE_NATPMP NAT-PMP support (libnatpmp required, default ON) START_WITH_NATPMP NAT-PMP support turned on by default at runtime (default OFF) Boost ----- For documentation on building Boost look at their official documentation: http://www.boost.org/build/doc/html/bbv2/installation.html Security -------- To help make your Bitcoin ABC installation more secure by making certain attacks impossible to exploit even if a vulnerability is found, binaries are hardened by default. This can be disabled by passing `-DENABLE_HARDENING=OFF`. Hardening enables the following features: * _Position Independent Executable_: Build position independent code to take advantage of Address Space Layout Randomization offered by some kernels. Attackers who can cause execution of code at an arbitrary memory location are thwarted if they don't know where anything useful is located. The stack and heap are randomly located by default, but this allows the code section to be randomly located as well. On an AMD64 processor where a library was not compiled with -fPIC, this will cause an error such as: "relocation R_X86_64_32 against `......' can not be used when making a shared object;" To test that you have built PIE executable, install scanelf, part of paxutils, and use: scanelf -e ./bitcoin The output should contain: TYPE ET_DYN * _Non-executable Stack_: If the stack is executable then trivial stack-based buffer overflow exploits are possible if vulnerable buffers are found. By default, Bitcoin ABC should be built with a non-executable stack, but if one of the libraries it uses asks for an executable stack or someone makes a mistake and uses a compiler extension which requires an executable stack, it will silently build an executable without the non-executable stack protection. To verify that the stack is non-executable after compiling use: scanelf -e ./bitcoin The output should contain: STK/REL/PTL RW- R-- RW- The `STK RW-` means that the stack is readable and writeable but not executable. Disable-wallet mode -------------------- When the intention is to run only a P2P node without a wallet, Bitcoin ABC may be compiled in disable-wallet mode by passing `-DBUILD_BITCOIN_WALLET=OFF` on the cmake command line. Mining is also possible in disable-wallet mode using the `getblocktemplate` RPC call. Additional cmake options -------------------------- A list of the cmake options and their current value can be displayed. From the build subdirectory (see above), run `cmake -LH ..`. Setup and Build Example: Arch Linux ----------------------------------- This example lists the steps necessary to setup and build a command line only, non-wallet distribution of the latest changes on Arch Linux: pacman -S base-devel boost cmake git libevent ninja python git clone https://github.com/Bitcoin-ABC/bitcoin-abc.git cd bitcoin-abc/ mkdir build cd build cmake -GNinja .. -DBUILD_BITCOIN_WALLET=OFF -DBUILD_BITCOIN_QT=OFF -DENABLE_UPNP=OFF -DBUILD_BITCOIN_ZMQ=OFF -DUSE_JEMALLOC=OFF ninja ARM Cross-compilation ------------------- These steps can be performed on, for example, a Debian VM. The depends system will also work on other Linux distributions, however the commands for installing the toolchain will be different. Make sure you install all the build requirements mentioned above. Then, install the toolchain and some additional dependencies: sudo apt-get install autoconf automake curl g++-arm-linux-gnueabihf gcc-arm-linux-gnueabihf gperf pkg-config To build executables for ARM: cd depends make build-linux-arm cd .. mkdir build cd build cmake -GNinja .. -DCMAKE_TOOLCHAIN_FILE=../cmake/platforms/LinuxARM.cmake -DENABLE_GLIBC_BACK_COMPAT=ON -DENABLE_STATIC_LIBSTDCXX=ON ninja For further documentation on the depends system see [README.md](../depends/README.md) in the depends directory. Build using a Docker container ------------------------------- Bitcoin ABC provides a [Docker image](https://hub.docker.com/r/bitcoinabc/bitcoin-abc-dev) with all the dependencies pre-installed, based on Debian. If the dependencies cannot be installed on your system but it can run a Docker container, this image can be pulled and used for the build. *Note: The image has all the dependencies and can weight a few gigabytes.* To get the latest image (current master): ```shell docker pull bitcoinabc/bitcoin-abc-dev ``` It is also possible to use a release version. Example for 0.22.4: ```shell docker pull bitcoinabc/bitcoin-abc-dev:0.22.4 ``` Running the container will start a `bash` shell at the project root: ```shell # On the host docker run -it bitcoinabc/bitcoin-abc-dev # Start the build in the container mkdir build cd build cmake -GNinja .. ninja ``` It is possible to bind the project to a local directory on the host machine. First create an empty volume on the host: ```shell # On the host mkdir bitcoin-abc-volume docker volume create \ --driver local \ --opt type=none \ --opt device=${PWD}/bitcoin-abc-volume \ --opt o=bind \ bitcoin-abc-volume ``` Then start the container with the volume bound to `/bitcoin-abc`: ```shell docker run -it -v bitcoin-abc-volume:/bitcoin-abc bitcoinabc/bitcoin-abc-dev ``` diff --git a/doc/build-windows.md b/doc/build-windows.md index cd19a8ed9..22a1b2e8a 100644 --- a/doc/build-windows.md +++ b/doc/build-windows.md @@ -1,156 +1,156 @@ WINDOWS BUILD NOTES ==================== Below are some notes on how to build Bitcoin ABC for Windows. The options known to work for building Bitcoin ABC on Windows are: -* On Linux, using the [Mingw-w64](https://mingw-w64.org/doku.php) cross compiler tool chain. Debian Buster is recommended +* On Linux, using the [Mingw-w64](https://mingw-w64.org/doku.php) cross compiler tool chain. Debian Bullseye is recommended and is the platform used to build the Bitcoin ABC Windows release binaries. * On Windows, using [Windows Subsystem for Linux (WSL)](https://msdn.microsoft.com/commandline/wsl/about) and the Mingw-w64 cross compiler tool chain. Other options which may work, but which have not been extensively tested are (please contribute instructions): * On Windows, using a POSIX compatibility layer application such as [cygwin](http://www.cygwin.com/) or [msys2](http://www.msys2.org/). * On Windows, using a native compiler tool chain such as [Visual Studio](https://www.visualstudio.com). In any case please make sure that the compiler supports C++17. Installing Windows Subsystem for Linux --------------------------------------- With Windows 10, Microsoft has released a new feature named the [Windows Subsystem for Linux (WSL)](https://msdn.microsoft.com/commandline/wsl/about). This feature allows you to run a bash shell directly on Windows in an Ubuntu-based environment. Within this environment you can cross compile for Windows without the need for a separate Linux VM or server. Note that while WSL can be installed with other Linux variants, such as OpenSUSE, the following instructions have only been tested with Ubuntu Bionic. This feature is not supported in versions of Windows prior to Windows 10 or on Windows Server SKUs. In addition, it is available [only for 64-bit versions of Windows](https://msdn.microsoft.com/en-us/commandline/wsl/install_guide). Full instructions to install WSL are available on the above link. To install WSL on Windows 10 with Fall Creators Update installed (version >= 16215.0) do the following: 1. Enable the Windows Subsystem for Linux feature * Open the Windows Features dialog (`OptionalFeatures.exe`) * Enable 'Windows Subsystem for Linux' * Click 'OK' and restart if necessary 2. Install Ubuntu * Open Microsoft Store and search for Ubuntu or use [this link](https://www.microsoft.com/store/productId/9NBLGGH4MSV6) * Click Install 3. Complete Installation * Open a cmd prompt and type "Ubuntu" * Create a new UNIX user account (this is a separate account from your Windows account) After the bash shell is active, you can follow the instructions below, starting with the "Cross-compilation" section. Compiling the 64-bit version is recommended, but it is possible to compile the 32-bit version. Cross-compilation for Ubuntu and Windows Subsystem for Linux ------------------------------------------------------------ At the time of writing the Windows Subsystem for Linux installs Ubuntu Bionic 18.04. The steps below can be performed on Ubuntu (including in a VM) or WSL. The depends system will also work on other Linux distributions, however the commands for installing the toolchain will be different. First, install the general dependencies: sudo apt update sudo apt upgrade sudo apt install autoconf automake build-essential bsdmainutils curl git libboost-all-dev libevent-dev libssl-dev libtool ninja-build pkg-config python3 The cmake version packaged with Ubuntu Bionic is too old for building Building Bitcoin ABC. To install the latest version: sudo apt-get install apt-transport-https ca-certificates gnupg software-properties-common wget wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | sudo apt-key add - sudo apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main' sudo apt update sudo apt install cmake A host toolchain (`build-essential`) is necessary because some dependency packages (such as `protobuf`) need to build host utilities that are used in the build process. See also: [dependencies.md](dependencies.md). ## Building for 64-bit Windows The first step is to install the mingw-w64 cross-compilation tool chain. Due to different Ubuntu packages for each distribution and problems with the Xenial packages the steps for each are different. Common steps to install mingw32 cross compiler tool chain: sudo apt install g++-mingw-w64-x86-64 Ubuntu Xenial 16.04 and Windows Subsystem for Linux [1](#footnote1),[2](#footnote2): sudo apt install software-properties-common sudo add-apt-repository "deb http://archive.ubuntu.com/ubuntu artful universe" sudo apt update sudo apt upgrade sudo update-alternatives --config x86_64-w64-mingw32-g++ # Set the default mingw32 g++ compiler option to posix. sudo update-alternatives --config x86_64-w64-mingw32-gcc # Set the default mingw32 gcc compiler option to posix. Ubuntu Artful 17.10 [2](#footnote2) and later, including Ubuntu Bionic on WSL: sudo update-alternatives --config x86_64-w64-mingw32-g++ # Set the default mingw32 g++ compiler option to posix. sudo update-alternatives --config x86_64-w64-mingw32-gcc # Set the default mingw32 gcc compiler option to posix. Once the toolchain is installed the build steps are common: Note that for WSL the Bitcoin ABC source path MUST be somewhere in the default mount file system, for example /usr/src/bitcoin-abc, AND not under /mnt/d/. This means you cannot use a directory that is located directly on the host Windows file system to perform the build. Acquire the source in the usual way: git clone https://github.com/Bitcoin-ABC/bitcoin-abc.git Once the source code is ready the build steps are below: PATH=$(echo "$PATH" | sed -e 's/:\/mnt.*//g') # strip out problematic Windows %PATH% imported var cd depends make build-win64 cd .. mkdir build cd build cmake -GNinja .. -DCMAKE_TOOLCHAIN_FILE=../cmake/platforms/Win64.cmake -DBUILD_BITCOIN_SEEDER=OFF # seeder not supported in Windows yet ninja ## Depends system For further documentation on the depends system see [README.md](../depends/README.md) in the depends directory. Installation ------------- After building using the Windows subsystem it can be useful to copy the compiled executables to a directory on the Windows drive in the same directory structure as they appear in the release `.zip` archive. This can be done in the following way. This will install to `c:\workspace\bitcoin-abc`, for example: cmake -GNinja .. -DCMAKE_TOOLCHAIN_FILE=../cmake/platforms/Win64.cmake -DBUILD_BITCOIN_SEEDER=OFF -DCMAKE_INSTALL_PREFIX=/mnt/c/workspace/bitcoin-abc ninja install Footnotes --------- 1: There is currently a bug in the 64 bit Mingw-w64 cross compiler packaged for WSL/Ubuntu Xenial 16.04 that causes two of the bitcoin executables to crash shortly after start up. The bug is related to the -fstack-protector-all g++ compiler flag which is used to mitigate buffer overflows. Installing the Mingw-w64 packages from the Ubuntu 17.10 distribution solves the issue, however, this is not an officially supported approach and it's only recommended if you are prepared to reinstall WSL/Ubuntu should something break. 2: Starting from Ubuntu Xenial 16.04, the Mingw-w64 packages install two different compiler options to allow a choice between either posix or win32 threads. The default option is win32 threads which is the more efficient since it will result in binary code that links directly with the Windows kernel32.lib. Unfortunately, the headers required to support win32 threads conflict with some of the classes in the C++11 standard library, in particular std::mutex. It's not possible to build the Bitcoin ABC code using the win32 version of the Mingw-w64 cross compilers (at least not without modifying headers in the Bitcoin ABC source code).