diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7bf8eb0701..15298fc586 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,227 +1,227 @@ Contributing to Bitcoin ABC =========================== The Bitcoin ABC project welcomes contributors! This guide is intended to help developers contribute effectively to Bitcoin ABC. Communicating with Developers ----------------------------- To get in contact with ABC developers, we monitor a telegram supergroup. The intent of this group is specifically to facilitate development of Bitcoin-ABC, and to welcome people who wish to participate. https://t.me/joinchat/HCYr50mxRWjA2uLqii-psw Acceptable use of this supergroup includes the following: * Introducing yourself to other ABC developers. * Getting help with your development environment. * Discussing how to complete a patch. It is not for: * Market discussion * Non-constructive criticism Bitcoin ABC Development Philosophy ---------------------------------- Bitcoin ABC aims for fast iteration and continuous integration. This means that there should be quick turnaround for patches to be proposed, reviewed, and committed. Changes should not sit in a queue for long. Here are some tips to help keep the development working as intended. These -are guidelines for the normal and expected development process. Developers -can use their judgement to deviate from these guidelines when they have a +are guidelines for the normal and expected development process. Developers +can use their judgement to deviate from these guidelines when they have a good reason to do so. - Keep each change small and self-contained. - Reach out for a 1-on-1 review so things move quickly. - Land the Diff quickly after it is accepted. - Don't amend changes after the Diff accepted, new Diff for another fix. - Review Diffs from other developers as quickly as possible. - Large changes should be broken into logical chunks that are easy to review, and keep the code in a functional state. - Do not mix moving stuff around with changing stuff. Do changes with renames on their own. - Sometimes you want to replace one subsystem by another implementation, in which case it is not possible to do things incrementally. In such cases, you keep both implementations in the codebase for a while, as described [here](https://www.gamasutra.com/view/news/128325/Opinion_Parallel_Implementations.php) - There are no "development" branches, all Diffs apply to the master branch, and should always improve it (no regressions). - Don't break the build, it is important to keep master green as much as possible. If a Diff is landed, and breaks the build, fix it quickly. If it cannot be fixed quickly, it should be reverted, and re-applied later when it no longer breaks the build. -- As soon as you see a bug, you fix it. Do not continue on. Fixing the bug becomes the +- As soon as you see a bug, you fix it. Do not continue on. Fixing the bug becomes the top priority, more important than completing other tasks. - Automate as much as possible, and spend time on things only humans can do. Here are some handy links for development practices aligned with Bitcoin ABC: - [Developer Notes](doc/developer-notes.md) - [Statement of Bitcoin ABC Values and Visions](https://www.yours.org/content/bitcoin-abc---our-values-and-vision-a282afaade7c) -- [How to Do Code Reviews Like a Human - Part 1](https://mtlynch.io/human-code-reviews-1/) +- [How to Do Code Reviews Like a Human - Part 1](https://mtlynch.io/human-code-reviews-1/) - [How to Do Code Reviews Like a Human - Part 2](https://mtlynch.io/human-code-reviews-2/) - [Large Diffs Are Hurting Your Ability To Ship](https://medium.com/@kurtisnusbaum/large-diffs-are-hurting-your-ability-to-ship-e0b2b41e8acf) - [Stacked Diffs: Keeping Phabricator Diffs Small](https://medium.com/@kurtisnusbaum/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6) - [Parallel Implementations](https://www.gamasutra.com/view/news/128325/Opinion_Parallel_Implementations.php) - [The Pragmatic Programmer: From Journeyman to Master](https://www.amazon.com/Pragmatic-Programmer-Journeyman-Master/dp/020161622X) - [Advantages of monolithic version control](https://danluu.com/monorepo/) - [The importance of fixing bugs immediately](https://youtu.be/E2MIpi8pIvY?t=16m0s) - [Slow Deployment Causes Meetings](https://www.facebook.com/notes/kent-beck/slow-deployment-causes-meetings/1055427371156793/) - [Good Work, Great Work, and Right Work](https://forum.dlang.org/post/q7u6g1$94p$1@digitalmars.com) - [Accelerate: The Science of Lean Software and DevOps](https://www.amazon.com/Accelerate-Software-Performing-Technology-Organizations/dp/1942788339) Getting set up with the Bitcoin ABC Repository ---------------------------------------------- 1. Create an account at https://reviews.bitcoinabc.org/ 2. Install Git and Arcanist on your machine Git documentation can be found at: https://git-scm.com/ Arcanist documentation can be found at: https://secure.phabricator.com/book/phabricator/article/arcanist_quick_start/ And: https://secure.phabricator.com/book/phabricator/article/arcanist/ To install these packages on Debian or Ubuntu, type: `sudo apt-get install git arcanist` 3. If you do not already have an SSH key set up, follow these steps: Type: `ssh-keygen -t rsa -b 4096 -C "your_email@example.com"` Enter a file in which to save the key (/home/*username*/.ssh/id_rsa): [Press enter] 4. Upload your SSH public key to reviews.bitcoinabc.org - Go to: `https://reviews.bitcoinabc.org/settings/user/*username*/page/ssh/` - Under "SSH Key Actions", Select "Upload Public Key" Paste contents from: `/home/*username*/.ssh/id_rsa.pub` 5. Clone the repository and install Arcanist certificate: ``` git clone ssh://vcs@reviews.bitcoinabc.org:2221/source/bitcoin-abc.git cd bitcoin-abc arc install-certificate ``` Note: Arcanist tooling will tend to fail if your remote origin is set to something other than the above. A common mistake is to clone from Github and then forget to update your remotes. Follow instructions provided by `arc install-certificate` to provide your API token. 6. Code formatting tools During submission of patches, arcanist will automatically run `arc lint` to enforce Bitcoin ABC code formatting standards, and often suggests changes. If code formatting tools do not install automatically on your system, you will have to install clang-format-8, clang-tidy (version >=8), autopep8, flake8, phpcs and shellcheck. To install clang-format-8 and clang-tidy on Ubuntu (>= 18.04+updates) or Debian (>= 10): ``` sudo apt-get install clang-format-8 clang-tidy-8 clang-tools-8 ``` If not available in the distribution, `clang-format-8` and `clang-tidy` can be installed from https://releases.llvm.org/download.html or https://apt.llvm.org. For example, for macOS: ``` curl http://releases.llvm.org/8.0.0/clang+llvm-8.0.0-x86_64-apple-darwin.tar.xz | tar -xJv ln -s $PWD/clang+llvm-8.0.0-x86_64-apple-darwin/bin/clang-format /usr/local/bin/clang-format ln -s $PWD/clang+llvm-8.0.0-x86_64-apple-darwin/bin/clang-tidy /usr/local/bin/clang-tidy ``` To install autopep8, flake8 and phpcs on Ubuntu: ``` sudo apt-get install python-autopep8 flake8 php-codesniffer shellcheck ``` Working with The Bitcoin ABC Repository --------------------------------------- A typical workflow would be: - Create a topic branch in Git for your changes git checkout -b 'my-topic-branch' - Make your changes, and commit them git commit -a -m 'my-commit' - Create a differential with Arcanist arc diff You should add suggested reviewers and a test plan to the commit message. Note that Arcanist is set up to look only at the most-recent commit message, So all you changes for this Diff should be in one Git commit. - For large changes, break them into several Diffs, as described in this [guide](https://medium.com/@kurtisnusbaum/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6). You can also include "Depends on Dxxx" in the Arcanist message to indicate dependence on other Diffs. - Log into Phabricator to see review and feedback. - Make changes as suggested by the reviewers. You can simply edit the files with my-topic-branch checked out, and then type `arc diff`. Arcanist will give you the option to add uncommited changes. Or, alternatively, you can commit the changes using `git commit -a --am` to add them to the last commit, or squash multiple commits by typing `git rebase -i master`. If you squash, make sure the commit message has the information needed for arcanist (such as the Diff number, reviewers, etc.). - Update your Diff by typing `arc diff` again. - When reviewers approve your Diff, it should be listed as "ready to Land" in Phabricator. When you want to commit your diff to the repository, check out type my-topic-branch in git, then type `arc land`. You have now successfully committed a change to the Bitcoin ABC repository. - When reviewing a Diff, apply the changeset on your local by using `arc patch D{NNNN}` - You will likely be re-writing git histories multiple times, which causes timestamp changes that require re-building a significant number of files. It's highly recommended to install `ccache` (re-run cmake if you install it later), as this will help cut your re-build times from several minutes to under a minute, in many cases. What to work on --------------- If you are looking for a useful task to contribute to the project, a good place to start is the list of tasks at https://reviews.bitcoinabc.org/maniphest/ You could also try [backporting](doc/backporting.md) some code from Bitcoin Core. Copyright --------- By contributing to this repository, you agree to license your work under the MIT license unless specified otherwise in `contrib/debian/copyright` or at the top of the file itself. Any work contributed where you are not the original author must contain its license header with the original author(s) and source. Disclosure Policy ----------------- See [DISCLOSURE_POLICY](DISCLOSURE_POLICY.md). diff --git a/contrib/README.md b/contrib/README.md index 825658b95e..cf75c5359b 100644 --- a/contrib/README.md +++ b/contrib/README.md @@ -1,47 +1,47 @@ Repository Tools --------------------- ### [Developer tools](/contrib/devtools) ### Specific tools for developers working on this repository. Contains the script `github-merge.py` for merging GitHub pull requests securely and signing them using GPG. ### [Verify-Commits](/contrib/verify-commits) ### Tool to verify that every merge commit was signed by a developer using the above `github-merge.py` script. ### [Linearize](/contrib/linearize) ### Construct a linear, no-fork, best version of the blockchain. ### [Qos](/contrib/qos) ### A Linux bash script that will set up traffic control (tc) to limit the outgoing bandwidth for connections to the Bitcoin network. This means one can have an always-on bitcoind instance running, and another local bitcoind/bitcoin-qt instance which connects to this node and receives blocks from it. ### [Seeds](/contrib/seeds) ### Utility to generate the pnSeed[] array that is compiled into the client. Build Tools and Keys --------------------- ### [Debian](/contrib/debian) ### Contains files used to package bitcoind/bitcoin-qt for Debian-based Linux systems. If you compile bitcoind/bitcoin-qt yourself, there are some useful files here. ### [Gitian-descriptors](/contrib/gitian-descriptors) ### Notes on getting Gitian builds up and running using KVM. ### [Gitian-keys](/contrib/gitian-keys) PGP keys used for signing Bitcoin Core [Gitian release](/doc/release-process.md) results. ### [MacDeploy](/contrib/macdeploy) ### -Scripts and notes for Mac builds. +Scripts and notes for Mac builds. ### [Gitian-build](/contrib/gitian-build.sh) ### Script for running full Gitian builds. -Test and Verify Tools +Test and Verify Tools --------------------- ### [TestGen](/contrib/testgen) ### Utilities to generate test vectors for the data-driven Bitcoin tests. ### [Verify Binaries](/contrib/verifybinaries) ### This script attempts to download and verify the signature file SHA256SUMS.asc from bitcoin.org. diff --git a/depends/description.md b/depends/description.md index 9fc7093be4..0a6f2e6442 100644 --- a/depends/description.md +++ b/depends/description.md @@ -1,53 +1,53 @@ -This is a system of building and caching dependencies necessary for building Bitcoin. +This is a system of building and caching dependencies necessary for building Bitcoin. There are several features that make it different from most similar systems: ### It is designed to be builder and host agnostic In theory, binaries for any target OS/architecture can be created, from a builder running any OS/architecture. In practice, build-side tools must be specified when the defaults don't fit, and packages must be amended to work on new hosts. For now, a build architecture of x86_64 is assumed, either on Linux or macOS. ### No reliance on timestamps File presence is used to determine what needs to be built. This makes the results distributable and easily digestable by automated builders. ### Each build only has its specified dependencies available at build-time. For each build, the sysroot is wiped and the (recursive) dependencies are installed. This makes each build deterministic, since there will never be any unknown files available to cause side-effects. ### Each package is cached and only rebuilt as needed. Before building, a unique build-id is generated for each package. This id consists of a hash of all files used to build the package (Makefiles, packages, etc), and as well as a hash of the same data for each recursive dependency. If any portion of a package's build recipe changes, it will be rebuilt as well as -any other package that depends on it. If any of the main makefiles (Makefile, +any other package that depends on it. If any of the main makefiles (Makefile, funcs.mk, etc) are changed, all packages will be rebuilt. After building, the results are cached into a tarball that can be re-used and distributed. ### Package build results are (relatively) deterministic. Each package is configured and patched so that it will yield the same build-results with each consequent build, within a reasonable set of constraints. Some things like timestamp insertion are unavoidable, and are beyond the scope of this system. Additionally, the toolchain itself must be capable of deterministic results. When revisions are properly bumped, a cached build should represent an exact single payload. ### Sources are fetched and verified automatically Each package must define its source location and checksum. The build will fail if the fetched source does not match. Sources may be pre-seeded and/or cached as desired. ### Self-cleaning Build and staging dirs are wiped after use, and any previous version of a cached result is removed following a successful build. Automated builders should be able to build each revision and store the results with no further intervention. diff --git a/depends/packages.md b/depends/packages.md index 96093c70da..696d07e4ab 100644 --- a/depends/packages.md +++ b/depends/packages.md @@ -1,199 +1,199 @@ Each recipe consists of 3 main parts: defining identifiers, setting build variables, and defining build commands. The package "mylib" will be used here as an example General tips: - mylib_foo is written as $(package)_foo in order to make recipes more similar. - Secondary dependency packages relative to the bitcoin binaries/libraries (i.e. those not in `ALLOWED_LIBRARIES` in `contrib/devtools/symbol-check.py`) don't need to be shared and should be built statically whenever possible. See [below](#secondary-dependencies) for more details. ## Identifiers Each package is required to define at least these variables: $(package)_version: Version of the upstream library or program. If there is no version, a placeholder such as 1.0 can be used. $(package)_download_path: Location of the upstream source, without the file-name. Usually http, https or ftp. Secure transmission options like https should be preferred if available. $(package)_file_name: The upstream source filename available at the download path. $(package)_sha256_hash: The sha256 hash of the upstream file These variables are optional: $(package)_build_subdir: cd to this dir before running configure/build/stage commands. - + $(package)_download_file: The file-name of the upstream source if it differs from how it should be stored locally. This can be used to avoid storing file-names with strange characters. - + $(package)_dependencies: Names of any other packages that this one depends on. - + $(package)_patches: Filenames of any patches needed to build the package $(package)_extra_sources: Any extra files that will be fetched via $(package)_fetch_cmds. These are specified so that they can be fetched and verified via 'make download'. ## Build Variables: After defining the main identifiers, build variables may be added or customized before running the build commands. They should be added to a function called $(package)_set_vars. For example: define $(package)_set_vars ... endef Most variables can be prefixed with the host, architecture, or both, to make the modifications specific to that case. For example: Universal: $(package)_cc=gcc Linux only: $(package)_linux_cc=gcc x86_64 only: $(package)_x86_64_cc = gcc x86_64 linux only: $(package)_x86_64_linux_cc = gcc These variables may be set to override or append their default values. $(package)_cc $(package)_cxx $(package)_objc $(package)_objcxx $(package)_ar $(package)_ranlib $(package)_libtool $(package)_nm $(package)_cflags $(package)_cxxflags $(package)_ldflags $(package)_cppflags $(package)_config_env $(package)_build_env $(package)_stage_env $(package)_build_opts $(package)_config_opts The *_env variables are used to add environment variables to the respective commands. Many variables respect a debug/release suffix as well, in order to use them for only the appropriate build config. For example: $(package)_cflags_release = -O3 $(package)_cflags_i686_debug = -g $(package)_config_opts_release = --disable-debug These will be used in addition to the options that do not specify debug/release. All builds are considered to be release unless DEBUG=1 is set by the user. Other variables may be defined as needed. ## Build commands: For each build, a unique build dir and staging dir are created. For example, `work/build/mylib/1.0-1adac830f6e` and `work/staging/mylib/1.0-1adac830f6e`. The following build commands are available for each recipe: $(package)_fetch_cmds: Runs from: build dir Fetch the source file. If undefined, it will be fetched and verified against its hash. $(package)_extract_cmds: Runs from: build dir Verify the source file against its hash and extract it. If undefined, the source is assumed to be a tarball. $(package)_preprocess_cmds: Runs from: build dir/$(package)_build_subdir Preprocess the source as necessary. If undefined, does nothing. $(package)_config_cmds: Runs from: build dir/$(package)_build_subdir Configure the source. If undefined, does nothing. $(package)_build_cmds: Runs from: build dir/$(package)_build_subdir Build the source. If undefined, does nothing. $(package)_stage_cmds: Runs from: build dir/$(package)_build_subdir Stage the build results. If undefined, does nothing. The following variables are available for each recipe: - + $(1)_staging_dir: package's destination sysroot path $(1)_staging_prefix_dir: prefix path inside of the package's staging dir $(1)_extract_dir: path to the package's extracted sources $(1)_build_dir: path where configure/build/stage commands will be run $(1)_patch_dir: path where the package's patches (if any) are found Notes on build commands: For packages built with autotools, $($(package)_autoconf) can be used in the configure step to (usually) correctly configure automatically. Any $($(package)_config_opts) will be appended. Most autotools projects can be properly staged using: $(MAKE) DESTDIR=$($(package)_staging_dir) install ## Build outputs: In general, the output of a depends package should not contain any libtool archives. Instead, the package should output `.pc` (`pkg-config`) files where possible. From the [Gentoo Wiki entry](https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Handling_Libtool_Archives): > Libtool pulls in all direct and indirect dependencies into the .la files it > creates. This leads to massive overlinking, which is toxic to the Gentoo > ecosystem, as it leads to a massive number of unnecessary rebuilds. ## Secondary dependencies: Secondary dependency packages relative to the bitcoin binaries/libraries (i.e. those not in `ALLOWED_LIBRARIES` in `contrib/devtools/symbol-check.py`) don't need to be shared and should be built statically whenever possible. This improves general build reliability as illustrated by the following example: When linking an executable against a shared library `libprimary` that has its own shared dependency `libsecondary`, we may need to specify the path to `libsecondary` on the link command using the `-rpath/-rpath-link` options, it is not sufficient to just say `libprimary`. For us, it's much easier to just link a static `libsecondary` into a shared `libprimary`. Especially because in our case, we are linking against a dummy `libprimary` anyway that we'll throw away. We don't care if the end-user has a static or dynamic `libsecondary`, that's not our concern. With a static `libsecondary`, when we need to link `libprimary` into our executable, there's no dependency chain to worry about as `libprimary` has all the symbols. ## Build targets: To build an individual package (useful for debugging), the following build targets are available. make ${package} make ${package}_fetched make ${package}_extracted make ${package}_preprocessed make ${package}_configured make ${package}_built make ${package}_staged make ${package}_postprocessed make ${package}_cached make ${package}_cached_checksum diff --git a/doc/backporting.md b/doc/backporting.md index 379db147a8..d25f4ddc57 100644 --- a/doc/backporting.md +++ b/doc/backporting.md @@ -1,91 +1,91 @@ BACKPORTING =========== The official Bitcoin-ABC guide to backporting code from Bitcoin Core. When searching for items to backport, especially be on the lookout for bug fixes, code cleanup, and important refactors, as these help improve Bitcoin ABC despite consensus-level differences between Bitcoin Cash and Bitcoin Core. Identifying commits ------------------- 1. Check out a copy of a Satoshi Bitcoin client somewhere on your machine. 2. Identify the subsystem you'd like to work on. 3. Tag the fork commit as `fork-commit`. Bitcoin-ABC was forked from Bitcoin Core at commit `964a185cc83af34587194a6ecda3ed9cf6b49263`. `> git tag -a fork-commit 964a185 -m 'Where the fun started'` 4. `git log --topo-order --graph fork-commit..HEAD -- ` 5. Find the commit, and if applicable, the associated merge commit that are worth backporting. The merge commit may indicate that there were other commits associated with this change that you will need to backport. E.g: ``` commit d083bd9b9c5249f21b8b7e4abd7aee48a25806b1 Merge: b3eb0d648 279fde58e Author: Wladimir J. van der Laan Date: Wed Jun 21 14:26:10 2017 +0200 Merge #10533: [tests] Use cookie auth instead of rpcuser and rpcpassword 279fde5 Check for rpcuser/rpcpassword first then for cookie (Andrew Chow) 3ec5ad8 Add test for rpcuser/rpcpassword (Andrew Chow) c53c983 Replace cookie auth in tests (Andrew Chow) Tree-SHA512: 21efb84c87080a895cac8a7fe4766738c34eebe9686c7d10af1bf91ed4ae422e2d5dbbebffd00d34744eb6bb2d0195ea3aca86deebf085bbdeeb1d8b474241ed ``` If you saw that commit `c53c983` was a good idea to backport, this merge commit would indicate that there are two other commits that are associated with this PR and likely to be needed. When trying to find a patch worth backporting, it's generally a good idea to backport significant refactors or bugfixes. This will help clean up the code in the ABC repository, fix bugs, and make future backports significantly easier. Backports are easiest done in topological order of commits. Backporting ----------- Before you begin backporting commits, you will need to add an additional remote to your Bitcoin-ABC repo. For Bitcoin Core, this repository would be added as: ``` git remote add core git@github.com:bitcoin/bitcoin.git git fetch core ``` (Assuming you have github ssh auth setup. The second command is required to obtain refs for cherry-picking.) Once you have identified your commit, or commits on question to backport you have two choices: 1. Backport each diff individually 2. Squash the commits together for backporting. -In either case, you will find there are likely merge conflicts. +In either case, you will find there are likely merge conflicts. Backport each diff individually ------------------------------- 1. `git checkout -b ` 2. `git cherry-pick ` 3. Run `git status` and fix conflicts. 4. `git add -u && git cherry-pick --continue` 5. Run `git show` side-by-side with `git show ` and verify that the changes are reasonable. If there are additional changes caused by the merge conflict 6. Run the build, and the rpc test suite and verify completion. 7. `arc diff` and at the bottom of the summary note: "Backport of Bitcoin Core PR# ". the PR# can be obtained by searching on github for the commit you are backporting. If you are backporting a commit which depends on another commit from the same PR, note that you are backporting `Part 1 of X`. Additionally, if it is the second or more of a series of commits in a backport, note which other phabricator revision this diff depends on by typing `Depends on DXXXX` at the bottom of the summary. Squash the commits together for backporting ------------------------------------------- 1. `git checkout -b PRXXXXX` 2. Perform steps 2 through 6 from "Backporting each diff individually" in repetition for each diff in a Bitcoin Core pull request. 3. `git rebase -i origin/master` and squash the commits together. 4. `arc diff` and at the bottom of the summary note: "Backport of Bitcoin Core PR# ". The title of the diff should be: `[Backport] ` diff --git a/doc/build-osx.md b/doc/build-osx.md index 508324f9ee..4da4082569 100644 --- a/doc/build-osx.md +++ b/doc/build-osx.md @@ -1,99 +1,99 @@ macOS Build Instructions and Notes ==================================== The commands in this guide should be executed in a Terminal application. The built-in one is located in `/Applications/Utilities/Terminal.app`. Preparation ----------- 1. Install Xcode from the app store if you don't have it already (it's a dependency for qt5) 2. Install the macOS command line tools: `xcode-select --install` When the popup appears, click `Install`. 3. Install [Homebrew](https://brew.sh). Dependencies ---------------------- Install dependencies: brew install berkeley-db boost cmake libevent miniupnpc ninja openssl protobuf python qrencode qt zeromq See [dependencies.md](dependencies.md) for a complete overview. If you want to build the disk image with `ninja osx-dmg` (.dmg / optional), you need RSVG: brew install librsvg Build Bitcoin ABC ----------------- Before you start building, please make sure that your compiler supports C++14. 1. Clone the Bitcoin ABC source code and cd into `bitcoin-abc` git clone https://github.com/Bitcoin-ABC/bitcoin-abc.git cd bitcoin-abc 2. Build Bitcoin ABC: Configure and build the headless Bitcoin ABC binaries as well as the GUI. You can disable the GUI build by passing `-DBUILD_BITCOIN_QT=OFF` to cmake. - It is recommended to create a build directory to build out-of-tree. + It is recommended to create a build directory to build out-of-tree. mkdir build cd build cmake -GNinja .. ninja 3. It is recommended to build and run the unit tests: ninja check 4. You can also create a .dmg that contains the .app bundle (optional): ninja osx-dmg Disable-wallet mode -------------------- When the intention is to run only a P2P node without a wallet, Bitcoin ABC may be compiled in disable-wallet mode with: cmake -GNinja .. -DBUILD_BITCOIN_WALLET=OFF Mining is also possible in disable-wallet mode using the `getblocktemplate` RPC call. Running ------- Bitcoin ABC is now available at `./src/bitcoind` Before running, it's recommended that you create an RPC configuration file: echo -e "rpcuser=bitcoinrpc\nrpcpassword=$(xxd -l 16 -p /dev/urandom)" > "/Users/${USER}/Library/Application Support/Bitcoin/bitcoin.conf" chmod 600 "/Users/${USER}/Library/Application Support/Bitcoin/bitcoin.conf" The first time you run bitcoind, it will start downloading the blockchain. This process could take many hours, or even days on slower than average systems. You can monitor the download process by looking at the debug.log file: tail -f $HOME/Library/Application\ Support/Bitcoin/debug.log Other commands: ------- ./src/bitcoind -daemon # Starts the bitcoin daemon. ./src/bitcoin-cli --help # Outputs a list of command-line options. ./src/bitcoin-cli help # Outputs a list of RPC commands when the daemon is running. Notes ----- * Building with downloaded Qt binaries is not officially supported. See the notes in [#7714](https://github.com/bitcoin/bitcoin/issues/7714) diff --git a/doc/build-windows.md b/doc/build-windows.md index f6e52da693..93af680f2f 100644 --- a/doc/build-windows.md +++ b/doc/build-windows.md @@ -1,156 +1,156 @@ WINDOWS BUILD NOTES ==================== Below are some notes on how to build Bitcoin ABC for Windows. The options known to work for building Bitcoin ABC on Windows are: * On Linux, using the [Mingw-w64](https://mingw-w64.org/doku.php) cross compiler tool chain. Debian Buster is recommended and is the platform used to build the Bitcoin ABC Windows release binaries. * On Windows, using [Windows Subsystem for Linux (WSL)](https://msdn.microsoft.com/commandline/wsl/about) and the Mingw-w64 cross compiler tool chain. Other options which may work, but which have not been extensively tested are (please contribute instructions): * On Windows, using a POSIX compatibility layer application such as [cygwin](http://www.cygwin.com/) or [msys2](http://www.msys2.org/). * On Windows, using a native compiler tool chain such as [Visual Studio](https://www.visualstudio.com). In any case please make sure that the compiler supports C++14. Installing Windows Subsystem for Linux --------------------------------------- With Windows 10, Microsoft has released a new feature named the [Windows Subsystem for Linux (WSL)](https://msdn.microsoft.com/commandline/wsl/about). This feature allows you to run a bash shell directly on Windows in an Ubuntu-based environment. Within this environment you can cross compile for Windows without the need for a separate Linux VM or server. Note that while WSL can be installed with other Linux variants, such as OpenSUSE, the following instructions have only been tested with Ubuntu Bionic. This feature is not supported in versions of Windows prior to Windows 10 or on Windows Server SKUs. In addition, it is available [only for 64-bit versions of Windows](https://msdn.microsoft.com/en-us/commandline/wsl/install_guide). Full instructions to install WSL are available on the above link. To install WSL on Windows 10 with Fall Creators Update installed (version >= 16215.0) do the following: 1. Enable the Windows Subsystem for Linux feature * Open the Windows Features dialog (`OptionalFeatures.exe`) * Enable 'Windows Subsystem for Linux' * Click 'OK' and restart if necessary 2. Install Ubuntu * Open Microsoft Store and search for Ubuntu or use [this link](https://www.microsoft.com/store/productId/9NBLGGH4MSV6) * Click Install 3. Complete Installation * Open a cmd prompt and type "Ubuntu" * Create a new UNIX user account (this is a separate account from your Windows account) After the bash shell is active, you can follow the instructions below, starting with the "Cross-compilation" section. Compiling the 64-bit version is recommended, but it is possible to compile the 32-bit version. Cross-compilation for Ubuntu and Windows Subsystem for Linux ------------------------------------------------------------ -At the time of writing the Windows Subsystem for Linux installs Ubuntu Bionic 18.04. +At the time of writing the Windows Subsystem for Linux installs Ubuntu Bionic 18.04. The steps below can be performed on Ubuntu (including in a VM) or WSL. The depends system will also work on other Linux distributions, however the commands for installing the toolchain will be different. First, install the general dependencies: sudo apt update sudo apt upgrade sudo apt install autoconf automake build-essential bsdmainutils curl git libboost-all-dev libevent-dec libssl-dev libtool ninja-build pkg-config python3 The cmake version packaged with Ubuntu Bionic is too old for building Building Bitcoin ABC. To install the latest version: sudo apt-get install apt-transport-https ca-certificates gnupg software-properties-common wget wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | sudo apt-key add - sudo apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main' sudo apt update sudo apt install cmake A host toolchain (`build-essential`) is necessary because some dependency packages (such as `protobuf`) need to build host utilities that are used in the build process. See also: [dependencies.md](dependencies.md). ## Building for 64-bit Windows The first step is to install the mingw-w64 cross-compilation tool chain. Due to different Ubuntu packages for each distribution and problems with the Xenial packages the steps for each are different. Common steps to install mingw32 cross compiler tool chain: sudo apt install g++-mingw-w64-x86-64 Ubuntu Xenial 16.04 and Windows Subsystem for Linux [1](#footnote1),[2](#footnote2): sudo apt install software-properties-common sudo add-apt-repository "deb http://archive.ubuntu.com/ubuntu artful universe" sudo apt update sudo apt upgrade sudo update-alternatives --config x86_64-w64-mingw32-g++ # Set the default mingw32 g++ compiler option to posix. sudo update-alternatives --config x86_64-w64-mingw32-gcc # Set the default mingw32 gcc compiler option to posix. Ubuntu Artful 17.10 [2](#footnote2) and later, including Ubuntu Bionic on WSL: sudo update-alternatives --config x86_64-w64-mingw32-g++ # Set the default mingw32 g++ compiler option to posix. sudo update-alternatives --config x86_64-w64-mingw32-gcc # Set the default mingw32 gcc compiler option to posix. Once the toolchain is installed the build steps are common: Note that for WSL the Bitcoin ABC source path MUST be somewhere in the default mount file system, for example /usr/src/bitcoin-abc, AND not under /mnt/d/. This means you cannot use a directory that is located directly on the host Windows file system to perform the build. Acquire the source in the usual way: git clone https://github.com/Bitcoin-ABC/bitcoin-abc.git Once the source code is ready the build steps are below: PATH=$(echo "$PATH" | sed -e 's/:\/mnt.*//g') # strip out problematic Windows %PATH% imported var cd depends make build-win64 cd .. mkdir build cd build cmake -GNinja .. -DCMAKE_TOOLCHAIN_FILE=../cmake/platforms/Win64.cmake -DBUILD_BITCOIN_SEEDER=OFF # seeder not supported in Windows yet ninja ## Depends system For further documentation on the depends system see [README.md](../depends/README.md) in the depends directory. Installation ------------- After building using the Windows subsystem it can be useful to copy the compiled executables to a directory on the windows drive in the same directory structure as they appear in the release `.zip` archive. This can be done in the following way. This will install to `c:\workspace\bitcoin-abc`, for example: cmake -GNinja .. -DCMAKE_TOOLCHAIN_FILE=../cmake/platforms/Win64.cmake -DBUILD_BITCOIN_SEEDER=OFF -DCMAKE_INSTALL_PREFIX=/mnt/c/workspace/bitcoin-abc ninja install Footnotes --------- 1: There is currently a bug in the 64 bit Mingw-w64 cross compiler packaged for WSL/Ubuntu Xenial 16.04 that causes two of the bitcoin executables to crash shortly after start up. The bug is related to the -fstack-protector-all g++ compiler flag which is used to mitigate buffer overflows. Installing the Mingw-w64 packages from the Ubuntu 17.10 distribution solves the issue, however, this is not an officially supported approach and it's only recommended if you are prepared to reinstall WSL/Ubuntu should something break. 2: Starting from Ubuntu Xenial 16.04, the Mingw-w64 packages install two different compiler options to allow a choice between either posix or win32 threads. The default option is win32 threads which is the more efficient since it will result in binary code that links directly with the Windows kernel32.lib. Unfortunately, the headers required to support win32 threads conflict with some of the classes in the C++11 standard library, in particular std::mutex. It's not possible to build the Bitcoin ABC code using the win32 version of the Mingw-w64 cross compilers (at least not without modifying headers in the Bitcoin ABC source code). diff --git a/doc/developer-notes.md b/doc/developer-notes.md index 011f81f23c..1897b0505c 100644 --- a/doc/developer-notes.md +++ b/doc/developer-notes.md @@ -1,932 +1,932 @@ Developer Notes =============== **Table of Contents** - [Developer Notes](#developer-notes) - [Coding Style](#coding-style) - [Doxygen comments](#doxygen-comments) - [Development tips and tricks](#development-tips-and-tricks) - [Compiling for debugging](#compiling-for-debugging) - [Compiling for gprof profiling](#compiling-for-gprof-profiling) - [debug.log](#debuglog) - [Writing tests](#writing-tests) - [Writing script integration tests](#writing-script-integration-tests) - [Testnet and Regtest modes](#testnet-and-regtest-modes) - [DEBUG_LOCKORDER](#debug_lockorder) - [Valgrind suppressions file](#valgrind-suppressions-file) - [Compiling for test coverage](#compiling-for-test-coverage) - [Sanitizers](#sanitizers) - [Locking/mutex usage notes](#lockingmutex-usage-notes) - [Threads](#threads) - [Ignoring IDE/editor files](#ignoring-ideeditor-files) - [Development guidelines](#development-guidelines) - [Wallet](#wallet) - [General C++](#general-c) - [C++ data structures](#c-data-structures) - [Strings and formatting](#strings-and-formatting) - [Variable names](#variable-names) - [Threads and synchronization](#threads-and-synchronization) - [Scripts](#scripts) - [Shebang](#shebang) - [Source code organization](#source-code-organization) - [GUI](#gui) - [Unit tests](#unit-tests) - [Third party libraries](#third-party-libraries) - [Git and GitHub tips](#git-and-github-tips) - [RPC interface guidelines](#rpc-interface-guidelines) Coding Style --------------- Various coding styles have been used during the history of the codebase, and the result is not very consistent. However, we're now trying to converge to a single style, so please use it in new code. Old code will be converted gradually and you are encouraged to use the provided [clang-format-diff script](/contrib/devtools/README.md#clang-format-diffpy) to clean up the patch automatically before submitting a pull request. - Basic rules specified in [src/.clang-format](/src/.clang-format). - Braces on new lines for namespaces, classes, functions, methods. - Braces on the same line for everything else. - 4 space indentation (no tabs) for every block except namespaces. - No indentation for `public`/`protected`/`private` or for `namespace`. - No extra spaces inside parenthesis; don't do ( this ) - No space after function names; one space after `if`, `for` and `while`. - Always add braces for block statements (e.g. `if`, `for`, `while`). - `++i` is preferred over `i++`. - `static_assert` is preferred over `assert` where possible. Generally; compile-time checking is preferred over run-time checking. - Use CamelCase for functions/methods, and lowerCamelCase for variables. - GLOBAL_CONSTANTS should use UPPER_SNAKE_CASE. - namespaces should use lower_snake_case. - - Function names should generally start with an English command-form verb + - Function names should generally start with an English command-form verb (e.g. `ValidateTransaction`, `AddTransactionToMempool`, `ConnectBlock`) - Variable names should generally be nouns or past/future tense verbs. (e.g. `canDoThing`, `signatureOperations`, `didThing`) - Avoid using globals, remove existing globals whenever possible. - Class member variable names should be prepended with `m_` - - DO choose easily readable identifier names. + - DO choose easily readable identifier names. - DO favor readability over brevity. - DO NOT use Hungarian notation. - - DO NOT use abbreviations or contractions within identifiers. + - DO NOT use abbreviations or contractions within identifiers. - WRONG: mempool - RIGHT: MemoryPool - WRONG: ChangeDir - RIGHT: ChangeDirectory - DO NOT use obscure acronyms, DO uppercase any acronyms. - FINALLY, do not migrate existing code unless refactoring. It makes forwarding-porting from Bitcoin Core more difficult. The naming convention roughly mirrors [Microsoft Naming Conventions](https://docs.microsoft.com/en-us/dotnet/standard/design-guidelines/general-naming-conventions) C++ Coding Standards should strive to follow the [LLVM Coding Standards](https://llvm.org/docs/CodingStandards.html) Code style example: ```c++ // namespaces should be lower_snake_case namespace foo_bar_bob { /** * Class is used for doing classy things. All classes should * have a doxygen comment describing their PURPOSE. That is to say, * why they exist. Functional details can be determined from the code. * @see PerformTask() */ class Class { private: //! memberVariable's name should be lowerCamelCase, and be a noun. int m_memberVariable; public: /** * The documentation before a function or class method should follow Doxygen - * spec. The name of the function should start with an english verb which + * spec. The name of the function should start with an english verb which * indicates the intended purpose of this code. - * + * * The function name should be should be CamelCase. - * + * * @param[in] s A description * @param[in] n Another argument description * @pre Precondition for function... */ bool PerformTask(const std::string& s, int n) { // Use lowerChamelCase for local variables. bool didMore = false; // Comment summarizing the intended purpose of this section of code for (int i = 0; i < n; ++i) { if (!DidSomethingFail()) { return false; } ... if (IsSomethingElse()) { DoMore(); didMore = true; } else { DoLess(); } } return didMore; } } } // namespace foo ``` Doxygen comments ----------------- To facilitate the generation of documentation, use doxygen-compatible comment blocks for functions, methods and fields. For example, to describe a function use: ```c++ /** * ... text ... * @param[in] arg1 A description * @param[in] arg2 Another argument description * @pre Precondition for function... */ bool function(int arg1, const char *arg2) ``` A complete list of `@xxx` commands can be found at http://www.stack.nl/~dimitri/doxygen/manual/commands.html. As Doxygen recognizes the comments by the delimiters (`/**` and `*/` in this case), you don't *need* to provide any commands for a comment to be valid; just a description text is fine. To describe a class use the same construct above the class definition: ```c++ /** * Alerts are for notifying old versions if they become too obsolete and * need to upgrade. The message is displayed in the status bar. * @see GetWarnings() */ class CAlert { ``` To describe a member or variable use: ```c++ int var; //!< Detailed description after the member ``` or ```cpp //! Description before the member int var; ``` Also OK: ```c++ /// /// ... text ... /// bool function2(int arg1, const char *arg2) ``` Not OK (used plenty in the current source, but not picked up): ```c++ // // ... text ... // ``` A full list of comment syntaxes picked up by doxygen can be found at http://www.stack.nl/~dimitri/doxygen/manual/docblocks.html, but if possible use one of the above styles. To build doxygen locally to test changes to the Doxyfile or visualize your comments before landing changes: ``` # In the build directory, call: doxygen doc/Doxyfile # output goes to doc/doxygen/html/ ``` Development tips and tricks --------------------------- ### Compiling for debugging Run configure with `--enable-debug` to add additional compiler flags that produce better debugging builds. ### Compiling for gprof profiling Run configure with the `--enable-gprof` option, then make. With `cmake` and `ninja`: ``` cmake -GNinja .. -DENABLE_HARDENING=OFF -DENABLE_PROFIILING=gprof ``` ### debug.log If the code is behaving strangely, take a look in the debug.log file in the data directory; error and debugging messages are written there. The `-debug=...` command-line option controls debugging; running with just `-debug` or `-debug=1` will turn on all categories (and give you a very large debug.log file). The Qt code routes `qDebug()` output to debug.log under category "qt": run with `-debug=qt` to see it. ### Writing tests For details on unit tests, see `unit-tests.md` For details on functional tests, see `functional-tests.md` ### Writing script integration tests Script integration tests are built using `src/test/script_tests.cpp`: 1. Uncomment the line with `#define UPDATE_JSON_TESTS` 2. Add a new TestBuilder to the `script_build` test to cover your test case. 3. `make && ./src/test/test_bitcoin --run_test=script_tests` 4. Copy your newly generated test JSON from `/src/script_tests.json.gen` to `src/test/data/script_tests.json`. Please commit your TestBuilder along with your generated test JSON and cleanup the uncommented #define before code review. ### Testnet and Regtest modes Run with the `-testnet` option to run with "play bitcoins" on the test network, if you are testing multi-machine code that needs to operate across the internet. If you are testing something that can run on one machine, run with the `-regtest` option. In regression test mode, blocks can be created on-demand; see [test/functional/](/test/functional) for tests that run in `-regtest` mode. ### DEBUG_LOCKORDER Bitcoin ABC is a multi-threaded application, and deadlocks or other multi-threading bugs can be very difficult to track down. The `--enable-debug` configure option adds `-DDEBUG_LOCKORDER` to the compiler flags. This inserts run-time checks to keep track of which locks are held, and adds warnings to the debug.log file if inconsistencies are detected. ### Valgrind suppressions file Valgrind is a programming tool for memory debugging, memory leak detection, and profiling. The repo contains a Valgrind suppressions file ([`valgrind.supp`](contrib/valgrind.supp)) which includes known Valgrind warnings in our dependencies that cannot be fixed in-tree. Example use: ```shell $ valgrind --suppressions=contrib/valgrind.supp src/test/test_bitcoin $ valgrind --suppressions=contrib/valgrind.supp --leak-check=full \ --show-leak-kinds=all src/test/test_bitcoin --log_level=test_suite $ valgrind -v --leak-check=full src/bitcoind -printtoconsole ``` ### Compiling for test coverage LCOV can be used to generate a test coverage report based upon `make check` execution. LCOV must be installed on your system (e.g. the `lcov` package on Debian/Ubuntu). To enable LCOV report generation during test runs: ```shell ./configure --enable-lcov make make cov # A coverage report will now be accessible at `./test_bitcoin.coverage/index.html`. ``` ### Sanitizers Bitcoin ABC can be compiled with various "sanitizers" enabled, which add instrumentation for issues regarding things like memory safety, thread race conditions, or undefined behavior. This is controlled with the `--with-sanitizers` configure flag, which should be a comma separated list of sanitizers to enable. The sanitizer list should correspond to supported `-fsanitize=` options in your compiler. These sanitizers have runtime overhead, so they are most useful when testing changes or producing debugging builds. Some examples: ```bash # Enable both the address sanitizer and the undefined behavior sanitizer ./configure --with-sanitizers=address,undefined # Enable the thread sanitizer ./configure --with-sanitizers=thread ``` If you are compiling with GCC you will typically need to install corresponding "san" libraries to actually compile with these flags, e.g. libasan for the address sanitizer, libtsan for the thread sanitizer, and libubsan for the undefined sanitizer. If you are missing required libraries, the configure script will fail with a linker error when testing the sanitizer flags. The test suite should pass cleanly with the `thread` and `undefined` sanitizers, but there are a number of known problems when using the `address` sanitizer. The address sanitizer is known to fail in [sha256_sse4::Transform](/src/crypto/sha256_sse4.cpp) which makes it unusable unless you also use `--disable-asm` when running configure. We would like to fix sanitizer issues, so please send pull requests if you can fix any errors found by the address sanitizer (or any other sanitizer). Not all sanitizer options can be enabled at the same time, e.g. trying to build with `--with-sanitizers=address,thread` will fail in the configure script as these sanitizers are mutually incompatible. Refer to your compiler manual to learn more about these options and which sanitizers are supported by your compiler. Additional resources: * [AddressSanitizer](https://clang.llvm.org/docs/AddressSanitizer.html) * [LeakSanitizer](https://clang.llvm.org/docs/LeakSanitizer.html) * [MemorySanitizer](https://clang.llvm.org/docs/MemorySanitizer.html) * [ThreadSanitizer](https://clang.llvm.org/docs/ThreadSanitizer.html) * [UndefinedBehaviorSanitizer](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html) * [GCC Instrumentation Options](https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html) * [Google Sanitizers Wiki](https://github.com/google/sanitizers/wiki) * [Issue #12691: Enable -fsanitize flags in Travis](https://github.com/bitcoin/bitcoin/issues/12691) Locking/mutex usage notes ------------------------- The code is multi-threaded, and uses mutexes and the `LOCK` and `TRY_LOCK` macros to protect data structures. Deadlocks due to inconsistent lock ordering (thread 1 locks `cs_main` and then `cs_wallet`, while thread 2 locks them in the opposite order: result, deadlock as each waits for the other to release its lock) are a problem. Compile with `-DDEBUG_LOCKORDER` (or use `--enable-debug`) to get lock order inconsistencies reported in the debug.log file. Re-architecting the core code so there are better-defined interfaces between the various components is a goal, with any necessary locking done by the components (e.g. see the self-contained `CBasicKeyStore` class and its `cs_KeyStore` lock for example). Threads ------- - ThreadScriptCheck : Verifies block scripts. - ThreadImport : Loads blocks from blk*.dat files or bootstrap.dat. - StartNode : Starts other threads. - ThreadDNSAddressSeed : Loads addresses of peers from the DNS. - ThreadMapPort : Universal plug-and-play startup/shutdown - ThreadSocketHandler : Sends/Receives data from peers on port 8333. - ThreadOpenAddedConnections : Opens network connections to added nodes. - ThreadOpenConnections : Initiates new connections to peers. - ThreadMessageHandler : Higher-level message handling (sending and receiving). - DumpAddresses : Dumps IP addresses of nodes to peers.dat. - ThreadRPCServer : Remote procedure call handler, listens on port 8332 for connections and services them. - Shutdown : Does an orderly shutdown of everything. Ignoring IDE/editor files -------------------------- In closed-source environments in which everyone uses the same IDE it is common to add temporary files it produces to the project-wide `.gitignore` file. However, in open source software such as Bitcoin Core, where everyone uses their own editors/IDE/tools, it is less common. Only you know what files your editor produces and this may change from version to version. The canonical way to do this is thus to create your local gitignore. Add this to `~/.gitconfig`: ``` [core] excludesfile = /home/.../.gitignore_global ``` (alternatively, type the command `git config --global core.excludesfile ~/.gitignore_global` on a terminal) Then put your favourite tool's temporary filenames in that file, e.g. ``` # NetBeans nbproject/ ``` Another option is to create a per-repository excludes file `.git/info/exclude`. These are not committed but apply only to one repository. If a set of tools is used by the build system or scripts the repository (for example, lcov) it is perfectly acceptable to add its files to `.gitignore` and commit them. Development guidelines ============================ A few non-style-related recommendations for developers, as well as points to pay attention to for reviewers of Bitcoin Core code. Wallet ------- - Make sure that no crashes happen with run-time option `-disablewallet`. - *Rationale*: In RPC code that conditionally uses the wallet (such as `validateaddress`) it is easy to forget that global pointer `pwalletMain` can be NULL. See `test/functional/disablewallet.py` for functional tests exercising the API with `-disablewallet` - Include `db_cxx.h` (BerkeleyDB header) only when `ENABLE_WALLET` is set - *Rationale*: Otherwise compilation of the disable-wallet build will fail in environments without BerkeleyDB General C++ ------------- - Assertions should not have side-effects - *Rationale*: Even though the source code is set to refuse to compile with assertions disabled, having side-effects in assertions is unexpected and makes the code harder to understand - If you use the `.h`, you must link the `.cpp` - *Rationale*: Include files define the interface for the code in implementation files. Including one but not linking the other is confusing. Please avoid that. Moving functions from the `.h` to the `.cpp` should not result in build errors - Use the RAII (Resource Acquisition Is Initialization) paradigm where possible. For example by using `unique_ptr` for allocations in a function. - *Rationale*: This avoids memory and resource leaks, and ensures exception safety - Use `std::make_unique()` to construct objects owned by `unique_ptr`s - *Rationale*: `std::make_unique` is concise and ensures exception safety in complex expressions. C++ data structures -------------------- - Never use the `std::map []` syntax when reading from a map, but instead use `.find()` - *Rationale*: `[]` does an insert (of the default element) if the item doesn't exist in the map yet. This has resulted in memory leaks in the past, as well as race conditions (expecting read-read behavior). Using `[]` is fine for *writing* to a map - Do not compare an iterator from one data structure with an iterator of another data structure (even if of the same type) - *Rationale*: Behavior is undefined. In C++ parlor this means "may reformat the universe", in practice this has resulted in at least one hard-to-debug crash bug - Watch out for out-of-bounds vector access. `&vch[vch.size()]` is illegal, including `&vch[0]` for an empty vector. Use `vch.data()` and `vch.data() + vch.size()` instead. - Vector bounds checking is only enabled in debug mode. Do not rely on it - Make sure that constructors initialize all fields. If this is skipped for a good reason (i.e., optimization on the critical path), add an explicit comment about this - *Rationale*: Ensure determinism by avoiding accidental use of uninitialized values. Also, static analyzers balk about this. - By default, declare single-argument constructors `explicit`. - *Rationale*: This is a precaution to avoid unintended conversions that might arise when single-argument constructors are used as implicit conversion functions. - Use explicitly signed or unsigned `char`s, or even better `uint8_t` and `int8_t`. Do not use bare `char` unless it is to pass to a third-party API. This type can be signed or unsigned depending on the architecture, which can lead to interoperability problems or dangerous conditions such as out-of-bounds array accesses - Prefer explicit constructions over implicit ones that rely on 'magical' C++ behavior - *Rationale*: Easier to understand what is happening, thus easier to spot mistakes, even for those that are not language lawyers - Initialize all non-static class members where they are defined - *Rationale*: Initializing the members in the declaration makes it easy to spot uninitialized ones, and avoids accidentally reading uninitialized memory ```cpp class A { uint32_t m_count{0}; } ``` Strings and formatting ------------------------ - Use `std::string`, avoid C string manipulation functions - *Rationale*: C++ string handling is marginally safer, less scope for buffer overflows and surprises with `\0` characters. Also some C string manipulations tend to act differently depending on platform, or even the user locale - Use `ParseInt32`, `ParseInt64`, `ParseUInt32`, `ParseUInt64`, `ParseDouble` from `utilstrencodings.h` for number parsing - *Rationale*: These functions do overflow checking, and avoid pesky locale issues Variable names -------------- The shadowing warning (`-Wshadow`) is enabled by default. It prevents issues rising from using a different variable with the same name. E.g. in member initializers, prepend `_` to the argument name shadowing the member name: ```c++ class AddressBookPage { Mode m_mode; } AddressBookPage::AddressBookPage(Mode _mode) : m_mode(_mode) ... ``` When using nested cycles, do not name the inner cycle variable the same as in upper cycle etc. Please name variables so that their names do not shadow variables defined in the source code. Threads and synchronization ---------------------------- - Build and run tests with `-DDEBUG_LOCKORDER` to verify that no potential deadlocks are introduced. As of 0.12, this is defined by default when configuring with `--enable-debug` - When using `LOCK`/`TRY_LOCK` be aware that the lock exists in the context of the current scope, so surround the statement and the code that needs the lock with braces OK: ```c++ { TRY_LOCK(cs_vNodes, lockNodes); ... } ``` Wrong: ```c++ TRY_LOCK(cs_vNodes, lockNodes); { ... } ``` Scripts -------------------------- ### Shebang - Use `#!/usr/bin/env bash` instead of obsolete `#!/bin/bash`. - [*Rationale*](https://github.com/dylanaraps/pure-bash-bible#shebang): `#!/bin/bash` assumes it is always installed to /bin/ which can cause issues; `#!/usr/bin/env bash` searches the user's PATH to find the bash binary. OK: ```bash #!/usr/bin/env bash ``` Wrong: ```bash #!/bin/bash ``` Source code organization -------------------------- - Implementation code should go into the `.cpp` file and not the `.h`, unless necessary due to template usage or when performance due to inlining is critical - *Rationale*: Shorter and simpler header files are easier to read, and reduce compile time - Use only the lowercase alphanumerics (`a-z0-9`), underscore (`_`) and hyphen (`-`) in source code filenames. - *Rationale*: `grep`:ing and auto-completing filenames is easier when using a consistent naming pattern. Potential problems when building on case-insensitive filesystems are avoided when using only lowercase characters in source code filenames. - Don't import anything into the global namespace (`using namespace ...`). Use fully specified types such as `std::string`. - *Rationale*: Avoids symbol conflicts - Terminate namespaces with a comment (`// namespace mynamespace`). The comment should be placed on the same line as the brace closing the namespace, e.g. ```c++ namespace mynamespace { ... } // namespace mynamespace namespace { ... } // namespace ``` - *Rationale*: Avoids confusion about the namespace context Header Inclusions ----------------- - Header inclusions should use angle brackets (`#include <>`). The include path should be relative to the `src` folder. e.g.: `#include ` - Native C++ headers should be preferred over C compatibility headers. - e.g.: use `` instead of `` + e.g.: use `` instead of `` - In order to make the code consistent, header files should be included in the following order, with each section separated by a newline: 1. In a .cpp file, the associated .h is in first position. In a test source, this is the header file under test. 2. The project headers. 3. The test headers. 4. The 3rd party libraries headers. Different libraries should be in different sections. 5. The system libraries. -All headers should be lexically ordered inside their block. +All headers should be lexically ordered inside their block. - Use include guards to avoid the problem of double inclusion. The header file `foo/bar.h` should use the include guard identifier `BITCOIN_FOO_BAR_H`, e.g. ```c++ #ifndef BITCOIN_FOO_BAR_H #define BITCOIN_FOO_BAR_H ... #endif // BITCOIN_FOO_BAR_H ``` GUI ----- - Do not display or manipulate dialogs in model code (classes `*Model`) - *Rationale*: Model classes pass through events and data from the core, they should not interact with the user. That's where View classes come in. The converse also holds: try to not directly access core data structures from Views. - Avoid adding slow or blocking code in the GUI thread. In particular do not add new `interface::Node` and `interface::Wallet` method calls, even if they may be fast now, in case they are changed to lock or communicate across processes in the future. Prefer to offload work from the GUI thread to worker threads (see `RPCExecutor` in console code as an example) or take other steps (see https://doc.qt.io/archives/qq/qq27-responsive-guis.html) to keep the GUI responsive. - *Rationale*: Blocking the GUI thread can increase latency, and lead to hangs and deadlocks. Unit Tests ----------- - - Test suite naming convention: The Boost test suite in file + - Test suite naming convention: The Boost test suite in file `src/test/foo_tests.cpp` should be named `foo_tests`. Test suite names must be unique. Third party libraries --------------------- Several parts of the repository are software maintained elsewhere. Changes to these should preferably be sent upstream but bugfixes may also be submitted to Bitcoin ABC so that they can be integrated quickly. Cosmetic changes should be purely taken upstream. Current third party libraries include: - src/leveldb - Upstream at https://github.com/google/leveldb ; Maintained by Google. - **Note**: Follow the instructions in [Upgrading LevelDB](#upgrading-leveldb) when merging upstream changes to Bitcoin ABC. - src/libsecp256k1 - Upstream at https://github.com/bitcoin-core/secp256k1/ ; actively maintained by Bitcoin Core contributors. Bitcoin ABC is using a modified version of libsecp256k1, some changes might be directly submitted to Bitcoin ABC. See the [secp256k1 README](/src/secp256k1/README.md) for details. - src/crypto/ctaes - Upstream at https://github.com/bitcoin-core/ctaes ; maintained by Bitcoin Core contributors. - src/univalue - Upstream at https://github.com/jgarzik/univalue ; maintained by Jeff Garzik. Upgrading LevelDB --------------------- Extra care must be taken when upgrading LevelDB. This section explains issues you must be aware of. ### File Descriptor Counts In most configurations we use the default LevelDB value for `max_open_files`, which is 1000 at the time of this writing. If LevelDB actually uses this many file descriptors it will cause problems with Bitcoin's `select()` loop, because it may cause new sockets to be created where the fd value is >= 1024. For this reason, on 64-bit Unix systems we rely on an internal LevelDB optimization that uses `mmap()` + `close()` to open table files without actually retaining references to the table file descriptors. If you are upgrading LevelDB, you must sanity check the changes to make sure that this assumption remains valid. In addition to reviewing the upstream changes in `env_posix.cc`, you can use `lsof` to check this. For example, on Linux this command will show open `.ldb` file counts: ```bash $ lsof -p $(pidof bitcoind) |\ awk 'BEGIN { fd=0; mem=0; } /ldb$/ { if ($4 == "mem") mem++; else fd++ } END { printf "mem = %s, fd = %s\n", mem, fd}' mem = 119, fd = 0 ``` The `mem` value shows how many files are mmap'ed, and the `fd` value shows you many file descriptors these files are using. You should check that `fd` is a small number (usually 0 on 64-bit hosts). See the notes in the `SetMaxOpenFiles()` function in `dbwrapper.cc` for more details. ### Consensus Compatibility It is possible for LevelDB changes to inadvertently change consensus compatibility between nodes. This happened in Bitcoin 0.8 (when LevelDB was first introduced). When upgrading LevelDB you should review the upstream changes to check for issues affecting consensus compatibility. For example, if LevelDB had a bug that accidentally prevented a key from being returned in an edge case, and that bug was fixed upstream, the bug "fix" would be an incompatible consensus change. In this situation the correct behavior would be to revert the upstream fix before applying the updates to Bitcoin's copy of LevelDB. In general you should be wary of any upstream changes affecting what data is returned from LevelDB queries. Git and GitHub tips --------------------- - Github is not typically the source of truth for pull requests. See CONTRIBUTING.md for instructions on setting up your repo correctly. - Similarly, your git remote origin should be set to: `ssh://vcs@reviews.bitcoinabc.org:2221/source/bitcoin-abc.git` instead of github.com. See CONTRIBUTING.md for details. - For resolving merge/rebase conflicts, it can be useful to enable diff3 style using `git config merge.conflictstyle diff3`. Instead of <<< yours === theirs >>> you will see <<< yours ||| original === theirs >>> This may make it much clearer what caused the conflict. In this style, you can often just look at what changed between *original* and *theirs*, and mechanically apply that to *yours* (or the other way around). - When reviewing patches which change indentation in C++ files, use `git diff -w` and `git show -w`. This makes the diff algorithm ignore whitespace changes. This feature is also available on github.com, by adding `?w=1` at the end of any URL which shows a diff. - When reviewing patches that change symbol names in many places, use `git diff --word-diff`. This will instead of showing the patch as deleted/added *lines*, show deleted/added *words*. - When reviewing patches that move code around, try using `git diff --patience commit~:old/file.cpp commit:new/file/name.cpp`, and ignoring everything except the moved body of code which should show up as neither `+` or `-` lines. In case it was not a pure move, this may even work when combined with the `-w` or `--word-diff` options described above. - When looking at other's pull requests, it may make sense to add the following section to your `.git/config` file: [remote "upstream-pull"] fetch = +refs/pull/*:refs/remotes/upstream-pull/* url = git@github.com:bitcoin/bitcoin.git This will add an `upstream-pull` remote to your git repository, which can be fetched using `git fetch --all` or `git fetch upstream-pull`. Afterwards, you can use `upstream-pull/NUMBER/head` in arguments to `git show`, `git checkout` and anywhere a commit id would be acceptable to see the changes from pull request NUMBER. RPC interface guidelines -------------------------- A few guidelines for introducing and reviewing new RPC interfaces: - Method naming: use consecutive lower-case names such as `getrawtransaction` and `submitblock` - *Rationale*: Consistency with existing interface. - Argument naming: use snake case `fee_delta` (and not, e.g. camel case `feeDelta`) - *Rationale*: Consistency with existing interface. - Use the JSON parser for parsing, don't manually parse integers or strings from arguments unless absolutely necessary. - *Rationale*: Introduces hand-rolled string manipulation code at both the caller and callee sites, which is error prone, and it is easy to get things such as escaping wrong. JSON already supports nested data structures, no need to re-invent the wheel. - *Exception*: AmountFromValue can parse amounts as string. This was introduced because many JSON parsers and formatters hard-code handling decimal numbers as floating point values, resulting in potential loss of precision. This is unacceptable for monetary values. **Always** use `AmountFromValue` and `ValueFromAmount` when inputting or outputting monetary values. The only exceptions to this are `prioritisetransaction` and `getblocktemplate` because their interface is specified as-is in BIP22. - Missing arguments and 'null' should be treated the same: as default values. If there is no default value, both cases should fail in the same way. The easiest way to follow this guideline is detect unspecified arguments with `params[x].isNull()` instead of `params.size() <= x`. The former returns true if the argument is either null or missing, while the latter returns true if is missing, and false if it is null. - *Rationale*: Avoids surprises when switching to name-based arguments. Missing name-based arguments are passed as 'null'. - Try not to overload methods on argument type. E.g. don't make `getblock(true)` and `getblock("hash")` do different things. - *Rationale*: This is impossible to use with `bitcoin-cli`, and can be surprising to users. - *Exception*: Some RPC calls can take both an `int` and `bool`, most notably when a bool was switched to a multi-value, or due to other historical reasons. **Always** have false map to 0 and true to 1 in this case. - Don't forget to fill in the argument names correctly in the RPC command table. - *Rationale*: If not, the call can not be used with name-based arguments. - Set okSafeMode in the RPC command table to a sensible value: safe mode is when the blockchain is regarded to be in a confused state, and the client deems it unsafe to do anything irreversible such as send. Anything that just queries should be permitted. - *Rationale*: Troubleshooting a node in safe mode is difficult if half the RPCs don't work. - Add every non-string RPC argument `(method, idx, name)` to the table `vRPCConvertParams` in `rpc/client.cpp`. - *Rationale*: `bitcoin-cli` and the GUI debug console use this table to determine how to convert a plaintext command line to JSON. If the types don't match, the method can be unusable from there. - A RPC method must either be a wallet method or a non-wallet method. Do not introduce new methods such as `signrawtransaction` that differ in behavior based on presence of a wallet. - *Rationale*: as well as complicating the implementation and interfering with the introduction of multi-wallet, wallet and non-wallet code should be separated to avoid introducing circular dependencies between code units. - Try to make the RPC response a JSON object. - *Rationale*: If a RPC response is not a JSON object then it is harder to avoid API breakage if new data in the response is needed. - Wallet RPCs call BlockUntilSyncedToCurrentChain to maintain consistency with `getblockchaininfo`'s state immediately prior to the call's execution. Wallet RPCs whose behavior does *not* depend on the current chainstate may omit this call. - *Rationale*: In previous versions of Bitcoin Core, the wallet was always in-sync with the chainstate (by virtue of them all being updated in the same cs_main lock). In order to maintain the behavior that wallet RPCs return results as of at least the highest best-known block an RPC client may be aware of prior to entering a wallet RPC call, we must block until the wallet is caught up to the chainstate as of the RPC call's entry. This also makes the API much easier for RPC clients to reason about. - Be aware of RPC method aliases and generally avoid registering the same callback function pointer for different RPCs. - *Rationale*: RPC methods registered with the same function pointer will be considered aliases and only the first method name will show up in the `help` rpc command list. - *Exception*: Using RPC method aliases may be appropriate in cases where a new RPC is replacing a deprecated RPC, to avoid both RPCs confusingly showing up in the command list. diff --git a/doc/functional-tests.md b/doc/functional-tests.md index 21e24f9659..62ca94bb03 100644 --- a/doc/functional-tests.md +++ b/doc/functional-tests.md @@ -1,300 +1,300 @@ # Functional tests The [/test/](/test/) directory contains integration tests that test bitcoind and its utilities in their entirety. It does not contain unit tests, which can be found in [/src/test](/src/test), [/src/wallet/test](/src/wallet/test), etc. There are currently two sets of tests in the [/test/](/test/) directory: - [functional](/test/functional) which test the functionality of bitcoind and bitcoin-qt by interacting with them through the RPC and P2P interfaces. - [util](/test/util) which tests the bitcoin utilities, currently only bitcoin-tx. The util tests are run as part of `make check` target. The functional tests are run by the Teamcity continuous build process whenever a diff is created or updated on Phabricator. Both sets of tests can also be run locally. # Running functional tests locally Build for your system first. Be sure to enable wallet, utils and daemon when you configure. Tests will not run otherwise. ### Functional tests #### Dependencies The ZMQ functional test requires a python ZMQ library. To install it: - On Unix, run `sudo apt-get install python3-zmq` - On mac OS, run `pip3 install pyzmq` #### Running the tests Individual tests can be run by directly calling the test script, eg: ``` test/functional/example_test.py ``` or can be run through the test_runner harness, eg: ``` test/functional/test_runner.py example_test ``` You can run any combination (incl. duplicates) of tests by calling: ``` test/functional/test_runner.py ... ``` Run the regression test suite with: ``` test/functional/test_runner.py ``` Run all possible tests with ``` test/functional/test_runner.py --extended ``` By default, up to 4 tests will be run in parallel by test_runner. To specify how many jobs to run, append `--jobs=n` The individual tests and the test_runner harness have many command-line options. Run `test_runner.py -h` to see them all. #### Troubleshooting and debugging test failures ##### Resource contention The P2P and RPC ports used by the bitcoind nodes-under-test are chosen to make conflicts with other processes unlikely. However, if there is another bitcoind process running on the system (perhaps from a previous test which hasn't successfully killed all its bitcoind nodes), then there may be a port conflict which will cause the test to fail. It is recommended that you run the tests on a system where no other bitcoind processes are running. On linux, the test_framework will warn if there is another bitcoind process running when the tests are started. If there are zombie bitcoind processes after test failure, you can kill them by running the following commands. **Note that these commands will kill all bitcoind processes running on the system, so should not be used if any non-test bitcoind processes are being run.** ```bash killall bitcoind ``` or ```bash pkill -9 bitcoind ``` ##### Data directory cache A pre-mined blockchain with 200 blocks is generated the first time a functional test is run and is stored in test/cache. This speeds up test startup times since new blockchains don't need to be generated for each test. However, the cache may get into a bad state, in which case tests will fail. If this happens, remove the cache directory (and make sure bitcoind processes are stopped as above): ```bash rm -rf cache killall bitcoind ``` ##### Test logging The tests contain logging at different levels (debug, info, warning, etc). By default: - When run through the test_runner harness, *all* logs are written to `test_framework.log` and no logs are output to the console. - When run directly, *all* logs are written to `test_framework.log` and INFO level and above are output to the console. - When run on Travis, no logs are output to the console. However, if a test fails, the `test_framework.log` and bitcoind `debug.log`s will all be dumped to the console to help troubleshooting. To change the level of logs output to the console, use the `-l` command line argument. `test_framework.log` and bitcoind `debug.log`s can be combined into a single aggregate log by running the `combine_logs.py` script. The output can be plain text, colorized text or html. For example: ``` combine_logs.py -c | less -r ``` will pipe the colorized logs from the test into less. Use `--tracerpc` to trace out all the RPC calls and responses to the console. For some tests (eg any that use `submitblock` to submit a full block over RPC), this can result in a lot of screen output. By default, the test data directory will be deleted after a successful run. Use `--nocleanup` to leave the test data directory intact. The test data directory is never deleted after a failed test. ##### Attaching a debugger A python debugger can be attached to tests at any point. Just add the line: ```py -import pdb; pdb.set_trace() +import pdb; pdb.set_trace() ``` anywhere in the test. You will then be able to inspect variables, as well as call methods that interact with the bitcoind nodes-under-test. If further introspection of the bitcoind instances themselves becomes necessary, this can be accomplished by first setting a pdb breakpoint at an appropriate location, running the test to that point, then using `gdb` to attach to the process and debug. For instance, to attach to `self.node[1]` during a run: ```bash 2017-06-27 14:13:56.686000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testo9vsdjo3 ``` use the directory path to get the pid from the pid file: ```bash cat /tmp/user/1000/testo9vsdjo3/node1/regtest/bitcoind.pid gdb /home/example/bitcoind ``` Note: gdb attach step may require `sudo`. To get rid of this, you can run: ```bash echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope ``` ##### Prevent using deprecated features Python will issue a `DeprecationWarning` when a deprecated feature is encountered in a script. By default, this warning message is ignored and not displayed to the user. This behavior can be changed by setting the environment variable `PYTHONWARNINGS` as follow: `PYTHONWARNINGS=default::DeprecationWarning` The warning message will now be printed to the `sys.stderr` output. ### Util tests Util tests can be run locally by running `test/util/bitcoin-util-test.py`. Use the `-v` option for verbose output. # Writing functional tests #### Example test The [example_test.py](example_test.py) is a heavily commented example of a test case that uses both the RPC and P2P interfaces. If you are writing your first test, copy that file and modify to fit your needs. #### Coverage Running `test_runner.py` with the `--coverage` argument tracks which RPCs are called by the tests and prints a report of uncovered RPCs in the summary. This can be used (along with the `--extended` argument) to find out which RPCs we don't have test cases for. #### Style guidelines - Where possible, try to adhere to [PEP-8 guidelines](https://www.python.org/dev/peps/pep-0008/) - Use a python linter like flake8 before submitting PRs to catch common style nits (eg trailing whitespace, unused imports, etc) - Avoid wildcard imports where possible - Use a module-level docstring to describe what the test is testing, and how it is testing it. - When subclassing the BitcoinTestFramwork, place overrides for the `set_test_params()`, `add_options()` and `setup_xxxx()` methods at the top of the subclass, then locally-defined helper methods, then the `run_test()` method. #### General test-writing advice - Set `self.num_nodes` to the minimum number of nodes necessary for the test. Having additional unrequired nodes adds to the execution time of the test as well as memory/CPU/disk requirements (which is important when running tests in parallel or on Travis). - Avoid stop-starting the nodes multiple times during the test if possible. A stop-start takes several seconds, so doing it several times blows up the runtime of the test. - Set the `self.setup_clean_chain` variable in `set_test_params()` to control whether or not to use the cached data directories. The cached data directories contain a 200-block pre-mined blockchain and wallets for four nodes. Each node has 25 mature blocks (25x50=1250 BTC) in its wallet. - When calling RPCs with lots of arguments, consider using named keyword arguments instead of positional arguments to make the intent of the call clear to readers. - Many of the core test framework classes such as `CBlock` and `CTransaction` don't allow new attributes to be added to their objects at runtime like typical Python objects allow. This helps prevent unpredictable side effects from typographical errors or usage of the objects outside of their intended purpose. #### RPC and P2P definitions Test writers may find it helpful to refer to the definitions for the RPC and P2P messages. These can be found in the following source files: - `/src/rpc/*` for RPCs - `/src/wallet/rpc*` for wallet RPCs - `ProcessMessage()` in `/src/net_processing.cpp` for parsing P2P messages #### Using the P2P interface - `messages.py` contains all the definitions for objects that pass over the network (`CBlock`, `CTransaction`, etc, along with the network-level wrappers for them, `msg_block`, `msg_tx`, etc). - P2P tests have two threads. One thread handles all network communication with the bitcoind(s) being tested in a callback-based event loop; the other implements the test logic. - `P2PConnection` is the class used to connect to a bitcoind. `P2PInterface` contains the higher level logic for processing P2P payloads and connecting to the Bitcoin Core node application logic. For custom behaviour, subclass the P2PInterface object and override the callback methods. - Can be used to write tests where specific P2P protocol behavior is tested. Examples tests are `p2p_unrequested_blocks.py`, `p2p_compactblocks.py`. ### test-framework modules #### [test_framework/authproxy.py](test_framework/authproxy.py) Taken from the [python-bitcoinrpc repository](https://github.com/jgarzik/python-bitcoinrpc). #### [test_framework/test_framework.py](test_framework/test_framework.py) Base class for functional tests. #### [test_framework/util.py](test_framework/util.py) Generally useful functions. #### [test_framework/mininode.py](test_framework/mininode.py) Basic code to support P2P connectivity to a bitcoind. #### [test_framework/script.py](test_framework/script.py) Utilities for manipulating transaction scripts (originally from python-bitcoinlib) #### [test_framework/key.py](test_framework/key.py) Wrapper around OpenSSL EC_Key (originally from python-bitcoinlib) #### [test_framework/bignum.py](test_framework/bignum.py) Helpers for script.py #### [test_framework/blocktools.py](test_framework/blocktools.py) Helper functions for creating blocks and transactions. diff --git a/doc/gitian-building.md b/doc/gitian-building.md index ca41d1974f..172a5e38cc 100644 --- a/doc/gitian-building.md +++ b/doc/gitian-building.md @@ -1,115 +1,115 @@ Gitian building =============== *Setup instructions for a Gitian build of Bitcoin ABC using a VM or physical system.* Gitian is the deterministic build process that is used to build the Bitcoin ABC executables. It provides a way to be reasonably sure that the executables are really built from the source on GitHub. It also makes sure that the same, tested dependencies are used and statically built into the executable. Multiple developers build the source code by following a specific descriptor ("recipe"), cryptographically sign the result, and upload the resulting signature. These results are compared and only if they match, the build is accepted and uploaded to bitcoinabc.org. More independent Gitian builders are needed, which is why this guide exists. It is preferred you follow these steps yourself instead of using someone else's VM image to avoid 'contaminating' the build. Table of Contents ----------------- - [Preparing the Gitian builder host](#preparing-the-gitian-builder-host) - [Getting and building the inputs](#getting-and-building-the-inputs) - [Building Bitcoin Core](#building-bitcoin-core) - [Building an alternative repository](#building-an-alternative-repository) - [Signing externally](#signing-externally) - [Uploading signatures](#uploading-signatures) Preparing the Gitian builder host --------------------------------- The first step is to prepare the host environment that will be used to perform the Gitian builds. This guide explains how to set up the environment, and how to start the builds. Gitian builds are known to be working on recent versions of Debian, Ubuntu and Fedora. If your machine is already running one of those operating systems, you can perform Gitian builds on the actual hardware. Alternatively, you can install one of the supported operating systems in a virtual machine. You can create the virtual machine using [vagrant](./gitian-building/gitian-building-vagrant.md) or chose to setup the VM manually. Any kind of virtualization can be used, for example: - [VirtualBox](https://www.virtualbox.org/) (covered by this guide) - [KVM](http://www.linux-kvm.org/page/Main_Page) - [LXC](https://linuxcontainers.org/) Please refer to the following documents to set up the operating systems and Gitian. | | Debian | Fedora | |-----------------------------------|------------------------------------------------------------------------------------|------------------------------------------------------------------------------------| | Setup virtual machine (optional) | [Create Debian VirtualBox](./gitian-building/gitian-building-create-vm-debian.md) | [Create Fedora VirtualBox](./gitian-building/gitian-building-create-vm-fedora.md) | | Setup Gitian | [Setup Gitian on Debian](./gitian-building/gitian-building-setup-gitian-debian.md) | [Setup Gitian on Fedora](./gitian-building/gitian-building-setup-gitian-fedora.md) | Note that a version of `lxc-execute` higher or equal to 2.1.1 is required. You can check the version with `lxc-execute --version`. Non-Debian / Ubuntu, Manual and Offline Building ------------------------------------------------ The instructions below use the automated script [gitian-build.py](https://github.com/Bitcoin-ABC/bitcoin-abc/blob/master/contrib/gitian-build.py) which only works in Debian/Ubuntu. For manual steps and instructions for fully offline signing, see [this guide](./gitian-building/gitian-building-manual.md). MacOS code signing ------------------ In order to sign builds for MacOS, you need to download the free SDK and extract a file. The steps are described [here](./gitian-building/gitian-building-mac-os-sdk.md). It is possible to download the resulting archive directly for users that desire to do so: ```bash cd ~/gitian-builder curl -LO https://storage.googleapis.com/f4936e83b2dcbca742be51fb9692b153/MacOSX10.11.sdk.tar.gz echo "4732b52b5ebe300c8c91cbeed6d19d59c1ff9c56c7a1dd6cfa518b9c2c72abde MacOSX10.11.sdk.tar.gz" | sha256sum -c mkdir -p inputs mv MacOSX10.11.sdk.tar.gz inputs ``` Alternatively, you can skip the macOS build by adding `--os=lw` below. Initial Gitian Setup -------------------- The `gitian-build.py` script will checkout different release tags, so it's best to copy it: ```bash cp bitcoin-abc/contrib/gitian-build.py . ``` You only need to do this once: ```bash ./gitian-build.py --setup satoshi 0.18.5 ``` -Where `satoshi` is your Github name and `0.18.5` is the most recent tag (without `v`). +Where `satoshi` is your Github name and `0.18.5` is the most recent tag (without `v`). Build binaries -------------- Windows and macOS have code signed binaries, but those won't be available until a few developers have gitian signed the non-codesigned binaries. To build the most recent tag: ```bash ./gitian-build.py --detach-sign --no-commit -b satoshi 0.18.5 ``` To speed up the build, use `-j 5 -m 5000` as the first arguments, where `5` is the number of CPU's you allocated to the VM plus one, and 5000 is a little bit less than then the MB's of RAM you allocated. If all went well, this produces a number of (uncommited) `.assert` files in the gitian.sigs repository. You need to copy these uncommited changes to your host machine, where you can sign them: ```bash export NAME=satoshi -gpg --output $VERSION-linux/$NAME/bitcoin-abc-linux-0.18.5-build.assert.sig --detach-sign 0.18.5-linux/$NAME/bitcoin-abc-linux-0.18.5-build.assert -gpg --output $VERSION-osx-unsigned/$NAME/bitcoin-abc-osx-0.18.5-build.assert.sig --detach-sign 0.18.5-osx-unsigned/$NAME/bitcoin-abc-osx-0.18.5-build.assert -gpg --output $VERSION-win-unsigned/$NAME/bitcoin-abc-win-0.18.5-build.assert.sig --detach-sign 0.18.5-win-unsigned/$NAME/bitcoin-abc-win-0.18.5-build.assert +gpg --output $VERSION-linux/$NAME/bitcoin-abc-linux-0.18.5-build.assert.sig --detach-sign 0.18.5-linux/$NAME/bitcoin-abc-linux-0.18.5-build.assert +gpg --output $VERSION-osx-unsigned/$NAME/bitcoin-abc-osx-0.18.5-build.assert.sig --detach-sign 0.18.5-osx-unsigned/$NAME/bitcoin-abc-osx-0.18.5-build.assert +gpg --output $VERSION-win-unsigned/$NAME/bitcoin-abc-win-0.18.5-build.assert.sig --detach-sign 0.18.5-win-unsigned/$NAME/bitcoin-abc-win-0.18.5-build.assert ``` diff --git a/doc/init.md b/doc/init.md index e1796860e6..5fcbad579c 100644 --- a/doc/init.md +++ b/doc/init.md @@ -1,125 +1,125 @@ Sample init scripts and service configuration for bitcoind ========================================================== Sample scripts and configuration files for systemd, Upstart and OpenRC can be found in the contrib/init folder. contrib/init/bitcoind.service: systemd service unit configuration contrib/init/bitcoind.openrc: OpenRC compatible SysV style init script contrib/init/bitcoind.openrcconf: OpenRC conf.d file contrib/init/bitcoind.conf: Upstart service configuration file contrib/init/bitcoind.init: CentOS compatible SysV style init script 1. Service User --------------------------------- All three Linux startup configurations assume the existence of a "bitcoin" user and group. They must be created before attempting to use these scripts. The macOS configuration assumes bitcoind will be set up for the current user. 2. Configuration --------------------------------- At a bare minimum, bitcoind requires that the rpcpassword setting be set when running as a daemon. If the configuration file does not exist or this setting is not set, bitcoind will shutdown promptly after startup. This password does not have to be remembered or typed as it is mostly used as a fixed token that bitcoind and client programs read from the configuration file, however it is recommended that a strong and secure password be used as this password is security critical to securing the wallet should the wallet be enabled. If bitcoind is run with the "-server" flag (set by default), and no rpcpassword is set, it will use a special cookie file for authentication. The cookie is generated with random content when the daemon starts, and deleted when it exits. Read access to this file controls who can access it through RPC. By default the cookie is stored in the data directory, but it's location can be overridden with the option '-rpccookiefile'. This allows for running bitcoind without having to do any manual configuration. `conf`, `pid`, and `wallet` accept relative paths which are interpreted as relative to the data directory. `wallet` *only* supports relative paths. For an example configuration file that describes the configuration settings, see `contrib/debian/examples/bitcoin.conf`. 3. Paths --------------------------------- 3a) Linux All three configurations assume several paths that might need to be adjusted. -Binary: `/usr/bin/bitcoind` -Configuration file: `/etc/bitcoin/bitcoin.conf` -Data directory: `/var/lib/bitcoind` -PID file: `/var/run/bitcoind/bitcoind.pid` (OpenRC and Upstart) or `/var/lib/bitcoind/bitcoind.pid` (systemd) -Lock file: `/var/lock/subsys/bitcoind` (CentOS) +Binary: `/usr/bin/bitcoind` +Configuration file: `/etc/bitcoin/bitcoin.conf` +Data directory: `/var/lib/bitcoind` +PID file: `/var/run/bitcoind/bitcoind.pid` (OpenRC and Upstart) or `/var/lib/bitcoind/bitcoind.pid` (systemd) +Lock file: `/var/lock/subsys/bitcoind` (CentOS) The configuration file, PID directory (if applicable) and data directory should all be owned by the bitcoin user and group. It is advised for security reasons to make the configuration file and data directory only readable by the bitcoin user and group. Access to bitcoin-cli and other bitcoind rpc clients can then be controlled by group membership. 3b) macOS -Binary: `/usr/local/bin/bitcoind` -Configuration file: `~/Library/Application Support/Bitcoin/bitcoin.conf` +Binary: `/usr/local/bin/bitcoind` +Configuration file: `~/Library/Application Support/Bitcoin/bitcoin.conf` Data directory: `~/Library/Application Support/Bitcoin` Lock file: `~/Library/Application Support/Bitcoin/.lock` 4. Installing Service Configuration ----------------------------------- 4a) systemd Installing this .service file consists of just copying it to /usr/lib/systemd/system directory, followed by the command `systemctl daemon-reload` in order to update running systemd configuration. To test, run `systemctl start bitcoind` and to enable for system startup run `systemctl enable bitcoind` 4b) OpenRC Rename bitcoind.openrc to bitcoind and drop it in /etc/init.d. Double check ownership and permissions and make it executable. Test it with `/etc/init.d/bitcoind start` and configure it to run on startup with `rc-update add bitcoind` 4c) Upstart (for Debian/Ubuntu based distributions) Drop bitcoind.conf in /etc/init. Test by running `service bitcoind start` it will automatically start on reboot. NOTE: This script is incompatible with CentOS 5 and Amazon Linux 2014 as they use old versions of Upstart and do not supply the start-stop-daemon utility. 4d) CentOS Copy bitcoind.init to /etc/init.d/bitcoind. Test by running `service bitcoind start`. Using this script, you can adjust the path and flags to the bitcoind program by setting the BITCOIND and FLAGS environment variables in the file /etc/sysconfig/bitcoind. You can also use the DAEMONOPTS environment variable here. 4e) macOS Copy org.bitcoin.bitcoind.plist into ~/Library/LaunchAgents. Load the launch agent by running `launchctl load ~/Library/LaunchAgents/org.bitcoin.bitcoind.plist`. This Launch Agent will cause bitcoind to start whenever the user logs in. NOTE: This approach is intended for those wanting to run bitcoind as the current user. You will need to modify org.bitcoin.bitcoind.plist if you intend to use it as a Launch Daemon with a dedicated bitcoin user. 5. Auto-respawn ----------------------------------- Auto respawning is currently only configured for Upstart and systemd. Reasonable defaults have been chosen but YMMV. diff --git a/doc/release-process.md b/doc/release-process.md index 39e92235ad..dcb98c85f7 100644 --- a/doc/release-process.md +++ b/doc/release-process.md @@ -1,72 +1,72 @@ Bitcoin ABC Release Process =========================== ## Before Release 1. Check configuration - Check features planned for the release are implemented and documented (or more informally, that the Release Manager agrees it is feature complete) - Check that finished tasks / tickets are marked as resolved 2. Verify tests passed - Any known issues or limitations should be documented in release notes - Known bugs should have tickets - Run `arc lint --everything` and check there is no linter error - Ensure that bitcoind and bitcoin-qt run with no issue on all supported platforms. Manually test bitcoin-qt by sending some transactions and navigating through the menus. - + 3. Update the documents / code which needs to be updated every release - Check that [release-notes.md](doc/release-notes.md) is complete, and fill in any missing items. - Update [bips.md](/doc/bips.md) to account for changes since the last release. - (major releases) Update [`BLOCK_CHAIN_SIZE`](/src/qt/intro.cpp) to the current size plus some overhead. - Regenerate manpages (run `contrib/devtools/gen-manpages.sh`, or for out-of-tree builds run `BUILDDIR=$PWD/build contrib/devtools/gen-manpages.sh`). - Update seeds as per [contrib/seeds/README.md](/contrib/seeds/README.md). 4. Add git tag for release a. Create the tag: `git tag vM.m.r` (M = major version, m = minor version, r = revision) b. Push the tag to Github: ``` git push master git push vM.m.r ``` 5. Increment version number for the next release in: - `doc/release-notes.md` (and copy existing one to versioned `doc/release-notes/*.md`) - `configure.ac` - `CMakeLists.txt` - `contrib/seeds/makeseeds.py` (only after a new major release) ## Release 6. Create Gitian Builds (see [gitian-building.md](/doc/gitian-building.md)) 7. Verify matching Gitian Builds, gather signatures 8. Verify IBD bith with and without `-checkpoints=0 -assumevalid=0` 9. Upload Gitian Builds to [bitcoinabc.org](https://download.bitcoinabc.org/) 10. Create a [GitHub release](https://github.com/Bitcoin-ABC/bitcoin-abc/releases): `contrib/release/github-release.sh -a -t -o ` 11. Create [Ubuntu PPA packages](https://launchpad.net/~bitcoin-abc/+archive/ubuntu/ppa): `contrib/release/debian-packages.sh "Your Name "` 12. Notify maintainers of AUR and Docker images to build their packages. They should be given 1-day advance notice if possible. ## After Release 13. Update version number on www.bitcoinabc.org 14. Publish signed checksums (various places, e.g. blog, reddit/r/BitcoinABC) 15. Announce Release: - [Reddit](https://www.reddit.com/r/BitcoinABC/) - Twitter @Bitcoin_ABC - - Public slack channels friendly to Bitcoin ABC announcements + - Public slack channels friendly to Bitcoin ABC announcements (eg. #abc-announce on BTCforks, #hardfork on BTCchat) diff --git a/doc/tor.md b/doc/tor.md index e979b74241..b50d66ccb7 100644 --- a/doc/tor.md +++ b/doc/tor.md @@ -1,121 +1,121 @@ # TOR SUPPORT IN BITCOIN It is possible to run Bitcoin ABC as a Tor hidden service, and connect to such services. The following directions assume you have a Tor proxy running on port 9050. Many distributions default to having a SOCKS proxy listening on port 9050, but others may not. In particular, the Tor Browser Bundle defaults to listening on port 9150. See [Tor Project FAQ:TBBSocksPort](https://www.torproject.org/docs/faq.html.en#TBBSocksPort) for how to properly configure Tor. ## 1. Run Bitcoin ABC behind a Tor proxy The first step is running Bitcoin ABC behind a Tor proxy. This will already anonymize all outgoing connections, but more is possible. -proxy=ip:port Set the proxy server. If SOCKS5 is selected (default), this proxy server will be used to try to reach .onion addresses as well. -onion=ip:port Set the proxy server to use for Tor hidden services. You do not need to set this if it's the same as -proxy. You can use -noonion to explicitly disable access to hidden service. -listen When using -proxy, listening is disabled by default. If you want to run a hidden service (see next section), you'll need to enable it explicitly. -connect=X When behind a Tor proxy, you can specify .onion addresses instead -addnode=X of IP addresses or hostnames in these parameters. It requires -seednode=X SOCKS5. In Tor mode, such addresses can also be exchanged with other P2P nodes. In a typical situation, this suffices to run behind a Tor proxy: ./bitcoind -proxy=127.0.0.1:9050 ## 2. Run a Bitcoin ABC hidden server If you configure your Tor system accordingly, it is possible to make your node also reachable from the Tor network. Add these lines to your /etc/tor/torrc (or equivalent config file): *Needed for Tor version 0.2.7.0 and older versions of Tor only. For newer versions of Tor see [Section 3](#3-automatically-listen-on-tor).* HiddenServiceDir /var/lib/tor/bitcoin-service/ HiddenServicePort 8333 127.0.0.1:8333 HiddenServicePort 18333 127.0.0.1:18333 The directory can be different of course, but (both) port numbers should be equal to your bitcoind's P2P listen port (8333 by default). -externalip=X You can tell bitcoin about its publicly reachable address using this option, and this can be a .onion address. Given the above configuration, you can find your .onion address in /var/lib/tor/bitcoin-service/hostname. For connections coming from unroutable addresses (such as 127.0.0.1, where the Tor proxy typically runs), .onion addresses are given preference for your node to advertise itself with. -listen You'll need to enable listening for incoming connections, as this is off by default behind a proxy. -discover When -externalip is specified, no attempt is made to discover local IPv4 or IPv6 addresses. If you want to run a dual stack, reachable from both Tor and IPv4 (or IPv6), you'll need to either pass your other addresses using -externalip, or explicitly enable -discover. Note that both addresses of a dual-stack system may be easily linkable using traffic analysis. In a typical situation, where you're only reachable via Tor, this should suffice: ./bitcoind -proxy=127.0.0.1:9050 -externalip=57qr3yd1nyntf5k.onion -listen (obviously, replace the .onion address with your own). It should be noted that you still listen on all devices and another node could establish a clearnet connection, when knowing your address. To mitigate this, additionally bind the address of your Tor proxy: ./bitcoind ... -bind=127.0.0.1 If you don't care too much about hiding your node, and want to be reachable on IPv4 as well, use `discover` instead: ./bitcoind ... -discover and open port 8333 on your firewall (or use -upnp). If you only want to use Tor to reach .onion addresses, but not use it as a proxy for normal IPv4/IPv6 communication, use: ./bitcoind -onion=127.0.0.1:9050 -externalip=57qr3yd1nyntf5k.onion -discover ## 3. Automatically listen on Tor Starting with Tor version 0.2.7.1 it is possible, through Tor's control socket API, to create and destroy 'ephemeral' hidden services programmatically. Bitcoin Core has been updated to make use of this. This means that if Tor is running (and proper authentication has been configured), -Bitcoin Core automatically creates a hidden service to listen on. This will positively +Bitcoin Core automatically creates a hidden service to listen on. This will positively affect the number of available .onion nodes. This new feature is enabled by default if Bitcoin ABC is listening (`-listen`), and requires a Tor connection to work. It can be explicitly disabled with `-listenonion=0` and, if not disabled, configured using the `-torcontrol` and `-torpassword` settings. To show verbose debugging information, pass `-debug=tor`. Connecting to Tor's control socket API requires one of two authentication methods to be configured. For cookie authentication the user running bitcoind must have write access to the `CookieAuthFile` specified in Tor configuration. In some cases, this is preconfigured and the creation of a hidden service is automatic. If permission problems are seen with `-debug=tor` they can be resolved by adding both the user running Tor and the user running bitcoind to the same group and setting permissions appropriately. On Debian-based systems the user running bitcoind can be added to the debian-tor group, which has the appropriate permissions. An alternative authentication method is the use of the `-torpassword` flag and a `hash-password` which can be enabled and specified in Tor configuration. ## 4. Privacy recommendations - Do not add anything but Bitcoin ABC ports to the hidden service created in section 2. If you run a web service too, create a new hidden service for that. Otherwise it is trivial to link them, which may reduce privacy. Hidden services created automatically (as in section 3) always have only one port open.