Changeset View
Changeset View
Standalone View
Standalone View
doc/gitian-building.md
Gitian building | Gitian building | ||||
================ | =============== | ||||
*Setup instructions for a Gitian build of Bitcoin ABC using an Ubuntu VM or physical system.* | *Setup instructions for a Gitian build of Bitcoin ABC using a VM or physical system.* | ||||
Gitian is the deterministic build process that is used to build the Bitcoin | Gitian is the deterministic build process that is used to build the Bitcoin | ||||
ABC executables. It provides a way to be reasonably sure that the | ABC executables. It provides a way to be reasonably sure that the | ||||
executables are really built from the source on GitHub. It also makes sure that | executables are really built from the source on GitHub. It also makes sure that | ||||
the same, tested dependencies are used and statically built into the executable. | the same, tested dependencies are used and statically built into the executable. | ||||
Multiple developers build the source code by following a specific descriptor | Multiple developers build the source code by following a specific descriptor | ||||
("recipe"), cryptographically sign the result, and upload the resulting signature. | ("recipe"), cryptographically sign the result, and upload the resulting signature. | ||||
These results are compared and only if they match, the build is accepted and uploaded | These results are compared and only if they match, the build is accepted and | ||||
to bitcoinabc.org. | uploaded to bitcoinabc.org. | ||||
More independent Gitian builders are needed, which is why this guide exists. | More independent Gitian builders are needed, which is why this guide exists. | ||||
It is preferred you follow these steps yourself instead of using someone else's | It is preferred you follow these steps yourself instead of using someone else's | ||||
VM image to avoid 'contaminating' the build. | VM image to avoid 'contaminating' the build. | ||||
Table of Contents | Table of Contents | ||||
------------------ | ----------------- | ||||
- [Preparing the Gitian builder host](#preparing-the-gitian-builder-host) | - [Preparing the Gitian builder host](#preparing-the-gitian-builder-host) | ||||
- [Setting up the Gitian image](#setting-up-the-gitian-image) | - [Getting and building the inputs](#getting-and-building-the-inputs) | ||||
- [Building Bitcoin ABC](#building-bitcoin-abc) | - [Building Bitcoin Core](#building-bitcoin-core) | ||||
- [Building an alternative repository](#building-an-alternative-repository) | |||||
- [Signing externally](#signing-externally) | |||||
- [Uploading signatures](#uploading-signatures) | |||||
Preparing the Gitian builder host | Preparing the Gitian builder host | ||||
--------------------------------- | --------------------------------- | ||||
The first step is to prepare the host environment that will be used to perform | The first step is to prepare the host environment that will be used to perform the Gitian builds. | ||||
the Gitian builds. This guide explains how to set up the environment, and how | This guide explains how to set up the environment, and how to start the builds. | ||||
to start the builds. | |||||
The gitian build is easiest performed under Ubuntu Xenial. This guide will | |||||
focus on creating and using a vagrant box. However, you may run the provided | |||||
provision script on your favorite clean VM image using any virtualization | |||||
option, or a baremetal linux machine. If you wish to setup machine using | |||||
another technology, please see the provided provisioning script for gitian in | |||||
`contrib/gitian/provisioner.sh` | |||||
Requirements: | |||||
- A machine with at least 64b of disk space | |||||
- 16GB of RAM | |||||
- Several installed tools: | |||||
- [Vagrant](https://www.vagrantup.com) | |||||
- [Packer](https://www.packer.io) | |||||
- [Virtualbox](https://www.virtualbox.org) | |||||
After you have installed each of these tools, you will need to create an | |||||
ubuntu xenial vagrant "box." This is most easily done using the [box-cutter | |||||
project](https://github.com/boxcutter/ubuntu). (Note: Canonical provides a | |||||
vagrant box, however its disk space is insufficient for this guide.) | |||||
```bash | Gitian builds are known to be working on recent versions of Debian, Ubuntu and Fedora. | ||||
pushd | If your machine is already running one of those operating systems, you can perform Gitian builds on the actual hardware. | ||||
cd /tmp/ | Alternatively, you can install one of the supported operating systems in a virtual machine. | ||||
git clone https://github.com/boxcutter/ubuntu.git | |||||
cd ubuntu | |||||
git checkout 7d1820c186d76122445c092bc2b872a8a94166ce | |||||
packer build -var-file=ubuntu1604.json -only=virtualbox-iso ubuntu.json | |||||
vagrant box add --name abc-xenial box/virtualbox/ubuntu1604-0.1.0.box | |||||
popd | |||||
``` | |||||
After completion you should be able to run add the box to vagrant as "abc-xenial" | You can create the virtual machine using [vagrant](./gitian-building/gitian-building-vagrant.md) or chose to setup the VM manually. | ||||
using `vagrant box add --name abc-xenial <path_to_packer_output.box>` | |||||
The final step for running vagrant is: | Any kind of virtualization can be used, for example: | ||||
- [VirtualBox](https://www.virtualbox.org/) (covered by this guide) | |||||
- [KVM](http://www.linux-kvm.org/page/Main_Page) | |||||
- [LXC](https://linuxcontainers.org/) | |||||
```bash | Please refer to the following documents to set up the operating systems and Gitian. | ||||
cd contrib/gitian/ | |||||
vagrant up | |||||
vagrant ssh | |||||
``` | |||||
This should drop you into a Xenial prompt as the user `vagrant`. | | | Debian | Fedora | | ||||
|-----------------------------------|------------------------------------------------------------------------------------|------------------------------------------------------------------------------------| | |||||
| Setup virtual machine (optional) | [Create Debian VirtualBox](./gitian-building/gitian-building-create-vm-debian.md) | [Create Fedora VirtualBox](./gitian-building/gitian-building-create-vm-fedora.md) | | |||||
| Setup Gitian | [Setup Gitian on Debian](./gitian-building/gitian-building-setup-gitian-debian.md) | [Setup Gitian on Fedora](./gitian-building/gitian-building-setup-gitian-fedora.md) | | |||||
Setting up the Gitian image | Note that a version of `lxc-execute` higher or equal to 2.1.1 is required. | ||||
--------------------------- | You can check the version with `lxc-execute --version`. | ||||
Gitian needs a virtual image of the operating system to build in. Currently | Non-Debian / Ubuntu, Manual and Offline Building | ||||
this is Ubuntu Xenial x86_64. This image will be copied and used every time | ------------------------------------------------ | ||||
that a build is started to make sure that the build is deterministic. Creating | The instructions below use the automated script [gitian-build.py](https://github.com/Bitcoin-ABC/bitcoin-abc/blob/master/contrib/gitian-build.py) which only works in Debian/Ubuntu. For manual steps and instructions for fully offline signing, see [this guide](./gitian-building/gitian-building-manual.md). | ||||
the image will take a while, but only has to be done once. | |||||
Execute the following as user `vagrant`: | MacOS code signing | ||||
------------------ | |||||
In order to sign builds for MacOS, you need to download the free SDK and extract a file. The steps are described [here](./gitian-building/gitian-building-mac-os-sdk.md). | |||||
It is possible to download the resulting archive directly for users that desire to do so: | |||||
```bash | ```bash | ||||
cd gitian-builder | curl -LO https://storage.googleapis.com/f4936e83b2dcbca742be51fb9692b153/MacOSX10.11.sdk.tar.gz | ||||
./bin/make-base-vm --lxc --distro debian --suite stretch --arch amd64 | echo "4732b52b5ebe300c8c91cbeed6d19d59c1ff9c56c7a1dd6cfa518b9c2c72abde MacOSX10.11.sdk.tar.gz" | sha256sum -c | ||||
``` | ``` | ||||
There will be a lot of warnings printed during the build of the image. These | Alternatively, you can skip the OSX build by adding `--os=lw` below. | ||||
can be ignored. | |||||
Building Bitcoin ABC | Initial Gitian Setup | ||||
-------------------- | -------------------- | ||||
The `gitian-build.py` script will checkout different release tags, so it's best to copy it: | |||||
To build Bitcoin ABC (for Linux, OS X and Windows) run the following commands: | |||||
```bash | ```bash | ||||
URL=https://github.com/bitcoin-abc/bitcoin-abc.git | cp bitcoin-abc/contrib/gitian-build.py . | ||||
COMMIT=v0.16.0 # or whatever release tag you wish | |||||
# Note the path to descriptors assumes vagrant was used. These files are within the ABC repository normally. | |||||
./bin/gbuild --commit bitcoin=${COMMIT} --url bitcoin=${URL} /vagrant/contrib/gitian-descriptors/gitian-linux.yml | |||||
# Note: If you plan on signing the binaries generated during this process, be | |||||
# sure to copy them otherwise they will be overwritten by the next gbuild call. | |||||
cp -r ./build/out/* /vagrant/gitian/linux | |||||
# Also copy the manifest files in the same manner: | |||||
cp ./result/bitcoin-abc-*-linux-res.yml /vagrant/gitian/linux/ | |||||
./bin/gbuild --commit bitcoin=${COMMIT} --url bitcoin=${URL} /vagrant/contrib/gitian-descriptors/gitian-win.yml | |||||
cp -r ./build/out/* /vagrant/gitian/win | |||||
cp ./result/bitcoin-abc-*-win-res.yml /vagrant/gitian/win/ | |||||
./bin/gbuild --commit bitcoin=${COMMIT} --url bitcoin=${URL} /vagrant/contrib/gitian-descriptors/gitian-osx.yml | |||||
cp -r ./build/out/* /vagrant/gitian/osx | |||||
cp ./result/bitcoin-abc-*-osx-res.yml /vagrant/gitian/osx/ | |||||
``` | ``` | ||||
Note on the OSX build: If you encounter an error about a missing MacOSX10.11.sdk.tar.gz, then follow these steps: | You only need to do this once: | ||||
``` | |||||
cd ./inputs | |||||
curl -LO https://storage.googleapis.com/f4936e83b2dcbca742be51fb9692b153/MacOSX10.11.sdk.tar.gz | |||||
``` | |||||
Note: For executing gitian builds on local changes, change URL and COMMIT: | |||||
```bash | ```bash | ||||
URL=/vagrant/ | ./gitian-build.py --setup satoshi 0.18.5 | ||||
COMMIT=<git-commit-hash> # replace <git-commit-hash> with your latest changes | |||||
``` | ``` | ||||
This may take some time as it will build all the dependencies needed for each | Where `satoshi` is your Github name and `0.18.5` is the most recent tag (without `v`). | ||||
descriptor. These dependencies will be cached after a successful build to | |||||
avoid rebuilding them when possible. | Build binaries | ||||
-------------- | |||||
Windows and OSX have code signed binaries, but those won't be available until a few developers have gitian signed the non-codesigned binaries. | |||||
At any time you can check the package installation and build progress with | To build the most recent tag: | ||||
```bash | ```bash | ||||
tail -f var/install.log | ./gitian-build.py --detach-sign --no-commit -b satoshi 0.18.5 | ||||
tail -f var/build.log | |||||
``` | ``` | ||||
Output from `gbuild` will look something like | To speed up the build, use `-j 5 -m 5000` as the first arguments, where `5` is the number of CPU's you allocated to the VM plus one, and 5000 is a little bit less than then the MB's of RAM you allocated. | ||||
Initialized empty Git repository in /home/vagrant/gitian-builder/inputs/bitcoin/.git/ | If all went well, this produces a number of (uncommited) `.assert` files in the gitian.sigs repository. | ||||
remote: Counting objects: 57959, done. | |||||
remote: Total 57959 (delta 0), reused 0 (delta 0), pack-reused 57958 | You need to copy these uncommited changes to your host machine, where you can sign them: | ||||
Receiving objects: 100% (57959/57959), 53.76 MiB | 484.00 KiB/s, done. | |||||
Resolving deltas: 100% (41590/41590), done. | ```bash | ||||
From https://github.com/bitcoin/bitcoin | export NAME=satoshi | ||||
... (new tags, new branch etc) | gpg --output $VERSION-linux/$NAME/bitcoin-abc-linux-0.18.5-build.assert.sig --detach-sign 0.18.5-linux/$NAME/bitcoin-abc-linux-0.18.5-build.assert | ||||
--- Building for trusty amd64 --- | gpg --output $VERSION-osx-unsigned/$NAME/bitcoin-abc-osx-0.18.5-build.assert.sig --detach-sign 0.18.5-osx-unsigned/$NAME/bitcoin-abc-osx-0.18.5-build.assert | ||||
Stopping target if it is up | gpg --output $VERSION-win-unsigned/$NAME/bitcoin-abc-win-0.18.5-build.assert.sig --detach-sign 0.18.5-win-unsigned/$NAME/bitcoin-abc-win-0.18.5-build.assert | ||||
Making a new image copy | ``` | ||||
stdin: is not a tty | |||||
Starting target | |||||
Checking if target is up | |||||
Preparing build environment | |||||
Updating apt-get repository (log in var/install.log) | |||||
Installing additional packages (log in var/install.log) | |||||
Grabbing package manifest | |||||
stdin: is not a tty | |||||
Creating build script (var/build-script) | |||||
lxc-start: Connection refused - inotify event with no name (mask 32768) | |||||
Running build script (log in var/build.log) |