No OneTemporary
Actions

Size

19 KB

Subscribers

None

View Options

	diff --git a/doc/gitian-building.md b/doc/gitian-building.md
	index 13f8ad316b..791f209bb1 100644
	--- a/doc/gitian-building.md
	+++ b/doc/gitian-building.md
	@@ -1,477 +1,477 @@
	Gitian building
	================

	Setup instructions for a Gitian build of Bitcoin using a Debian VM or physical system.

	Gitian is the deterministic build process that is used to build the Bitcoin
	Core executables. It provides a way to be reasonably sure that the
	executables are really built from the source on GitHub. It also makes sure that
	the same, tested dependencies are used and statically built into the executable.

	Multiple developers build the source code by following a specific descriptor
	("recipe"), cryptographically sign the result, and upload the resulting signature.
	These results are compared and only if they match, the build is accepted and uploaded
	to bitcoin.org.

	More independent Gitian builders are needed, which is why this guide exists.
	It is preferred you follow these steps yourself instead of using someone else's
	VM image to avoid 'contaminating' the build.

	Table of Contents
	------------------

	- [Create a new VirtualBox VM](#create-a-new-virtualbox-vm)
	- [Connecting to the VM](#connecting-to-the-vm)
	- [Setting up Debian for Gitian building](#setting-up-debian-for-gitian-building)
	- [Installing Gitian](#installing-gitian)
	- [Setting up the Gitian image](#setting-up-the-gitian-image)
	- [Getting and building the inputs](#getting-and-building-the-inputs)
	- [Building Bitcoin](#building-bitcoin)
	- [Building an alternative repository](#building-an-alternative-repository)
	- [Signing externally](#signing-externally)
	- [Uploading signatures](#uploading-signatures)

	Preparing the Gitian builder host
	---------------------------------

	The first step is to prepare the host environment that will be used to perform the Gitian builds.
	This guide explains how to set up the environment, and how to start the builds.

	Debian Linux was chosen as the host distribution because it has a lightweight install (in contrast to Ubuntu) and is readily available.
	Any kind of virtualization can be used, for example:
	- [VirtualBox](https://www.virtualbox.org/) (covered by this guide)
	- [KVM](http://www.linux-kvm.org/page/Main_Page)
	- [LXC](https://linuxcontainers.org/), see also [Gitian host docker container](https://github.com/gdm85/tenku/tree/master/docker/gitian-bitcoin-host/README.md).

	You can also install Gitian on actual hardware instead of using virtualization.

	Create a new VirtualBox VM
	---------------------------
	In the VirtualBox GUI click "New" and choose the following parameters in the wizard:

	![](gitian-building/create_new_vm.png)

	- Type: Linux, Debian (64-bit)

	![](gitian-building/create_vm_memsize.png)

	- Memory Size: at least 1024MB, anything less will really slow down the build.

	![](gitian-building/create_vm_hard_disk.png)

	- Hard Disk: Create a virtual hard disk now

	![](gitian-building/create_vm_hard_disk_file_type.png)

	- Hard Disk file type: Use the default, VDI (VirtualBox Disk Image)

	![](gitian-building/create_vm_storage_physical_hard_disk.png)

	- Storage on physical hard disk: Dynamically Allocated

	![](gitian-building/create_vm_file_location_size.png)

	- File location and size: at least 40GB; as low as 20GB may be possible, but better to err on the safe side
	- Click `Create`

	After creating the VM, we need to configure it.

	- Click the `Settings` button, then go to the `Network` tab. Adapter 1 should be attached to `NAT`.

	![](gitian-building/network_settings.png)

	- Click `Advanced`, then `Port Forwarding`. We want to set up a port through which we can reach the VM to get files in and out.
	- Create a new rule by clicking the plus icon.

	![](gitian-building/port_forwarding_rules.png)

	- Set up the new rule the following way:
	- Name: `SSH`
	- Protocol: `TCP`
	- Leave Host IP empty
	- Host Port: `22222`
	- Leave Guest IP empty
	- Guest Port: `22`

	- Click `Ok` twice to save.

	Get the [Debian 8.x net installer](http://cdimage.debian.org/debian-cd/8.4.0/amd64/iso-cd/debian-8.4.0-amd64-netinst.iso) (a more recent minor version should also work, see also [Debian Network installation](https://www.debian.org/CD/netinst/)).
	This DVD image can be validated using a SHA256 hashing tool, for example on
	Unixy OSes by entering the following in a terminal:

	echo "7a6b418e6a4ee3ca75dda04d79ed96c9e2c33bb0c703ca7e40c6374ab4590748 debian-8.4.0-amd64-netinst.iso" \| sha256sum -c
	# (must return OK)

	Then start the VM. On the first launch you will be asked for a CD or DVD image. Choose the downloaded iso.

	![](gitian-building/select_startup_disk.png)

	Installing Debian
	------------------

	This section will explain how to install Debian on the newly created VM.

	- Choose the non-graphical installer. We do not need the graphical environment; it will only increase installation time and disk usage.

	![](gitian-building/debian_install_1_boot_menu.png)

	Note: Navigating in the Debian installer:
	To keep a setting at the default and proceed, just press `Enter`.
	To select a different button, press `Tab`.

	- Choose locale and keyboard settings (doesn't matter, you can just go with the defaults or select your own information)

	![](gitian-building/debian_install_2_select_a_language.png)
	![](gitian-building/debian_install_3_select_location.png)
	![](gitian-building/debian_install_4_configure_keyboard.png)

	- The VM will detect network settings using DHCP, this should all proceed automatically
	- Configure the network:
	- Hostname `debian`.
	- Leave domain name empty.

	![](gitian-building/debian_install_5_configure_the_network.png)

	- Choose a root password and enter it twice (remember it for later)

	![](gitian-building/debian_install_6a_set_up_root_password.png)

	- Name the new user `debian` (the full name doesn't matter, you can leave it empty)
	- Set the account username as `debian`

	![](gitian-building/debian_install_7_set_up_user_fullname.png)
	![](gitian-building/debian_install_8_set_up_username.png)

	- Choose a user password and enter it twice (remember it for later)

	![](gitian-building/debian_install_9_user_password.png)

	- The installer will set up the clock using a time server; this process should be automatic
	- Set up the clock: choose a time zone (depends on the locale settings that you picked earlier; specifics don't matter)

	![](gitian-building/debian_install_10_configure_clock.png)

	- Disk setup
	- Partitioning method: Guided - Use the entire disk

	![](gitian-building/debian_install_11_partition_disks.png)

	- Select disk to partition: SCSI1 (0,0,0)

	![](gitian-building/debian_install_12_choose_disk.png)

	- Partition Disks -> All files in one partition

	![](gitian-building/all_files_in_one_partition.png)

	- Finish partitioning and write changes to disk -> Yes (`Tab`, `Enter` to select the `Yes` button)

	![](gitian-building/debian_install_14_finish.png)
	![](gitian-building/debian_install_15_write_changes.png)

	- The base system will be installed, this will take a minute or so
	- Choose a mirror (any will do)

	![](gitian-building/debian_install_16_choose_a_mirror.png)

	- Enter proxy information (unless you are on an intranet, leave this empty)

	![](gitian-building/debian_install_18_proxy_settings.png)

	- Wait a bit while 'Select and install software' runs
	- Participate in popularity contest -> No
	- Choose software to install. We need just the base system.
	- Make sure only 'SSH server' and 'Standard System Utilities' are checked
	- Uncheck 'Debian Desktop Environment' and 'Print Server'

	![](gitian-building/debian_install_19_software_selection.png)

	- Install the GRUB boot loader to the master boot record? -> Yes

	![](gitian-building/debian_install_20_install_grub.png)

	- Device for boot loader installation -> ata-VBOX_HARDDISK

	![](gitian-building/debian_install_21_install_grub_bootloader.png)

	- Installation Complete -> Continue
	- After installation, the VM will reboot and you will have a working Debian VM. Congratulations!

	![](gitian-building/debian_install_22_finish_installation.png)


	After Installation
	-------------------
	The next step in the guide involves logging in as root via SSH.
	SSH login for root users is disabled by default, so we'll enable that now.

	Login to the VM using username `root` and the root password you chose earlier.
	You'll be presented with a screen similar to this.

	![](gitian-building/debian_root_login.png)

	Type:

	```
	sed -i 's/^PermitRootLogin.*/PermitRootLogin yes/' /etc/ssh/sshd_config
	```
	and press enter. Then,
	```
	/etc/init.d/ssh restart
	```
	and enter to restart SSH. Logout by typing 'logout' and pressing 'enter'.

	Connecting to the VM
	----------------------

	After the VM has booted you can connect to it using SSH, and files can be copied from and to the VM using a SFTP utility.
	Connect to `localhost`, port `22222` (or the port configured when installing the VM).
	On Windows you can use [putty](http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html) and [WinSCP](http://winscp.net/eng/index.php).

	For example, to connect as `root` from a Linux command prompt use

	$ ssh root@localhost -p 22222
	The authenticity of host '[localhost]:22222 ([127.0.0.1]:22222)' can't be established.
	RSA key fingerprint is ae:f5:c8:9f:17:c6:c7:1b:c2:1b:12:31:1d:bb:d0:c7.
	Are you sure you want to continue connecting (yes/no)? yes
	Warning: Permanently added '[localhost]:22222' (RSA) to the list of known hosts.
	root@localhost's password: (enter root password configured during install)

	The programs included with the Debian GNU/Linux system are free software;
	the exact distribution terms for each program are described in the
	individual files in /usr/share/doc/*/copyright.

	Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
	permitted by applicable law.
	root@debian:~#

	Replace `root` with `debian` to log in as user.

	Setting up Debian for Gitian building
	--------------------------------------

	In this section we will be setting up the Debian installation for Gitian building.

	First we need to log in as `root` to set up dependencies and make sure that our
	user can use the sudo command. Type/paste the following in the terminal:

	```bash
	apt-get install git ruby sudo apt-cacher-ng qemu-utils debootstrap lxc python-cheetah parted kpartx bridge-utils make ubuntu-archive-keyring curl
	adduser debian sudo
	```

	Then set up LXC and the rest with the following, which is a complex jumble of settings and workarounds:

	```bash
	# the version of lxc-start in Debian needs to run as root, so make sure
	# that the build script can execute it without providing a password
	echo "%sudo ALL=NOPASSWD: /usr/bin/lxc-start" > /etc/sudoers.d/gitian-lxc
	echo "%sudo ALL=NOPASSWD: /usr/bin/lxc-execute" >> /etc/sudoers.d/gitian-lxc
	# make /etc/rc.local script that sets up bridge between guest and host
	echo '#!/bin/sh -e' > /etc/rc.local
	echo 'brctl addbr br0' >> /etc/rc.local
	echo 'ifconfig br0 10.0.3.2/24 up' >> /etc/rc.local
	echo 'iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE' >> /etc/rc.local
	echo 'echo 1 > /proc/sys/net/ipv4/ip_forward' >> /etc/rc.local
	echo 'exit 0' >> /etc/rc.local
	# make sure that USE_LXC is always set when logging in as debian,
	# and configure LXC IP addresses
	echo 'export USE_LXC=1' >> /home/debian/.profile
	echo 'export GITIAN_HOST_IP=10.0.3.2' >> /home/debian/.profile
	echo 'export LXC_GUEST_IP=10.0.3.5' >> /home/debian/.profile
	reboot
	```

	At the end the VM is rebooted to make sure that the changes take effect. The steps in this
	section only need to be performed once.

	Installing Gitian
	------------------

	Re-login as the user `debian` that was created during installation.
	The rest of the steps in this guide will be performed as that user.

	There is no `python-vm-builder` package in Debian, so we need to install it from source ourselves,

	```bash
	wget http://archive.ubuntu.com/ubuntu/pool/universe/v/vm-builder/vm-builder_0.12.4+bzr494.orig.tar.gz
	echo "76cbf8c52c391160b2641e7120dbade5afded713afaa6032f733a261f13e6a8e vm-builder_0.12.4+bzr494.orig.tar.gz" \| sha256sum -c
	# (verification -- must return OK)
	tar -zxvf vm-builder_0.12.4+bzr494.orig.tar.gz
	cd vm-builder-0.12.4+bzr494
	sudo python setup.py install
	cd ..
	```

	Note: When sudo asks for a password, enter the password for the user debian not for root.

	Clone the git repositories for bitcoin and Gitian.

	```bash
	git clone https://github.com/devrandom/gitian-builder.git
	git clone https://github.com/bitcoin/bitcoin
	git clone https://github.com/bitcoin-core/gitian.sigs.git
	```

	Setting up the Gitian image
	-------------------------

	Gitian needs a virtual image of the operating system to build in.
	Currently this is Ubuntu Trusty x86_64.
	This image will be copied and used every time that a build is started to
	make sure that the build is deterministic.
	Creating the image will take a while, but only has to be done once.

	Execute the following as user `debian`:

	```bash
	cd gitian-builder
	bin/make-base-vm --lxc --arch amd64 --suite trusty
	```

	There will be a lot of warnings printed during the build of the image. These can be ignored.

	Note: When sudo asks for a password, enter the password for the user debian not for root.

	Getting and building the inputs
	--------------------------------

	Follow the instructions in [doc/release-process.md](release-process.md#fetch-and-build-inputs-first-time-or-when-dependency-versions-change)
	in the bitcoin repository under 'Fetch and build inputs' to install sources which require
	manual intervention. Also optionally follow the next step: 'Seed the Gitian sources cache
	and offline git repositories' which will fetch the remaining files required for building
	offline.

	Building Bitcoin
	----------------

	To build Bitcoin (for Linux, OS X and Windows) just follow the steps under 'perform
	Gitian builds' in [doc/release-process.md](release-process.md#perform-gitian-builds) in the bitcoin repository.

	This may take some time as it will build all the dependencies needed for each descriptor.
	These dependencies will be cached after a successful build to avoid rebuilding them when possible.

	At any time you can check the package installation and build progress with

	```bash
	tail -f var/install.log
	tail -f var/build.log
	```

	Output from `gbuild` will look something like

	Initialized empty Git repository in /home/debian/gitian-builder/inputs/bitcoin/.git/
	remote: Counting objects: 57959, done.
	remote: Total 57959 (delta 0), reused 0 (delta 0), pack-reused 57958
	Receiving objects: 100% (57959/57959), 53.76 MiB \| 484.00 KiB/s, done.
	Resolving deltas: 100% (41590/41590), done.
	From https://github.com/bitcoin/bitcoin
	... (new tags, new branch etc)
	--- Building for trusty amd64 ---
	Stopping target if it is up
	Making a new image copy
	stdin: is not a tty
	Starting target
	Checking if target is up
	Preparing build environment
	Updating apt-get repository (log in var/install.log)
	Installing additional packages (log in var/install.log)
	Grabbing package manifest
	stdin: is not a tty
	Creating build script (var/build-script)
	lxc-start: Connection refused - inotify event with no name (mask 32768)
	Running build script (log in var/build.log)

	Building an alternative repository
	-----------------------------------

	If you want to do a test build of a pull on GitHub it can be useful to point
	the Gitian builder at an alternative repository, using the same descriptors
	and inputs.

	For example:
	```bash
	URL=https://github.com/laanwj/bitcoin.git
	COMMIT=2014_03_windows_unicode_path
	./bin/gbuild --commit bitcoin=${COMMIT} --url bitcoin=${URL} ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
	./bin/gbuild --commit bitcoin=${COMMIT} --url bitcoin=${URL} ../bitcoin/contrib/gitian-descriptors/gitian-win.yml
	./bin/gbuild --commit bitcoin=${COMMIT} --url bitcoin=${URL} ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml
	```

	Building fully offline
	-----------------------

	For building fully offline including attaching signatures to unsigned builds, the detached-sigs repository
	and the bitcoin git repository with the desired tag must both be available locally, and then gbuild must be
	told where to find them. It also requires an apt-cacher-ng which is fully-populated but set to offline mode, or
	manually disabling gitian-builder's use of apt-get to update the VM build environment.

	To configure apt-cacher-ng as an offline cacher, you will need to first populate its cache with the relevant
	files. You must additionally patch target-bin/bootstrap-fixup to set its apt sources to something other than
	plain archive.ubuntu.com: us.archive.ubuntu.com works.

	So, if you use LXC:

	```bash
	export PATH="$PATH":/path/to/gitian-builder/libexec
	export USE_LXC=1
	cd /path/to/gitian-builder
	./libexec/make-clean-vm --suite trusty --arch amd64

	LXC_ARCH=amd64 LXC_SUITE=trusty on-target -u root apt-get update
	LXC_ARCH=amd64 LXC_SUITE=trusty on-target -u root \
	-e DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends -y install \
	$( sed -ne '/^packages:/,/[^-] ./ {/^- ./{s/"//g;s/- //;p}}' ../bitcoin/contrib/gitian-descriptors/*\|sort\|uniq )
	LXC_ARCH=amd64 LXC_SUITE=trusty on-target -u root apt-get -q -y purge grub
	LXC_ARCH=amd64 LXC_SUITE=trusty on-target -u root -e DEBIAN_FRONTEND=noninteractive apt-get -y dist-upgrade
	```

	And then set offline mode for apt-cacher-ng:

	```
	/etc/apt-cacher-ng/acng.conf
	[...]
	Offlinemode: 1
	[...]

	service apt-cacher-ng restart
	```

	Then when building, override the remote URLs that gbuild would otherwise pull from the Gitian descriptors::
	```bash

	cd /some/root/path/
	git clone https://github.com/bitcoin-core/bitcoin-detached-sigs.git

	-BTCPATH=/some/root/path/bitcoin.git
	-SIGPATH=/some/root/path/bitcoin-detached-sigs.git
	+BTCPATH=/some/root/path/bitcoin
	+SIGPATH=/some/root/path/bitcoin-detached-sigs

	./bin/gbuild --url bitcoin=${BTCPATH},signature=${SIGPATH} ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml
	```

	Signing externally
	-------------------

	If you want to do the PGP signing on another device, that's also possible; just define `SIGNER` as mentioned
	and follow the steps in the build process as normal.

	gpg: skipped "laanwj": secret key not available

	When you execute `gsign` you will get an error from GPG, which can be ignored. Copy the resulting `.assert` files
	in `gitian.sigs` to your signing machine and do

	```bash
	gpg --detach-sign ${VERSION}-linux/${SIGNER}/bitcoin-linux-build.assert
	gpg --detach-sign ${VERSION}-win/${SIGNER}/bitcoin-win-build.assert
	gpg --detach-sign ${VERSION}-osx-unsigned/${SIGNER}/bitcoin-osx-build.assert
	```

	This will create the `.sig` files that can be committed together with the `.assert` files to assert your
	Gitian build.

	Uploading signatures
	---------------------

	After building and signing you can push your signatures (both the `.assert` and `.assert.sig` files) to the
	[bitcoin-core/gitian.sigs](https://github.com/bitcoin-core/gitian.sigs/) repository, or if that's not possible create a pull
	request. You can also mail the files to Wladimir (laanwj@gmail.com) and he will commit them.

File Metadata

Mime Type: text/x-diff
Expires: Sun, Mar 2, 10:43 (1 d, 11 h)
Storage Engine: blob
Storage Format: Raw Data
Storage Handle: 5186065
Default Alt Text: (19 KB)

No OneTemporaryActions

View Options

File Metadata

Event Timeline

No OneTemporary
Actions