Page MenuHomePhabricator
Files F12910247

No OneTemporary
Actions

View Options

diff --git a/contrib/devtools/README.md b/contrib/devtools/README.md
index 3d0171e1b..938bd89d3 100644
--- a/contrib/devtools/README.md
+++ b/contrib/devtools/README.md
@@ -1,130 +1,130 @@
Contents
========
This directory contains tools for developers working on this repository.
copyright\_header.py
====================
Provides utilities for managing copyright headers of `The Bitcoin developers`
in repository source files. It has three subcommands:
```
$ ./copyright_header.py report <base_directory> [verbose]
$ ./copyright_header.py update <base_directory>
$ ./copyright_header.py insert <file>
```
Running these subcommands without arguments displays a usage string.
copyright\_header.py report \<base\_directory\> [verbose]
---------------------------------------------------------
Produces a report of all copyright header notices found inside the source files
of a repository. Useful to quickly visualize the state of the headers.
Specifying `verbose` will list the full filenames of files of each category.
copyright\_header.py update \<base\_directory\> [verbose]
---------------------------------------------------------
Updates all the copyright headers of `The Bitcoin developers` which were changed
in a year more recent than is listed. For example:
```
// Copyright (c) <firstYear>-<lastYear> The Bitcoin developers
```
will be updated to:
```
// Copyright (c) <firstYear>-<lastModifiedYear> The Bitcoin developers
```
where `<lastModifiedYear>` is obtained from the `git log` history.
This subcommand also handles copyright headers that have only a single year. In
those cases:
```
// Copyright (c) <year> The Bitcoin developers
```
will be updated to:
```
// Copyright (c) <year>-<lastModifiedYear> The Bitcoin developers
```
where the update is appropriate.
copyright\_header.py insert \<file\>
------------------------------------
Inserts a copyright header for `The Bitcoin developers` at the top of the file
in either Python or C++ style as determined by the file extension. If the file
is a Python file and it has `#!` starting the first line, the header is
inserted in the line below it.
The copyright dates will be set to be `<year_introduced>-<current_year>` where
`<year_introduced>` is according to the `git log` history. If
`<year_introduced>` is equal to `<current_year>`, it will be set as a single
year rather than two hyphenated years.
If the file already has a copyright for `The Bitcoin developers`, the script
will exit.
gen-manpages.sh
===============
A small script to automatically create manpages in ../../doc/man by running the release binaries with the -help option.
This requires help2man which can be found at: https://www.gnu.org/software/help2man/
With in-tree builds this tool can be run from any directory within the
repostitory. To use this tool with out-of-tree builds set `BUILDDIR`. For
example:
```bash
BUILDDIR=$PWD/build contrib/devtools/gen-manpages.sh
```
optimize-pngs.py
================
A script to optimize png files in the bitcoin
repository (requires pngcrush).
security-check.py and test-security-check.py
============================================
Perform basic ELF security checks on a series of executables.
symbol-check.py
===============
A script to check that the (Linux) executables produced by gitian only contain
allowed gcc, glibc and libstdc++ version symbols. This makes sure they are
still compatible with the minimum supported Linux distribution versions.
Example usage after a gitian build:
- find ../gitian-builder/build -type f -executable | xargs python3 contrib/devtools/symbol-check.py
+ find contrib/gitian-builder/build -type f -executable | xargs python3 contrib/devtools/symbol-check.py
If only supported symbols are used the return value will be 0 and the output will be empty.
If there are 'unsupported' symbols, the return value will be 1 a list like this will be printed:
.../64/test_bitcoin: symbol memcpy from unsupported version GLIBC_2.14
.../64/test_bitcoin: symbol __fdelt_chk from unsupported version GLIBC_2.15
.../64/test_bitcoin: symbol std::out_of_range::~out_of_range() from unsupported version GLIBCXX_3.4.15
.../64/test_bitcoin: symbol _ZNSt8__detail15_List_nod from unsupported version GLIBCXX_3.4.15
update-translations.py
======================
Run this script from the root of the repository to update all translations from transifex.
It will do the following automatically:
- Fetch all translations
- Post-process them into valid and committable format
- Add missing translations to the build system (TODO)
See doc/translation-process.md for more information.
circular-dependencies.py
========================
Run this script from the root of the source tree (`src/`) to find circular dependencies in the source code.
This looks only at which files include other files, treating the `.cpp` and `.h` file as one unit.
Example usage:
cd <project root>/src
../contrib/devtools/circular-dependencies.py {*,*/*,*/*/*}.{h,cpp}
diff --git a/contrib/devtools/symbol-check.py b/contrib/devtools/symbol-check.py
index 34276f37d..92eab6ac6 100755
--- a/contrib/devtools/symbol-check.py
+++ b/contrib/devtools/symbol-check.py
@@ -1,193 +1,193 @@
#!/usr/bin/env python3
# Copyright (c) 2014 Wladimir J. van der Laan
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
'''
A script to check that the (Linux) executables produced by gitian only contain
allowed gcc, glibc and libstdc++ version symbols. This makes sure they are
still compatible with the minimum supported Linux distribution versions.
Example usage:
- find ../gitian-builder/build -type f -executable | xargs python3 contrib/devtools/symbol-check.py
+ find contrib/gitian-builder/build -type f -executable | xargs python3 contrib/devtools/symbol-check.py
'''
import subprocess
import re
import sys
import os
# Debian 8.11 (Jessie) has:
#
# - g++ version 4.9.2 (https://packages.debian.org/search?suite=default&section=all&arch=any&searchon=names&keywords=g%2B%2B)
# - libc version 2.19.18 (https://packages.debian.org/search?suite=default&section=all&arch=any&searchon=names&keywords=libc6)
# - libstdc++ version 4.8.4 (https://packages.debian.org/search?suite=default&section=all&arch=any&searchon=names&keywords=libstdc%2B%2B6)
#
# Ubuntu 14.04 (Trusty Tahr) has:
#
# - g++ version 4.8.2 (https://packages.ubuntu.com/search?suite=trusty&section=all&arch=any&keywords=g%2B%2B&searchon=names)
# - libc version 2.19.0 (https://packages.ubuntu.com/search?suite=trusty&section=all&arch=any&keywords=libc6&searchon=names)
# - libstdc++ version 4.8.2 (https://packages.ubuntu.com/search?suite=trusty&section=all&arch=any&keywords=libstdc%2B%2B&searchon=names)
#
# Taking the minimum of these as our target.
#
# According to GNU ABI document (http://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html) this corresponds to:
# GCC 4.8.0: GCC_4.8.0
# GCC 4.8.0: GLIBCXX_3.4.18, CXXABI_1.3.7
# (glibc) GLIBC_2_19
#
MAX_VERSIONS = {
'GCC': (4, 8, 0),
'CXXABI': (1, 3, 7),
'GLIBCXX': (3, 4, 18),
'GLIBC': (2, 19),
'LIBATOMIC': (1, 0)
}
# See here for a description of _IO_stdin_used:
# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=634261#109
# Ignore symbols that are exported as part of every executable
IGNORE_EXPORTS = {
'_edata', '_end', '__end__', '_init', '__bss_start', '__bss_start__', '_bss_end__', '__bss_end__', '_fini', '_IO_stdin_used', 'stdin', 'stdout', 'stderr',
# Jemalloc exported symbols
'__malloc_hook', 'malloc', 'calloc', 'malloc_usable_size',
'__free_hook', 'free',
'__realloc_hook', 'realloc',
'__memalign_hook', 'memalign', 'posix_memalign', 'aligned_alloc', 'valloc',
# Figure out why we get these symbols exported on xenial.
'_ZNKSt5ctypeIcE8do_widenEc', 'in6addr_any', 'optarg',
'_ZNSt16_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_destroyEv'
}
READELF_CMD = os.getenv('READELF', '/usr/bin/readelf')
CPPFILT_CMD = os.getenv('CPPFILT', '/usr/bin/c++filt')
# Allowed NEEDED libraries
ALLOWED_LIBRARIES = {
# bitcoind and bitcoin-qt
'libgcc_s.so.1', # GCC base support
'libc.so.6', # C library
'libpthread.so.0', # threading
'libanl.so.1', # DNS resolve
'libm.so.6', # math library
'librt.so.1', # real-time (clock)
'libatomic.so.1',
'ld-linux-x86-64.so.2', # 64-bit dynamic linker
'ld-linux.so.2', # 32-bit dynamic linker
'ld-linux-aarch64.so.1', # 64-bit ARM dynamic linker
'ld-linux-armhf.so.3', # 32-bit ARM dynamic linker
# bitcoin-qt only
'libxcb.so.1', # part of X11
'libfontconfig.so.1', # font support
'libfreetype.so.6', # font parsing
'libdl.so.2' # programming interface to dynamic linker
}
ARCH_MIN_GLIBC_VER = {
'80386': (2, 1),
'X86-64': (2, 2, 5),
'ARM': (2, 4),
'AArch64': (2, 17)
}
class CPPFilt(object):
'''
Demangle C++ symbol names.
Use a pipe to the 'c++filt' command.
'''
def __init__(self):
self.proc = subprocess.Popen(
CPPFILT_CMD, stdin=subprocess.PIPE, stdout=subprocess.PIPE, universal_newlines=True)
def __call__(self, mangled):
self.proc.stdin.write(mangled + '\n')
self.proc.stdin.flush()
return self.proc.stdout.readline().rstrip()
def close(self):
self.proc.stdin.close()
self.proc.stdout.close()
self.proc.wait()
def read_symbols(executable, imports=True):
'''
Parse an ELF executable and return a list of (symbol,version) tuples
for dynamic, imported symbols.
'''
p = subprocess.Popen([READELF_CMD, '--dyn-syms', '-W', '-h', executable], stdout=subprocess.PIPE,
stderr=subprocess.PIPE, stdin=subprocess.PIPE, universal_newlines=True)
(stdout, stderr) = p.communicate()
if p.returncode:
raise IOError('Could not read symbols for {}: {}'.format(
executable, stderr.strip()))
syms = []
for line in stdout.splitlines():
line = line.split()
if 'Machine:' in line:
arch = line[-1]
if len(line) > 7 and re.match('[0-9]+:$', line[0]):
(sym, _, version) = line[7].partition('@')
is_import = line[6] == 'UND'
if version.startswith('@'):
version = version[1:]
if is_import == imports:
syms.append((sym, version, arch))
return syms
def check_version(max_versions, version, arch):
if '_' in version:
(lib, _, ver) = version.rpartition('_')
else:
lib = version
ver = '0'
ver = tuple([int(x) for x in ver.split('.')])
if lib not in max_versions:
return False
return ver <= max_versions[lib] or lib == 'GLIBC' and ver <= ARCH_MIN_GLIBC_VER[arch]
def read_libraries(filename):
p = subprocess.Popen([READELF_CMD, '-d', '-W', filename], stdout=subprocess.PIPE,
stderr=subprocess.PIPE, stdin=subprocess.PIPE, universal_newlines=True)
(stdout, stderr) = p.communicate()
if p.returncode:
raise IOError('Error opening file')
libraries = []
for line in stdout.splitlines():
tokens = line.split()
if len(tokens) > 2 and tokens[1] == '(NEEDED)':
match = re.match(
r'^Shared library: \[(.*)\]$', ' '.join(tokens[2:]))
if match:
libraries.append(match.group(1))
else:
raise ValueError('Unparseable (NEEDED) specification')
return libraries
if __name__ == '__main__':
cppfilt = CPPFilt()
retval = 0
for filename in sys.argv[1:]:
# Check imported symbols
for sym, version, arch in read_symbols(filename, True):
if version and not check_version(MAX_VERSIONS, version, arch):
print('{}: symbol {} from unsupported version {}'.format(
filename, cppfilt(sym), version))
retval = 1
# Check exported symbols
for sym, version, arch in read_symbols(filename, False):
if sym in IGNORE_EXPORTS:
continue
print('{}: export of symbol {} not allowed'.format(
filename, cppfilt(sym)))
retval = 1
# Check dependency libraries
for library_name in read_libraries(filename):
if library_name not in ALLOWED_LIBRARIES:
print('{}: NEEDED library {} is not allowed'.format(
filename, library_name))
retval = 1
sys.exit(retval)
diff --git a/contrib/gitian/provisioner.sh b/contrib/gitian/provisioner.sh
index 849084cb8..78e884b33 100644
--- a/contrib/gitian/provisioner.sh
+++ b/contrib/gitian/provisioner.sh
@@ -1,68 +1,64 @@
#!/usr/bin/env bash
export LC_ALL=C
set -e
# User to setup for building. Vagrant is the default
export BUILDUSER=${BUILDUSER:-vagrant}
apt-get update
apt-get install -y git ruby sudo apt-cacher-ng qemu-utils debootstrap \
lxc python-cheetah parted kpartx bridge-utils make curl
# the version of lxc-start in Debian needs to run as root, so make sure
# that the build script can execute it without providing a password
echo "%sudo ALL=NOPASSWD: /usr/bin/lxc-start" > /etc/sudoers.d/gitian-lxc
echo "%sudo ALL=NOPASSWD: /usr/bin/lxc-execute" >> /etc/sudoers.d/gitian-lxc
# make sure that USE_LXC is always set when logging in as vagrant,
# and configure LXC IP addresses
{
echo 'export USE_LXC=1'
echo 'export LXC_BRIDGE=lxcbr0'
echo 'export GITIAN_HOST_IP=10.0.3.1'
echo 'export LXC_GUEST_IP=10.0.3.5'
} >> /home/${BUILDUSER}/.profile
# Setup bridge
echo 'USE_LXC_BRIDGE="true"' > /etc/default/lxc-net
{
echo 'lxc.network.type = veth'
echo 'lxc.network.link = lxcbr0'
echo 'lxc.network.flags = up'
echo 'lxc.network.hwaddr = 00:16:3e:xx:xx:xx'
} > /etc/lxc/default.conf
service lxc-net restart
# chdir into build user home directory
cd /home/${BUILDUSER}/
## Install vm-builder
wget http://archive.ubuntu.com/ubuntu/pool/universe/v/vm-builder/vm-builder_0.12.4+bzr494.orig.tar.gz
echo "76cbf8c52c391160b2641e7120dbade5afded713afaa6032f733a261f13e6a8e vm-builder_0.12.4+bzr494.orig.tar.gz" | sha256sum -c
# (verification -- must return OK)
tar -zxvf vm-builder_0.12.4+bzr494.orig.tar.gz
cd vm-builder-0.12.4+bzr494
sudo python setup.py install
cd ..
-## Install Gitian
-git clone https://github.com/devrandom/gitian-builder.git
-cd gitian-builder
-
git config --global user.email "vagrant@vagrant.com"
git config --global user.name "vagrant"
chown -R ${BUILDUSER}:${BUILDUSER} /home/${BUILDUSER}
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
echo "!!! Provisioning Complete !!!!"
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
echo
echo "As the user ${BUILDUSER} run the following commands to produce a linux build:"
echo "export COMMIT=v0.18.3"
echo "export URL=https://github.com/Bitcoin-ABC/bitcoin-abc.git"
-echo "cd gitian-builder"
+echo "cd contrib/gitian-builder"
echo "./bin/make-base-vm --lxc --distro debian --suite stretch --arch amd64"
echo "./bin/gbuild --commit bitcoin=\${COMMIT} --url bitcoin=\${URL} /vagrant/contrib/gitian-descriptors/gitian-linux.yml"
diff --git a/contrib/teamcity/gitian.sh b/contrib/teamcity/gitian.sh
index e9241874f..0351f47bd 100755
--- a/contrib/teamcity/gitian.sh
+++ b/contrib/teamcity/gitian.sh
@@ -1,47 +1,52 @@
#!/usr/bin/env bash
export LC_ALL=C
set -euxo pipefail
: "${TOPLEVEL:=$(git rev-parse --show-toplevel)}"
: "${BUILD_DIR:=${TOPLEVEL}/build}"
: "${THREADS:=$(nproc || sysctl -n hw.ncpu)}"
COMMIT=$(git -C "${TOPLEVEL}" rev-parse HEAD)
export COMMIT
export USE_LXC=1
export GITIAN_HOST_IP=10.0.3.1
export LXC_BRIDGE=lxcbr0
export LXC_GUEST_IP=10.0.3.5
-cd ~/gitian-builder
+cd "${TOPLEVEL}/contrib/gitian-builder"
+
+# Build the base image if it doesn't already exist
+if [ ! -f "base-buster-amd64" ]; then
+ ./bin/make-base-vm --lxc --arch amd64 --distro debian --suite buster
+fi
if [[ "${OS_NAME}" == "osx" ]]; then
OSX_SDK="MacOSX10.14.sdk.tar.gz"
OSX_SDK_SHA256="2322086a96349db832abbcadea493b79db843553a2e604163238d99fa058a286"
mkdir -p inputs
pushd inputs
if ! echo "${OSX_SDK_SHA256} ${OSX_SDK}" | sha256sum -c; then
rm -f "${OSX_SDK}"
wget https://storage.googleapis.com/27cd7b2a42a430926cc621acdc3bda72a8ed2b0efc080e3/"${OSX_SDK}"
echo "${OSX_SDK_SHA256} ${OSX_SDK}" | sha256sum -c
fi
popd
fi
RESULT_DIR="${BUILD_DIR}/gitian-results"
OS_DIR="${RESULT_DIR}/${OS_NAME}"
mkdir -p "${OS_DIR}"
move_log() {
mv var/install.log "${RESULT_DIR}/"
mv var/build.log "${RESULT_DIR}/"
}
trap "move_log" ERR
./bin/gbuild -j${THREADS} -m3500 --commit bitcoin=${COMMIT} --url bitcoin="${TOPLEVEL}" "${TOPLEVEL}/contrib/gitian-descriptors/gitian-${OS_NAME}.yml"
move_log
mv result/*.yml "${OS_DIR}/"
mv build/out/* "${OS_DIR}/"
diff --git a/doc/gitian-building.md b/doc/gitian-building.md
index 172a5e38c..7b2596f3a 100644
--- a/doc/gitian-building.md
+++ b/doc/gitian-building.md
@@ -1,115 +1,115 @@
Gitian building
===============
*Setup instructions for a Gitian build of Bitcoin ABC using a VM or physical system.*
Gitian is the deterministic build process that is used to build the Bitcoin
ABC executables. It provides a way to be reasonably sure that the
executables are really built from the source on GitHub. It also makes sure that
the same, tested dependencies are used and statically built into the executable.
Multiple developers build the source code by following a specific descriptor
("recipe"), cryptographically sign the result, and upload the resulting signature.
These results are compared and only if they match, the build is accepted and
uploaded to bitcoinabc.org.
More independent Gitian builders are needed, which is why this guide exists.
It is preferred you follow these steps yourself instead of using someone else's
VM image to avoid 'contaminating' the build.
Table of Contents
-----------------
- [Preparing the Gitian builder host](#preparing-the-gitian-builder-host)
- [Getting and building the inputs](#getting-and-building-the-inputs)
- [Building Bitcoin Core](#building-bitcoin-core)
- [Building an alternative repository](#building-an-alternative-repository)
- [Signing externally](#signing-externally)
- [Uploading signatures](#uploading-signatures)
Preparing the Gitian builder host
---------------------------------
The first step is to prepare the host environment that will be used to perform the Gitian builds.
This guide explains how to set up the environment, and how to start the builds.
Gitian builds are known to be working on recent versions of Debian, Ubuntu and Fedora.
If your machine is already running one of those operating systems, you can perform Gitian builds on the actual hardware.
Alternatively, you can install one of the supported operating systems in a virtual machine.
You can create the virtual machine using [vagrant](./gitian-building/gitian-building-vagrant.md) or chose to setup the VM manually.
Any kind of virtualization can be used, for example:
- [VirtualBox](https://www.virtualbox.org/) (covered by this guide)
- [KVM](http://www.linux-kvm.org/page/Main_Page)
- [LXC](https://linuxcontainers.org/)
Please refer to the following documents to set up the operating systems and Gitian.
| | Debian | Fedora |
|-----------------------------------|------------------------------------------------------------------------------------|------------------------------------------------------------------------------------|
| Setup virtual machine (optional) | [Create Debian VirtualBox](./gitian-building/gitian-building-create-vm-debian.md) | [Create Fedora VirtualBox](./gitian-building/gitian-building-create-vm-fedora.md) |
| Setup Gitian | [Setup Gitian on Debian](./gitian-building/gitian-building-setup-gitian-debian.md) | [Setup Gitian on Fedora](./gitian-building/gitian-building-setup-gitian-fedora.md) |
Note that a version of `lxc-execute` higher or equal to 2.1.1 is required.
You can check the version with `lxc-execute --version`.
Non-Debian / Ubuntu, Manual and Offline Building
------------------------------------------------
The instructions below use the automated script [gitian-build.py](https://github.com/Bitcoin-ABC/bitcoin-abc/blob/master/contrib/gitian-build.py) which only works in Debian/Ubuntu. For manual steps and instructions for fully offline signing, see [this guide](./gitian-building/gitian-building-manual.md).
MacOS code signing
------------------
In order to sign builds for MacOS, you need to download the free SDK and extract a file. The steps are described [here](./gitian-building/gitian-building-mac-os-sdk.md).
It is possible to download the resulting archive directly for users that desire to do so:
```bash
-cd ~/gitian-builder
+cd bitcoin-abc/contrib/gitian-builder
curl -LO https://storage.googleapis.com/f4936e83b2dcbca742be51fb9692b153/MacOSX10.11.sdk.tar.gz
echo "4732b52b5ebe300c8c91cbeed6d19d59c1ff9c56c7a1dd6cfa518b9c2c72abde MacOSX10.11.sdk.tar.gz" | sha256sum -c
mkdir -p inputs
mv MacOSX10.11.sdk.tar.gz inputs
```
Alternatively, you can skip the macOS build by adding `--os=lw` below.
Initial Gitian Setup
--------------------
The `gitian-build.py` script will checkout different release tags, so it's best to copy it:
```bash
cp bitcoin-abc/contrib/gitian-build.py .
```
You only need to do this once:
```bash
./gitian-build.py --setup satoshi 0.18.5
```
Where `satoshi` is your Github name and `0.18.5` is the most recent tag (without `v`).
Build binaries
--------------
Windows and macOS have code signed binaries, but those won't be available until a few developers have gitian signed the non-codesigned binaries.
To build the most recent tag:
```bash
./gitian-build.py --detach-sign --no-commit -b satoshi 0.18.5
```
To speed up the build, use `-j 5 -m 5000` as the first arguments, where `5` is the number of CPU's you allocated to the VM plus one, and 5000 is a little bit less than then the MB's of RAM you allocated.
If all went well, this produces a number of (uncommited) `.assert` files in the gitian.sigs repository.
You need to copy these uncommited changes to your host machine, where you can sign them:
```bash
export NAME=satoshi
gpg --output $VERSION-linux/$NAME/bitcoin-abc-linux-0.18.5-build.assert.sig --detach-sign 0.18.5-linux/$NAME/bitcoin-abc-linux-0.18.5-build.assert
gpg --output $VERSION-osx-unsigned/$NAME/bitcoin-abc-osx-0.18.5-build.assert.sig --detach-sign 0.18.5-osx-unsigned/$NAME/bitcoin-abc-osx-0.18.5-build.assert
gpg --output $VERSION-win-unsigned/$NAME/bitcoin-abc-win-0.18.5-build.assert.sig --detach-sign 0.18.5-win-unsigned/$NAME/bitcoin-abc-win-0.18.5-build.assert
```
diff --git a/doc/gitian-building/gitian-building-manual.md b/doc/gitian-building/gitian-building-manual.md
index b683f5d0f..36e86e9e5 100644
--- a/doc/gitian-building/gitian-building-manual.md
+++ b/doc/gitian-building/gitian-building-manual.md
@@ -1,116 +1,116 @@
Getting and building the inputs
-------------------------------
At this point you have two options, you can either use the automated script (found in [https://github.com/Bitcoin-ABC/bitcoin-abc/blob/master/contrib/gitian-build.py](https://github.com/Bitcoin-ABC/bitcoin-abc/blob/master/contrib/gitian-build.py), only works in Debian/Ubuntu) or you could manually do everything by following this guide.
If you are using the automated script, then run it with the `--setup` command. Afterwards, run it with the `--build` command (example: `contrib/gitian-build.py -b signer 0.15.0`). Otherwise ignore this.
Follow the instructions in [https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change)
in the bitcoin repository under 'Fetch and create inputs' to install sources which require
manual intervention. Also optionally follow the next step: 'Seed the Gitian sources cache
and offline git repositories' which will fetch the remaining files required for building
offline.
Building Bitcoin ABC
--------------------
To build Bitcoin ABC (for Linux, OS X and Windows) just follow the steps under 'perform
Gitian builds' in [https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#setup-and-perform-gitian-builds) in the bitcoin repository.
This may take some time as it will build all the dependencies needed for each descriptor.
These dependencies will be cached after a successful build to avoid rebuilding them when possible.
At any time you can check the package installation and build progress with
```bash
tail -f var/install.log
tail -f var/build.log
```
Output from `gbuild` will look something like
- Initialized empty Git repository in /home/gitianuser/gitian-builder/inputs/bitcoin/.git/
+ Initialized empty Git repository in /home/gitianuser/bitcoin-abc/contrib/gitian-builder/inputs/bitcoin/.git/
remote: Counting objects: 57959, done.
remote: Total 57959 (delta 0), reused 0 (delta 0), pack-reused 57958
Receiving objects: 100% (57959/57959), 53.76 MiB | 484.00 KiB/s, done.
Resolving deltas: 100% (41590/41590), done.
From https://github.com/Bitcoin-ABC/bitcoin-abc.git
... (new tags, new branch etc)
--- Building for trusty amd64 ---
Stopping target if it is up
Making a new image copy
stdin: is not a tty
Starting target
Checking if target is up
Preparing build environment
Updating apt-get repository (log in var/install.log)
Installing additional packages (log in var/install.log)
Grabbing package manifest
stdin: is not a tty
Creating build script (var/build-script)
lxc-start: Connection refused - inotify event with no name (mask 32768)
Running build script (log in var/build.log)
Building an alternative repository
----------------------------------
If you want to do a test build of a pull on GitHub it can be useful to point
the Gitian builder at an alternative repository, using the same descriptors
and inputs.
For example:
```bash
URL=https://github.com/Bitcoin-ABC/bitcoin-abc.git
COMMIT=v0.18.5
./bin/gbuild --commit bitcoin=${COMMIT} --url bitcoin=${URL} ../bitcoin-abc/contrib/gitian-descriptors/gitian-linux.yml
./bin/gbuild --commit bitcoin=${COMMIT} --url bitcoin=${URL} ../bitcoin-abc/contrib/gitian-descriptors/gitian-win.yml
./bin/gbuild --commit bitcoin=${COMMIT} --url bitcoin=${URL} ../bitcoin-abc/contrib/gitian-descriptors/gitian-osx.yml
```
Building fully offline
----------------------
For building fully offline including attaching signatures to unsigned builds, the detached-sigs repository
and the bitcoin git repository with the desired tag must both be available locally, and then gbuild must be
told where to find them. It also requires an apt-cacher-ng which is fully-populated but set to offline mode, or
manually disabling gitian-builder's use of apt-get to update the VM build environment.
To configure apt-cacher-ng as an offline cacher, you will need to first populate its cache with the relevant
files. You must additionally patch target-bin/bootstrap-fixup to set its apt sources to something other than
plain archive.ubuntu.com: us.archive.ubuntu.com works.
So, if you use LXC:
```bash
-export PATH="$PATH":${HOME}/gitian-builder/libexec
+export PATH="$PATH":${HOME}/bitcoin-abc/contrib/gitian-builder/libexec
export USE_LXC=1
-cd ~/gitian-builder
+cd ~/bitcoin-abc/contrib/gitian-builder
./libexec/make-clean-vm --suite buster --arch amd64
LXC_ARCH=amd64 LXC_SUITE=buster on-target -u root dpkg --add-architecture i386
LXC_ARCH=amd64 LXC_SUITE=buster on-target -u root apt-get update
LXC_ARCH=amd64 LXC_SUITE=buster on-target -u root \
-e DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends -y install \
- $( sed -ne '/^packages:/,/^[^-]/ {/^- .*/{s/"//g;s/- //;p}}' ../bitcoin-abc/contrib/gitian-descriptors/*|sort|uniq )
+ $( sed -ne '/^packages:/,/^[^-]/ {/^- .*/{s/"//g;s/- //;p}}' ../gitian-descriptors/*|sort|uniq )
LXC_ARCH=amd64 LXC_SUITE=buster on-target -u root apt-get -q -y purge grub
LXC_ARCH=amd64 LXC_SUITE=buster on-target -u root -e DEBIAN_FRONTEND=noninteractive apt-get -y dist-upgrade
```
And then set offline mode for apt-cacher-ng:
```
/etc/apt-cacher-ng/acng.conf
[...]
Offlinemode: 1
[...]
sudo service apt-cacher-ng restart
```
Then when building, override the remote URLs that gbuild would otherwise pull from the Gitian descriptors::
```bash
cd ~
export URL=${HOME}/bitcoin-abc
export COMMIT=<commmit hash or tag>
./bin/gbuild --commit bitcoin=${COMMIT} --url bitcoin=${URL} ${URL}/contrib/gitian-descriptors/gitian-win.yml
```
diff --git a/doc/gitian-building/gitian-building-setup-gitian-debian.md b/doc/gitian-building/gitian-building-setup-gitian-debian.md
index 54cc34982..77fb4f497 100644
--- a/doc/gitian-building/gitian-building-setup-gitian-debian.md
+++ b/doc/gitian-building/gitian-building-setup-gitian-debian.md
@@ -1,121 +1,120 @@
Table of Contents
-----------------
- [Setting up Debian for Gitian building](#setting-up-debian-for-gitian-building)
- [Installing Gitian](#installing-gitian)
- [Setting up the Gitian image](#setting-up-the-gitian-image)
Setting up Debian for Gitian building
-------------------------------------
In this section we will be setting up the Debian installation for Gitian building.
We assume that a user `gitianuser` was previously added.
First we need to set up dependencies. Type/paste the following in the terminal:
```bash
sudo apt-get install git ruby apt-cacher-ng qemu-utils debootstrap lxc python-cheetah parted kpartx bridge-utils make ubuntu-archive-keyring curl firewalld apparmor iptables
```
Find the device name of your network card. You can list your connections with
```bash
ip address
```
Examples: `eth0`, `eno1`, ...
Save it to the NET_DEV variable:
```bash
sudo -s
NET_DEV=<your device name>
```
Then set up LXC and the rest with the following, which is a complex jumble of settings and workarounds:
```bash
# the version of lxc-start in Debian needs to run as root, so make sure
# that the build script can execute it without providing a password
echo "%sudo ALL=NOPASSWD: /usr/bin/lxc-start" > /etc/sudoers.d/gitian-lxc
echo "%sudo ALL=NOPASSWD: /usr/bin/lxc-execute" >> /etc/sudoers.d/gitian-lxc
# make /etc/rc.local script that sets up bridge between guest and host
echo '#!/bin/sh -e' > /etc/rc.local
echo 'brctl addbr br0' >> /etc/rc.local
echo 'ip addr add 10.0.3.1/24 broadcast 10.0.3.255 dev br0' >> /etc/rc.local
echo 'ip link set br0 up' >> /etc/rc.local
echo 'firewall-cmd --zone=trusted --add-interface=br0' >> /etc/rc.local
echo "iptables -t nat -A POSTROUTING -o ${NET_DEV} -j MASQUERADE" >> /etc/rc.local
echo 'echo 1 > /proc/sys/net/ipv4/ip_forward' >> /etc/rc.local
echo 'exit 0' >> /etc/rc.local
chmod +x /etc/rc.local
# make sure that USE_LXC is always set when logging in as gitianuser,
# and configure LXC IP addresses
echo 'export USE_LXC=1' >> /home/gitianuser/.profile
echo 'export GITIAN_HOST_IP=10.0.3.1' >> /home/gitianuser/.profile
echo 'export LXC_GUEST_IP=10.0.3.5' >> /home/gitianuser/.profile
reboot
```
At the end Debian is rebooted to make sure that the changes take effect. The steps in this
section only need to be performed once.
**Note**: When sudo asks for a password, enter the password for the user `gitianuser` not for `root`.
Installing Gitian
-----------------
Re-login as the user `gitianuser` that was created during installation.
The rest of the steps in this guide will be performed as that user.
There is no `python-vm-builder` package in Debian, so we need to install it from source ourselves,
```bash
wget http://archive.ubuntu.com/ubuntu/pool/universe/v/vm-builder/vm-builder_0.12.4+bzr494.orig.tar.gz
echo "76cbf8c52c391160b2641e7120dbade5afded713afaa6032f733a261f13e6a8e vm-builder_0.12.4+bzr494.orig.tar.gz" | sha256sum -c
# (verification -- must return OK)
tar -zxvf vm-builder_0.12.4+bzr494.orig.tar.gz
cd vm-builder-0.12.4+bzr494
sudo python setup.py install
cd ..
```
**Note**: When sudo asks for a password, enter the password for the user `gitianuser` not for `root`.
-Clone the git repositories for bitcoin and Gitian.
+Clone the git repository for Bitcoin ABC which contains the Gitian sources.
```bash
-git clone https://github.com/devrandom/gitian-builder.git
git clone https://github.com/Bitcoin-ABC/bitcoin-abc.git
```
Setting up the Gitian image
---------------------------
Gitian needs a virtual image of the operating system to build in.
Currently this is Debian 10 Buster x86_64.
This image will be copied and used every time that a build is started to
make sure that the build is deterministic.
Creating the image will take a while, but only has to be done once.
Execute the following as user `gitianuser`:
```bash
-cd gitian-builder
+cd bitcoin-abc/contrib/gitian-builder
bin/make-base-vm --lxc --arch amd64 --distro debian --suite buster
```
There will be a lot of warnings printed during the build of the image. These can be ignored.
**Note**: When sudo asks for a password, enter the password for the user `gitianuser` not for `root`.
Downloading dependencies
------------------------
Gitian may have problems connecting to the internet, which can make the build
flacky. In order to avoid this, it is possible to download dependencies ahead
of time.
Execute the following as user `gitianuser`:
```bash
-cd gitian-builder
-make -C ../bitcoin-abc/depends download SOURCES_PATH=`pwd`/cache/common
+cd bitcoin-abc/contrib/gitian-builder
+make -C ../../depends download SOURCES_PATH=`pwd`/cache/common
```
diff --git a/doc/gitian-building/gitian-building-setup-gitian-fedora.md b/doc/gitian-building/gitian-building-setup-gitian-fedora.md
index e7311b6e5..2202e2212 100644
--- a/doc/gitian-building/gitian-building-setup-gitian-fedora.md
+++ b/doc/gitian-building/gitian-building-setup-gitian-fedora.md
@@ -1,71 +1,68 @@
Table of Contents
------------------
- [Setting up Fedora for Gitian building](#setting-up-fedora-for-gitian-building)
- [Installing Gitian](#installing-gitian)
- [Setting up the Gitian image](#setting-up-the-gitian-image)
Setting up Fedora for Gitian building
--------------------------------------
In this section we will be setting up the Fedora installation for Gitian building.
We assume that a user `gitianuser` was previously created and added to the `wheel` group.
First we need to set up dependencies. Type/paste the following in the terminal:
```bash
sudo dnf install git ruby gnupg docker tar rsync wget curl
```
Then set up docker and the rest with the following, which is a complex jumble of settings and workarounds:
```bash
sudo -s
systemctl enable docker.service
# the version of docker in Fedora needs to run as root, so make sure
# that the build script can execute it without providing a password
# https://docs.docker.com/install/linux/linux-postinstall/
groupadd docker
usermod -aG docker gitianuser
echo 'export USE_DOCKER=1' >> /home/gitianuser/.bash_profile
reboot
```
At the end Fedora is rebooted to make sure that the changes take effect. The steps in this
section only need to be performed once.
**Note**: When sudo asks for a password, enter the password for the user `gitianuser` not for `root`.
Installing Gitian
------------------
Login as the user `gitianuser` that was created during installation.
The rest of the steps in this guide will be performed as that user.
-Clone the git repositories for bitcoin and Gitian.
+Clone the git repository for Bitcoin ABC which contains the Gitian sources.
```bash
-git clone https://github.com/devrandom/gitian-builder.git
-git clone https://github.com/bitcoin/bitcoin
-git clone https://github.com/bitcoin-core/gitian.sigs.git
-git clone https://github.com/bitcoin-core/bitcoin-detached-sigs.git
+git clone https://github.com/Bitcoin-ABC/bitcoin-abc.git
```
Setting up the Gitian image
-------------------------
Gitian needs a virtual image of the operating system to build in.
Currently this is Debian 10 Buster x86_64.
This image will be copied and used every time that a build is started to
make sure that the build is deterministic.
Creating the image will take a while, but only has to be done once.
Execute the following as user `gitianuser`:
```bash
-cd gitian-builder
+cd bitcoin-abc/contrib/gitian-builder
bin/make-base-vm --docker --arch amd64 --suite buster
```
**Note**: When sudo asks for a password, enter the password for the user `gitianuser` not for `root`.
diff --git a/doc/gitian-building/gitian-building-vagrant.md b/doc/gitian-building/gitian-building-vagrant.md
index 279adfb8e..6725fc232 100644
--- a/doc/gitian-building/gitian-building-vagrant.md
+++ b/doc/gitian-building/gitian-building-vagrant.md
@@ -1,68 +1,68 @@
Table of Contents
-----------------
- [Preparing the Gitian builder host](#preparing-the-gitian-builder-host)
- [Setting up the Gitian image](#setting-up-the-gitian-image)
Preparing the Gitian builder host
---------------------------------
The first step is to prepare the host environment that will be used to perform
the Gitian builds. This guide explains how to set up the environment, and how
to start the builds.
Requirements:
- A machine with at least 64b of disk space
- 16GB of RAM
- Several installed tools:
- [Vagrant](https://www.vagrantup.com)
- [Packer](https://www.packer.io)
- [Virtualbox](https://www.virtualbox.org)
After you have installed each of these tools, you will need to create an
ubuntu xenial vagrant "box." This is most easily done using the [box-cutter
project](https://github.com/boxcutter/ubuntu). (Note: Canonical provides a
vagrant box, however its disk space is insufficient for this guide.)
```bash
pushd
cd /tmp/
git clone https://github.com/boxcutter/ubuntu.git
cd ubuntu
git checkout 7d1820c186d76122445c092bc2b872a8a94166ce
packer build -var-file=ubuntu1604.json -only=virtualbox-iso ubuntu.json
vagrant box add --name abc-xenial box/virtualbox/ubuntu1604-0.1.0.box
popd
```
After completion you should be able to run add the box to vagrant as "abc-xenial"
using `vagrant box add --name abc-xenial <path_to_packer_output.box>`
The final step for running vagrant is:
```bash
cd contrib/gitian/
vagrant up
vagrant ssh
```
This should drop you into a Xenial prompt as the user `vagrant`.
Setting up the Gitian image
---------------------------
Gitian needs a virtual image of the operating system to build in. Currently
this is Debian Buster x86_64. This image will be copied and used every time
that a build is started to make sure that the build is deterministic. Creating
the image will take a while, but only has to be done once.
Execute the following as user `vagrant`:
```bash
-cd gitian-builder
+cd bitcoin-abc/contrib/gitian-builder
./bin/make-base-vm --lxc --distro debian --suite buster --arch amd64
```
There will be a lot of warnings printed during the build of the image. These
can be ignored.

File Metadata

Mime Type
text/x-diff
Expires
Wed, Jan 29, 16:55 (10 h, 54 m)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
5053343
Default Alt Text
(39 KB)

Event Timeline