Paths

Table of Contentst

Differential D15522

[electrum] bump docker to ubuntu 20.04 for AppImage building
ClosedPublic
Actions

Authored by PiRK on Feb 23 2024, 09:17.

Details

Reviewers

bytesofman

Group Reviewers

Restricted Project

Commits

rABCf2199508fbfb: [electrum] bump docker to ubuntu 20.04 for AppImage building

Summary

We want to build the linux releases on an old linux version to produce an AppImage that works on the widest possible range of linux distros, but Ubuntu 18.04 is now too old for compatibility with various python packages.

For instance we need Ubuntu 20.04 because newer cryptography version (updated in a follow-up commit to bump the package python version) need a newer OpenSSL version than is available on Bionic.

see https://github.com/pyca/cryptography/pull/8449

This is a partial backport of electroncash#2774, electroncash#2778 and electroncash#2755

Test Plan

cd electrum
contrib/make_clean
contrib/build-linux/appimage/build.sh

Then run the release binary: dist/ElectrumABC-5.2.11-x86_64.AppImage

Diff Detail

Repository

rABC Bitcoin ABC

Lint

Lint Passed

Unit

No Test Coverage

Build Status

Buildable 27360
Build 54283: Build Diff	electrum-tests
Build 54282: arc lint + arc unit

Event Timeline

PiRK created this revision.Feb 23 2024, 09:17

Herald added a reviewer: Restricted Project. · View Herald TranscriptFeb 23 2024, 09:17

PiRK requested review of this revision.Feb 23 2024, 09:17

teamcity edited the summary of this revision. (Show Details)Feb 23 2024, 09:17

Harbormaster completed remote builds in B27343: Diff 45507.Feb 23 2024, 09:21

PiRK mentioned this in D15523: [doc] document the procedure to backport from subtree projects.Feb 23 2024, 09:27

bytesofman added a subscriber: bytesofman.Feb 23 2024, 11:26

bytesofman added inline comments.

electrum/contrib/build-linux/appimage/Dockerfile_ub1804
58 ↗	(On Diff #45507)	why don't we need to do this anymore?

💬 INFO:  Building Python
patching file Makefile.pre.in
Hunk #1 succeeded at 784 (offset -11 lines).
./Modules/posixmodule.c:7179:10: fatal error: stropts.h: No such file or directory
 7179 | #include <stropts.h>
      |          ^~~~~~~~~~~
compilation terminated.
make: *** [Makefile:1992: Modules/posixmodule.o] Error 1
make: *** Waiting for unfinished jobs....
🗯  ERROR:  Could not build Python
🗯  ERROR:  Build inside docker container failed

not sure if there are some deps I am missing or?

OS
Pop!_OS 22.04 LTS

Ran

cd electrum
./contrib/build-linux/appimage/build.sh

bytesofman requested changes to this revision.Feb 23 2024, 12:17

This revision now requires changes to proceed.Feb 23 2024, 12:17

In D15522#349986, @bytesofman wrote:
not sure if there are some deps I am missing or?

OS
Pop!_OS 22.04 LTS

Ran
cd electrum
./contrib/build-linux/appimage/build.sh

That's odd. I don't see why the build would depend on the host OS, the whole point of the docker is that it should be reproducible regardless. I will have to find a way to reproduce this.

electrum/contrib/build-linux/appimage/Dockerfile_ub1804
58 ↗	(On Diff #45507)	Good question. This was added in https://github.com/Bitcoin-ABC/ElectrumABC/commit/b83cf8795f3ac35368292d76d7b91a0112905fef. That commit removes usages of `sudo` in the main script calling the `docker` commands (`build.sh`), but that is unrelated to the use of root or `sudo` in the script running in the docker environment itself (`_build.sh`) So either it was never needed or it is no longer needed due to a difference in the docker version packaged in ubuntu 20.04. I'm trying to rerun the old version with ub1804 without that block to see if it was needed before, but now the build is failing to build python for other unrelated reasons.

my output from docker -v

Docker version 25.0.3, build 4debf41

PiRK mentioned this in rABC3f04a19d3094: [doc] document the procedure to backport from subtree projects.Feb 23 2024, 14:10

rebase

still checking why the build does not work on other machines. Testing a new python version.

Harbormaster completed remote builds in B27360: Diff 45529.Feb 23 2024, 15:58

PiRK added a child revision: D15527: [electrum] build: Update to Python 3.11.8.Feb 23 2024, 16:01

test plan update to include make_clean so that previous build caches are cleared
We cannot cache a build configuration with Ubuntu 18.04 and reuse it with Ubuntu 20.04.

$ cd electrum/
$ ./contrib/make_clean

gives

Removing .pytest_cache/
Removing ElectrumABC.egg-info/
Removing __pycache__/
Removing build/
Removing contrib/build-linux/appimage/.cache/
Removing contrib/build-linux/appimage/build/
Removing contrib/build/
Removing dist/
Removing electrumabc/__pycache__/
Removing electrumabc/avalanche/__pycache__/
Removing electrumabc/libsecp256k1.so.0
Removing electrumabc/libzbar.so.0
Removing electrumabc/locale/
Removing electrumabc/qrreaders/__pycache__/
Removing electrumabc/slp/__pycache__/
Removing electrumabc/tests/__pycache__/
Removing electrumabc/tests/regtest/__pycache__/
Removing electrumabc/tor/__pycache__/
Removing electrumabc/utils/__pycache__/
Removing electrumabc_gui/__pycache__/
Removing electrumabc_gui/qt/__pycache__/
Removing electrumabc_gui/qt/avalanche/__pycache__/
Removing electrumabc_gui/qt/multi_transactions_dialog.py.rej
Removing electrumabc_gui/qt/qrreader/__pycache__/
Removing electrumabc_gui/qt/utils/__pycache__/
Removing electrumabc_gui/qt/utils/darkdetect/__pycache__/
Removing electrumabc_plugins/__pycache__/
Removing electrumabc_plugins/audio_modem/__pycache__/
Removing electrumabc_plugins/cosigner_pool/__pycache__/
Removing electrumabc_plugins/digitalbitbox/__pycache__/
Removing electrumabc_plugins/email_requests/__pycache__/
Removing electrumabc_plugins/fusion/__pycache__/
Removing electrumabc_plugins/fusion/tests/__pycache__/
Removing electrumabc_plugins/hw_wallet/__pycache__/
Removing electrumabc_plugins/keepkey/__pycache__/
Removing electrumabc_plugins/ledger/__pycache__/
Removing electrumabc_plugins/satochip/__pycache__/
Removing electrumabc_plugins/trezor/__pycache__/
Removing electrumabc_plugins/virtualkeyboard/__pycache__/

Then,

$ ./contrib/build-linux/appimage/build.sh

gives

~/github/abc/bitcoin-abc/electrum ~/github/abc/bitcoin-abc/electrum
💬 INFO:  Using docker: Docker version 25.0.3, build 4debf41
💬 INFO:  Creating docker image ...
[+] Building 1.0s (6/6) FINISHED                                                   docker:default
 => [internal] load build definition from Dockerfile_ub2004                                  0.0s
 => => transferring dockerfile: 2.74kB                                                       0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04@sha256:f2034e7195f61334e6ca  0.8s
 => [internal] load .dockerignore                                                            0.0s
 => => transferring context: 2B                                                              0.0s
 => [1/2] FROM docker.io/library/ubuntu:20.04@sha256:f2034e7195f61334e6caff6ecf2e965f92d11e  0.0s
 => CACHED [2/2] RUN echo deb http://archive.ubuntu.com/ubuntu/ focal main restricted unive  0.0s
 => exporting to image                                                                       0.0s
 => => exporting layers                                                                      0.0s
 => => writing image sha256:d3a8013506a88d63ebece75c46283f55f897cdf8c9a4125e531edf864110a14  0.0s
 => => naming to docker.io/library/electrumabc-appimage-builder-img-ub2004                   0.0s
docker: Error response from daemon: failed to create endpoint electrumabc-appimage-builder-cont-ub2004 on network bridge: failed to add the host (vetha30e739) <=> sandbox (vethc47f229) pair interfaces: operation not supported.
ERRO[0000] error waiting for container: context canceled 
🗯  ERROR:  Build inside docker container failed

... a strange error. restarting docker did not fix. my user is in the docker group. after some digging, apparently rebooting your machine can fix this? tried that...did work, including running the binary after. Build took a long time (maybe ~10 min).

https://unix.stackexchange.com/questions/360063/docker-error-response-from-daemon-failed-to-create-endpoint-failed-to-add-ho

Includes this (well-upvoted) comment

Things like this make me want to leave computers behind and look into woodcutting

is this you?

I did see this warning during the run:

WARNING: You are using pip version 21.3.1; however, version 24.0 is available.
You should consider upgrading via the '/opt/electrumabc/contrib/build-linux/appimage/build/appimage/ElectrumABC.AppDir/usr/bin/python3.9 -m pip install --upgrade pip' command.

weird since it is from the docker container? maybe the deps in one of the images could be upgraded? It does work now...so perhaps best not to touch

This revision is now accepted and ready to land.Mar 1 2024, 13:12

In D15522#351905, @bytesofman wrote:

Includes this (well-upvoted) comment

Things like this make me want to leave computers behind and look into woodcutting

is this you?

It wasn't me, but I feel his pain.

In D15522#351905, @bytesofman wrote:
I did see this warning during the run:
WARNING: You are using pip version 21.3.1; however, version 24.0 is available.
You should consider upgrading via the '/opt/electrumabc/contrib/build-linux/appimage/build/appimage/ElectrumABC.AppDir/usr/bin/python3.9 -m pip install --upgrade pip' command.
weird since it is from the docker container? maybe the deps in one of the images could be upgraded? It does work now...so perhaps best not to touch

The pip version is pinpointed to an older version for build reproducibility, but I'm about to bump it to 23.2 in D15527. Maybe I'll try to bump it further, but this warning is very common and no matter what version we pick it will reappear as new versions are released. It is safe to ignore this, as long as the build works.