Page MenuHomePhabricator

scripts: use LIEF for ELF security & symbol checks
ClosedPublic

Authored by PiRK on Jun 2 2023, 10:49.

Details

Reviewers
Fabien
Group Reviewers
Restricted Project
Commits
rABCe0502575596f: scripts: use LIEF for ELF security & symbol checks
Summary

test-*-check: Pass in *FLAGS and compile with them

These test-*-check scripts should compile "test" binaries in a way that
is as close to what autotools would do, since the goal is to make sure
that if we run the *-check script, they can correctly detect flaws in
binaries which are compiled by our autotools-based system.

Therefore, we should emulate what happens when the binary is linked in
autotools, meaning that for C binaries, we need to supply the CFLAGS,
CPPFLAGS, and LDFLAGS flags in that order.

Note to future developers: perhaps it'd be nice to have these
test-*-check scripts be part of configure.ac to avoid having to manually
replicate autoconf-like behaviour every time we find a discrepancy. Of
course, that would also mean you'd have to write more m4...

scripts: use LIEF for ELF checks in symbol-check.py

Co-authored-by: Carl Dong <contact@carldong.me>

Note that TestSymbolChecks.test_PE currently still fails for another reason on my machine

'test1: symbol __libc_start_main from unsupported version GLIBC_2.34(2)\n'

I'm still looking for the backport to make this work.

scripts: only parse the binary once in symbol-check.py

scripts: use LIEF for ELF checks in security-check.py

scripts: only parse the binary once in security-check.py

scripts: remove pixie.py

This is a backport of core#22392
Depends on D13970

Test Plan

Check for no regressions in unit tests:

contrib/devtools/test-security-check.py
contrib-devtools/test-symbol-check.py

gitian builds

Diff Detail

Repository
rABC Bitcoin ABC
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

PiRK requested review of this revision.Jun 2 2023, 10:49

fix a flake8 error (don't use bare except)

I'm not sure what the expected errors can be, so just catch all exceptions

@bot gitian-osx gitian-linux gitian-win

Tail of the build log:

 * [new tag]             phabricator/diff/40476 -> phabricator/diff/40476
 * [new tag]             phabricator/diff/40477 -> phabricator/diff/40477
 * [new tag]             phabricator/diff/40478 -> phabricator/diff/40478
 * [new tag]             phabricator/diff/40479 -> phabricator/diff/40479
 * [new tag]             phabricator/diff/40480 -> phabricator/diff/40480
 * [new tag]             phabricator/diff/40486 -> phabricator/diff/40486
 * [new tag]             phabricator/diff/40487 -> phabricator/diff/40487
 * [new tag]             phabricator/diff/40489 -> phabricator/diff/40489
 * [new tag]             phabricator/diff/40490 -> phabricator/diff/40490
 * [new tag]             phabricator/diff/40491 -> phabricator/diff/40491
 * [new tag]             phabricator/diff/40495 -> phabricator/diff/40495
 * [new tag]             phabricator/diff/40497 -> phabricator/diff/40497
 * [new tag]             phabricator/diff/40501 -> phabricator/diff/40501
 * [new tag]             phabricator/diff/40503 -> phabricator/diff/40503
 * [new tag]             phabricator/diff/40504 -> phabricator/diff/40504
 * [new tag]             phabricator/diff/40505 -> phabricator/diff/40505
 * [new tag]             phabricator/diff/40507 -> phabricator/diff/40507
 * [new tag]             phabricator/diff/40508 -> phabricator/diff/40508
 * [new tag]             phabricator/diff/40509 -> phabricator/diff/40509
 * [new tag]             phabricator/diff/40510 -> phabricator/diff/40510
 * [new tag]             phabricator/diff/40511 -> phabricator/diff/40511
 * [new tag]             phabricator/diff/40512 -> phabricator/diff/40512
 * [new tag]             phabricator/diff/40513 -> phabricator/diff/40513
 * [new tag]             phabricator/diff/40514 -> phabricator/diff/40514
 * [new tag]             phabricator/diff/40515 -> phabricator/diff/40515
 * [new tag]             phabricator/diff/40516 -> phabricator/diff/40516
 * [new tag]             phabricator/diff/40522 -> phabricator/diff/40522
 * [new tag]             phabricator/diff/40525 -> phabricator/diff/40525
 * [new tag]             phabricator/diff/40526 -> phabricator/diff/40526
 * [new tag]             phabricator/diff/40527 -> phabricator/diff/40527
 * [new tag]             phabricator/diff/40531 -> phabricator/diff/40531
 * [new tag]             phabricator/diff/40541 -> phabricator/diff/40541
 * [new tag]             phabricator/diff/40542 -> phabricator/diff/40542
 * [new tag]             phabricator/diff/40543 -> phabricator/diff/40543
 * [new tag]             phabricator/diff/40547 -> phabricator/diff/40547
 * [new tag]             phabricator/diff/8992  -> phabricator/diff/8992
 * [new tag]             phabricator/diff/8993  -> phabricator/diff/8993
 * [new branch]          master                 -> master
--- Building for bullseye amd64 ---
Stopping target if it is up
Error response from daemon: No such container: gitian-target
Error: No such container: gitian-target
Making a new image copy
Starting target
Checking if target is up.
Preparing build environment
Updating apt-get repository (log in var/install.log)
Installing additional packages (log in var/install.log)
Upgrading system, may take a while (log in var/install.log)
Creating package manifest
Creating build script (var/build-script)
Running build script (log in var/build.log)
./bin/gbuild:23:in `system!': failed to run on-target setarch x86_64 bash -x < var/build-script > var/build.log 2>&1 (RuntimeError)
	from ./bin/gbuild:185:in `build_one_configuration'
	from ./bin/gbuild:339:in `block (2 levels) in <main>'
	from ./bin/gbuild:334:in `each'
	from ./bin/gbuild:334:in `block in <main>'
	from ./bin/gbuild:332:in `each'
	from ./bin/gbuild:332:in `<main>'
Build gitian-osx failed with exit code 1

Tail of the build log:

 * [new tag]             phabricator/diff/40382 -> phabricator/diff/40382
 * [new tag]             phabricator/diff/40384 -> phabricator/diff/40384
 * [new tag]             phabricator/diff/40400 -> phabricator/diff/40400
 * [new tag]             phabricator/diff/40401 -> phabricator/diff/40401
 * [new tag]             phabricator/diff/40402 -> phabricator/diff/40402
 * [new tag]             phabricator/diff/40404 -> phabricator/diff/40404
 * [new tag]             phabricator/diff/40405 -> phabricator/diff/40405
 * [new tag]             phabricator/diff/40408 -> phabricator/diff/40408
 * [new tag]             phabricator/diff/40409 -> phabricator/diff/40409
 * [new tag]             phabricator/diff/40429 -> phabricator/diff/40429
 * [new tag]             phabricator/diff/40430 -> phabricator/diff/40430
 * [new tag]             phabricator/diff/40431 -> phabricator/diff/40431
 * [new tag]             phabricator/diff/40432 -> phabricator/diff/40432
 * [new tag]             phabricator/diff/40434 -> phabricator/diff/40434
 * [new tag]             phabricator/diff/40441 -> phabricator/diff/40441
 * [new tag]             phabricator/diff/40450 -> phabricator/diff/40450
 * [new tag]             phabricator/diff/40461 -> phabricator/diff/40461
 * [new tag]             phabricator/diff/40463 -> phabricator/diff/40463
 * [new tag]             phabricator/diff/40469 -> phabricator/diff/40469
 * [new tag]             phabricator/diff/40476 -> phabricator/diff/40476
 * [new tag]             phabricator/diff/40477 -> phabricator/diff/40477
 * [new tag]             phabricator/diff/40479 -> phabricator/diff/40479
 * [new tag]             phabricator/diff/40486 -> phabricator/diff/40486
 * [new tag]             phabricator/diff/40487 -> phabricator/diff/40487
 * [new tag]             phabricator/diff/40489 -> phabricator/diff/40489
 * [new tag]             phabricator/diff/40490 -> phabricator/diff/40490
 * [new tag]             phabricator/diff/40507 -> phabricator/diff/40507
 * [new tag]             phabricator/diff/40508 -> phabricator/diff/40508
 * [new tag]             phabricator/diff/40511 -> phabricator/diff/40511
 * [new tag]             phabricator/diff/40526 -> phabricator/diff/40526
 * [new tag]             phabricator/diff/40527 -> phabricator/diff/40527
 * [new tag]             phabricator/diff/40529 -> phabricator/diff/40529
 * [new tag]             phabricator/diff/40541 -> phabricator/diff/40541
 * [new tag]             phabricator/diff/40542 -> phabricator/diff/40542
 * [new tag]             phabricator/diff/40547 -> phabricator/diff/40547
 * [new tag]             phabricator/diff/8992  -> phabricator/diff/8992
 * [new tag]             phabricator/diff/8993  -> phabricator/diff/8993
 * [new branch]          master                 -> master
--- Building for bullseye amd64 ---
Stopping target if it is up
Error response from daemon: No such container: gitian-target
Error: No such container: gitian-target
Making a new image copy
Starting target
Checking if target is up.
Preparing build environment
Updating apt-get repository (log in var/install.log)
Installing additional packages (log in var/install.log)
Upgrading system, may take a while (log in var/install.log)
Creating package manifest
Creating build script (var/build-script)
Running build script (log in var/build.log)
./bin/gbuild:23:in `system!': failed to run on-target setarch x86_64 bash -x < var/build-script > var/build.log 2>&1 (RuntimeError)
	from ./bin/gbuild:185:in `build_one_configuration'
	from ./bin/gbuild:339:in `block (2 levels) in <main>'
	from ./bin/gbuild:334:in `each'
	from ./bin/gbuild:334:in `block in <main>'
	from ./bin/gbuild:332:in `each'
	from ./bin/gbuild:332:in `<main>'
Build gitian-linux failed with exit code 1
PiRK planned changes to this revision.Jun 2 2023, 13:03

rebase after bumping glibc 2.28

@bot gitian-osx gitian-linux gitian-win

This revision is now accepted and ready to land.Jun 2 2023, 16:18