Page MenuHomePhabricator

contrib: Parse ELF directly for symbol and security checks
ClosedPublic

Authored by PiRK on Apr 5 2023, 14:10.

Details

Reviewers
Fabien
Group Reviewers
Restricted Project
Commits
rABC091ec4f079cf: contrib: Parse ELF directly for symbol and security checks
Summary

Instead of the ever-messier text parsing of the output of the readelf
tool (which is clearly meant for human consumption not to be machine
parseable), parse the ELF binaries directly.

Add a small dependency-less ELF parser specific to the checks.

This is slightly more secure, too, because it removes potential
ambiguity due to misparsing and changes in the output format of elfread. It
also allows for stricter and more specific ELF format checks in the future.

This removes the build-time dependency for readelf.

This is a backport of core#20434
Depends on D13582

Test Plan

gitian builds

Diff Detail

Repository
rABC Bitcoin ABC
Branch
pr18796
Lint
Lint Passed
Unit
No Test Coverage
Build Status
Buildable 23003
Build 45626: Build Diff
Build 45625: arc lint + arc unit

Event Timeline

@bot gitian-linux gitian-win gitian-osx

PiRK published this revision for review.Apr 5 2023, 14:13

Tail of the build log:

[14:17:58] :	 [Step 1/1] Checking if target is up.
[14:17:58] :	 [Step 1/1] Preparing build environment
[14:17:58] :	 [Step 1/1] Updating apt-get repository (log in var/install.log)
[14:34:29] :	 [Step 1/1] Installing additional packages (log in var/install.log)
[14:34:29]i:	 [Step 1/1] ##teamcity[importData timestamp='2023-04-05T14:34:29.234' path='results/artifacts/junit/*.xml' type='junit']
[14:34:29] :	 [Step 1/1] Importing data from 'results/artifacts/junit/*.xml' (not existing file) with 'junit' processor
[14:34:29]W:	 [Step 1/1] Publishing artifacts (2s)
[14:34:29] :		 [Publishing artifacts] Collecting files to publish: [+:results/artifacts=>artifacts.tar.gz]
[14:34:29] :	 [Step 1/1] Ant JUnit report watcher
[14:34:29]W:		 [Publishing artifacts] Artifacts path 'results/artifacts' not found
[14:34:29] :		 [Ant JUnit report watcher] Watching paths:
[14:34:29] :		 [Ant JUnit report watcher] /home/teamcity/buildAgent/work/jailed-build/results/artifacts/junit/*.xml
[14:34:29]i:	 [Step 1/1] ##teamcity[publishArtifacts '+:results/artifacts=>artifacts.tar.gz']
[14:34:31]E:	 [Step 1/1] Build gitian-osx failed with exit code 1
[14:34:29]i:	 [Step 1/1] ##teamcity[buildProblem timestamp='2023-04-05T14:34:29.368' description='Build gitian-osx failed with exit code 1']
[14:34:29] :	 [Step 1/1] Upgrading system, may take a while (log in var/install.log)
[14:34:29] :	 [Step 1/1] Creating package manifest
[14:34:29] :	 [Step 1/1] Creating build script (var/build-script)
[14:34:29] :	 [Step 1/1] Running build script (log in var/build.log)
[14:34:29] :	 [Step 1/1] ./bin/gbuild:23:in `system!': failed to run on-target setarch x86_64 bash -x < var/build-script > var/build.log 2>&1 (RuntimeError)
[14:34:29] :	 [Step 1/1] 	from ./bin/gbuild:185:in `build_one_configuration'
[14:34:29] :	 [Step 1/1] 	from ./bin/gbuild:339:in `block (2 levels) in <main>'
[14:34:29] :	 [Step 1/1] 	from ./bin/gbuild:334:in `each'
[14:34:29] :	 [Step 1/1] 	from ./bin/gbuild:334:in `block in <main>'
[14:34:29] :	 [Step 1/1] 	from ./bin/gbuild:332:in `each'
[14:34:29] :	 [Step 1/1] 	from ./bin/gbuild:332:in `<main>'
[14:34:29] :	 [Step 1/1] Build gitian-osx failed with exit code 1
[14:34:29] :	 [Step 1/1] ~/infra ~/buildAgent/work/jailed-build/bitcoin-abc ~/buildAgent/work/jailed-build
[14:34:29]W:	 [Step 1/1] + RESULT=1
[14:34:29]W:	 [Step 1/1] + pushd /home/teamcity/infra
[14:34:29]W:	 [Step 1/1] + docker-compose stop apt-cache-proxy
[14:34:29]W:	 [Step 1/1] Stopping abc-apt-cache-proxy ... 
[14:34:40] :	 [Step 1/1] ~/buildAgent/work/jailed-build/bitcoin-abc ~/buildAgent/work/jailed-build
[14:34:40]W:	 [Step 1/1]  [1A [2K
[14:34:40]W:	 [Step 1/1] Stopping abc-apt-cache-proxy ...  [32mdone [0m
[14:34:40]W:	 [Step 1/1]  [1B+ popd
[14:34:40]W:	 [Step 1/1] + exit 1
[14:34:40]W:	 [Step 1/1] Process exited with code 1
[14:34:40]E:	 [Step 1/1] Process exited with code 1 (Step: Command Line)
[14:34:40] :	 [Step 1/1] Waiting for 1 service processes to complete
[14:34:40]E:	 [Step 1/1] Ant JUnit report watcher
[14:34:40]E:		 [Ant JUnit report watcher] No reports found for paths:
[14:34:40]E:		 [Ant JUnit report watcher] /home/teamcity/buildAgent/work/jailed-build/results/artifacts/junit/*.xml
[14:34:40]E:	 [Step 1/1] Step Command Line failed
[14:34:41]E: Ant JUnit report watcher
[14:34:41]E:	 [Ant JUnit report watcher] No reports found for paths:
[14:34:41]E:	 [Ant JUnit report watcher] +:results/test_bitcoin.xml
[14:34:41]E:	 [Ant JUnit report watcher] +:results/**/junit_results*.xml
[14:34:41] : Publishing internal artifacts (1s)
[14:34:42] :	 [Publishing internal artifacts] Publishing 1 file using [WebPublisher]
[14:34:42] :	 [Publishing internal artifacts] Publishing 1 file using [ArtifactsCachePublisher]
[14:34:41]W: Publishing artifacts (2s)
[14:34:41] :	 [Publishing artifacts] Collecting files to publish: [+:results/**/junit_results*.xml, +:bitcoin-abc/abc-ci-builds/gitian-osx/gitian-results => gitian-osx.tar.gz]
[14:34:41]W:	 [Publishing artifacts] Artifacts path 'results/**/junit_results*.xml' not found
[14:34:41] :	 [Publishing artifacts] Creating archive gitian-osx.tar.gz
[14:34:41] :		 [Creating archive gitian-osx.tar.gz] Creating /home/teamcity/buildAgent/temp/buildTmp/TarPreprocessor2100032004465348228/gitian-osx.tar.gz
[14:34:41] :		 [Creating archive gitian-osx.tar.gz] Archive was created, file size 210.73 KB (215789 bytes)
[14:34:43] :	 [Publishing artifacts] Publishing 1 file using [WebPublisher]: bitcoin-abc/abc-ci-builds/gitian-osx/gitian-results => gitian-osx.tar.gz
[14:34:43] :	 [Publishing artifacts] Publishing 1 file using [ArtifactsCachePublisher]: bitcoin-abc/abc-ci-builds/gitian-osx/gitian-results => gitian-osx.tar.gz
[14:34:44] : Build finished

Tail of the build log:

[14:37:32] :	 [Step 1/1]  * [new tag]             phabricator/diff/39293 -> phabricator/diff/39293
[14:37:32] :	 [Step 1/1]  * [new tag]             phabricator/diff/8992  -> phabricator/diff/8992
[14:37:32] :	 [Step 1/1]  * [new tag]             phabricator/diff/8993  -> phabricator/diff/8993
[14:37:32] :	 [Step 1/1]  * [new branch]          master                 -> master
[14:37:32] :	 [Step 1/1] --- Building for bullseye amd64 ---
[14:37:32] :	 [Step 1/1] Stopping target if it is up
[14:37:32] :	 [Step 1/1] Error response from daemon: No such container: gitian-target
[14:37:32] :	 [Step 1/1] Error: No such container: gitian-target
[14:37:32] :	 [Step 1/1] Making a new image copy
[14:37:32] :	 [Step 1/1] Starting target
[14:37:32] :	 [Step 1/1] Checking if target is up.
[14:37:32] :	 [Step 1/1] Preparing build environment
[14:37:32] :	 [Step 1/1] Updating apt-get repository (log in var/install.log)
[14:37:32] :	 [Step 1/1] Installing additional packages (log in var/install.log)
[14:37:32] :	 [Step 1/1] Upgrading system, may take a while (log in var/install.log)
[14:37:32] :	 [Step 1/1] Creating package manifest
[14:37:32] :	 [Step 1/1] Creating build script (var/build-script)
[14:37:32] :	 [Step 1/1] Running build script (log in var/build.log)
[14:37:32] :	 [Step 1/1] ./bin/gbuild:23:in `system!': failed to run on-target setarch x86_64 bash -x < var/build-script > var/build.log 2>&1 (RuntimeError)
[14:37:32] :	 [Step 1/1] 	from ./bin/gbuild:185:in `build_one_configuration'
[14:37:32] :	 [Step 1/1] 	from ./bin/gbuild:339:in `block (2 levels) in <main>'
[14:37:32] :	 [Step 1/1] 	from ./bin/gbuild:334:in `each'
[14:37:32] :	 [Step 1/1] 	from ./bin/gbuild:334:in `block in <main>'
[14:37:32] :	 [Step 1/1] 	from ./bin/gbuild:332:in `each'
[14:37:32] :	 [Step 1/1] 	from ./bin/gbuild:332:in `<main>'
[14:37:32] :	 [Step 1/1] Build gitian-win failed with exit code 1
[14:37:32] :	 [Step 1/1] ~/infra ~/buildAgent/work/jailed-build/bitcoin-abc ~/buildAgent/work/jailed-build
[14:37:32]W:	 [Step 1/1] + RESULT=1
[14:37:32]W:	 [Step 1/1] + pushd /home/teamcity/infra
[14:37:32]W:	 [Step 1/1] + docker-compose stop apt-cache-proxy
[14:37:33]W:	 [Step 1/1] Stopping abc-apt-cache-proxy ... 
[14:37:43]W:	 [Step 1/1]  [1A [2K
[14:37:43]W:	 [Step 1/1] Stopping abc-apt-cache-proxy ...  [32mdone [0m
[14:37:43]W:	 [Step 1/1]  [1B
[14:37:43]W:	 [Step 1/1] + popd
[14:37:43] :	 [Step 1/1] ~/buildAgent/work/jailed-build/bitcoin-abc ~/buildAgent/work/jailed-build
[14:37:43]W:	 [Step 1/1] + exit 1
[14:37:43]W:	 [Step 1/1] Process exited with code 1
[14:37:43]E:	 [Step 1/1] Process exited with code 1 (Step: Command Line)
[14:37:43] :	 [Step 1/1] Waiting for 1 service processes to complete
[14:37:43]E:	 [Step 1/1] Ant JUnit report watcher
[14:37:43]E:		 [Ant JUnit report watcher] No reports found for paths:
[14:37:43]E:		 [Ant JUnit report watcher] /home/teamcity/buildAgent/work/jailed-build/results/artifacts/junit/*.xml
[14:37:44]E:	 [Step 1/1] Step Command Line failed
[14:37:44]E: Ant JUnit report watcher
[14:37:44]E:	 [Ant JUnit report watcher] No reports found for paths:
[14:37:44]E:	 [Ant JUnit report watcher] +:results/test_bitcoin.xml
[14:37:44]E:	 [Ant JUnit report watcher] +:results/**/junit_results*.xml
[14:37:44] : Publishing internal artifacts (4s)
[14:37:48] :	 [Publishing internal artifacts] Publishing 1 file using [WebPublisher]
[14:37:48] :	 [Publishing internal artifacts] Publishing 1 file using [ArtifactsCachePublisher]
[14:37:44]W: Publishing artifacts (5s)
[14:37:44] :	 [Publishing artifacts] Collecting files to publish: [+:results/**/junit_results*.xml, +:bitcoin-abc/abc-ci-builds/gitian-win/gitian-results => gitian-win.tar.gz]
[14:37:44]W:	 [Publishing artifacts] Artifacts path 'results/**/junit_results*.xml' not found
[14:37:44] :	 [Publishing artifacts] Creating archive gitian-win.tar.gz
[14:37:44] :		 [Creating archive gitian-win.tar.gz] Creating /home/teamcity/buildAgent/temp/buildTmp/TarPreprocessor3059956386502173685/gitian-win.tar.gz
[14:37:44] :		 [Creating archive gitian-win.tar.gz] Archive was created, file size 388.12 KB (397442 bytes)
[14:37:49] :	 [Publishing artifacts] Publishing 1 file using [WebPublisher]: bitcoin-abc/abc-ci-builds/gitian-win/gitian-results => gitian-win.tar.gz
[14:37:49] :	 [Publishing artifacts] Publishing 1 file using [ArtifactsCachePublisher]: bitcoin-abc/abc-ci-builds/gitian-win/gitian-results => gitian-win.tar.gz
[14:37:50] : Build finished
Fabien requested changes to this revision.Apr 5 2023, 14:39
Fabien added a subscriber: Fabien.
Fabien added inline comments.
contrib/devtools/pixie.py
6 ↗(On Diff #39293)
contrib/devtools/symbol-check.py
75 ↗(On Diff #39293)

we don't need that

88 ↗(On Diff #39293)

same for the last 2 ones

189 ↗(On Diff #39293)
This revision now requires changes to proceed.Apr 5 2023, 14:39

rebase and address review

@bot gitian-linux gitian-win gitian-osx

Tail of the build log:

[15:21:27] :	 [Step 1/1]  * [new tag]             phabricator/diff/39298 -> phabricator/diff/39298
[15:21:27] :	 [Step 1/1]  * [new tag]             phabricator/diff/39299 -> phabricator/diff/39299
[15:21:27] :	 [Step 1/1]  * [new tag]             phabricator/diff/8992  -> phabricator/diff/8992
[15:21:27] :	 [Step 1/1]  * [new tag]             phabricator/diff/8993  -> phabricator/diff/8993
[15:21:27] :	 [Step 1/1]  * [new branch]          master                 -> master
[15:21:27] :	 [Step 1/1] --- Building for bullseye amd64 ---
[15:21:27] :	 [Step 1/1] Stopping target if it is up
[15:21:27] :	 [Step 1/1] Error response from daemon: No such container: gitian-target
[15:21:27] :	 [Step 1/1] Error: No such container: gitian-target
[15:21:27] :	 [Step 1/1] Making a new image copy
[15:21:27] :	 [Step 1/1] Starting target
[15:21:27] :	 [Step 1/1] Checking if target is up.
[15:21:27] :	 [Step 1/1] Preparing build environment
[15:21:27] :	 [Step 1/1] Updating apt-get repository (log in var/install.log)
[15:21:27] :	 [Step 1/1] Installing additional packages (log in var/install.log)
[15:21:27] :	 [Step 1/1] Upgrading system, may take a while (log in var/install.log)
[15:21:27] :	 [Step 1/1] Creating package manifest
[15:21:27] :	 [Step 1/1] Creating build script (var/build-script)
[15:21:27] :	 [Step 1/1] Running build script (log in var/build.log)
[15:21:27] :	 [Step 1/1] ./bin/gbuild:23:in `system!': failed to run on-target setarch x86_64 bash -x < var/build-script > var/build.log 2>&1 (RuntimeError)
[15:21:27] :	 [Step 1/1] 	from ./bin/gbuild:185:in `build_one_configuration'
[15:21:27] :	 [Step 1/1] 	from ./bin/gbuild:339:in `block (2 levels) in <main>'
[15:21:27] :	 [Step 1/1] 	from ./bin/gbuild:334:in `each'
[15:21:27] :	 [Step 1/1] 	from ./bin/gbuild:334:in `block in <main>'
[15:21:27] :	 [Step 1/1] 	from ./bin/gbuild:332:in `each'
[15:21:27] :	 [Step 1/1] 	from ./bin/gbuild:332:in `<main>'
[15:21:27] :	 [Step 1/1] Build gitian-osx failed with exit code 1
[15:21:27] :	 [Step 1/1] ~/infra ~/buildAgent/work/jailed-build/bitcoin-abc ~/buildAgent/work/jailed-build
[15:21:27]W:	 [Step 1/1] + RESULT=1
[15:21:27]W:	 [Step 1/1] + pushd /home/teamcity/infra
[15:21:27]W:	 [Step 1/1] + docker-compose stop apt-cache-proxy
[15:21:27]W:	 [Step 1/1] Stopping abc-apt-cache-proxy ... 
[15:21:38]W:	 [Step 1/1]  [1A [2K
[15:21:38] :	 [Step 1/1] ~/buildAgent/work/jailed-build/bitcoin-abc ~/buildAgent/work/jailed-build
[15:21:38]W:	 [Step 1/1] Stopping abc-apt-cache-proxy ...  [32mdone [0m
[15:21:38]W:	 [Step 1/1]  [1B+ popd
[15:21:38]W:	 [Step 1/1] + exit 1
[15:21:38]W:	 [Step 1/1] Process exited with code 1
[15:21:38]E:	 [Step 1/1] Process exited with code 1 (Step: Command Line)
[15:21:38] :	 [Step 1/1] Waiting for 1 service processes to complete
[15:21:38]E:	 [Step 1/1] Ant JUnit report watcher
[15:21:38]E:		 [Ant JUnit report watcher] No reports found for paths:
[15:21:38]E:		 [Ant JUnit report watcher] /home/teamcity/buildAgent/work/jailed-build/results/artifacts/junit/*.xml
[15:21:38]E:	 [Step 1/1] Step Command Line failed
[15:21:39]E: Ant JUnit report watcher
[15:21:39]E:	 [Ant JUnit report watcher] No reports found for paths:
[15:21:39]E:	 [Ant JUnit report watcher] +:results/test_bitcoin.xml
[15:21:39]E:	 [Ant JUnit report watcher] +:results/**/junit_results*.xml
[15:21:39] : Publishing internal artifacts (4s)
[15:21:43] :	 [Publishing internal artifacts] Publishing 1 file using [ArtifactsCachePublisher]
[15:21:43] :	 [Publishing internal artifacts] Publishing 1 file using [WebPublisher]
[15:21:39]W: Publishing artifacts (7s)
[15:21:39] :	 [Publishing artifacts] Collecting files to publish: [+:results/**/junit_results*.xml, +:bitcoin-abc/abc-ci-builds/gitian-osx/gitian-results => gitian-osx.tar.gz]
[15:21:39]W:	 [Publishing artifacts] Artifacts path 'results/**/junit_results*.xml' not found
[15:21:39] :	 [Publishing artifacts] Creating archive gitian-osx.tar.gz
[15:21:39] :		 [Creating archive gitian-osx.tar.gz] Creating /home/teamcity/buildAgent/temp/buildTmp/TarPreprocessor9088384818573142517/gitian-osx.tar.gz
[15:21:39] :		 [Creating archive gitian-osx.tar.gz] Archive was created, file size 209.65 KB (214683 bytes)
[15:21:45] :	 [Publishing artifacts] Publishing 1 file using [ArtifactsCachePublisher]: bitcoin-abc/abc-ci-builds/gitian-osx/gitian-results => gitian-osx.tar.gz
[15:21:45] :	 [Publishing artifacts] Publishing 1 file using [WebPublisher]: bitcoin-abc/abc-ci-builds/gitian-osx/gitian-results => gitian-osx.tar.gz
[15:21:47] : Build finished
PiRK planned changes to this revision.Apr 5 2023, 15:41
This revision is now accepted and ready to land.Apr 5 2023, 16:40