Page MenuHomePhabricator

Merge #16957: 9% less memory: make SaltedOutpointHasher noexcept
ClosedPublic

Authored by jasonbcox on Oct 20 2020, 00:20.

Details

Summary

67d99900b0d770038c9c5708553143137b124a6c make SaltedOutpointHasher noexcept (Martin Ankerl)

Pull request description:

If the hash is not `noexcept`, `unorderd_map` has to assume that it can throw an exception. Thus when rehashing care needs to be taken. libstdc++ solves this by simply caching the hash value, which increases memory of each node by 8 bytes. Adding `noexcept` prevents this caching. In my experiments with `-reindex-chainstate -stopatheight=594000`, memory usage (maximum resident set size) has decreased by 9.4% while runtime has increased by 1.6% due to additional hashing. Additionally, memusage::DynamicUsage() is now more accurate and does not underestimate.

|                                       | runtime h:mm:ss | max RSS kbyte |
|---------------------------------------|-----------------|--------------|
| master                                |         4:13:59 |      7696728 |
| 2019-09-SaltedOutpointHasher-noexcept |         4:18:11 |      6971412 |
| change                                |          +1.65% |       -9,42% |

Comparison of progress masters vs. 2019-09-SaltedOutpointHasher-noexcept
![out](https://user-images.githubusercontent.com/14386/65541887-69424e00-df0e-11e9-8644-b3a068ed8c3f.png)

Backport of Core PR16957

Release notes were backported from Core PR17422

Test Plan

ninja check check-functional for sanity

Event Timeline

Tail of the build log:

  -t abc-base-image-"${HASH}" debian:buster

(cat <<EOF
results() {
  set +e
  shopt -s dotglob nullglob
  mv /work/abc-ci-builds/"build-debug"/* /results
  chown -R ${ME} /work
  chown -R ${ME} /results
  chown -R ${ME} /root/.ccache
}
trap "results" EXIT
export TEAMCITY_VERSION="2019.2.4 (build 72059)"
export BASE_CACHE="/root/abc-depends/cache"
export SDK_ARCHIVE_DIR="/root/abc-depends/osx-sdk"
export SOURCES_PATH="/root/abc-depends/sources"
mkdir -p "/root/abc-depends/cache" "/root/abc-depends/osx-sdk" "/root/abc-depends/sources"
./contrib/teamcity/build-configurations.py "build-debug"
EOF
) > run-command.sh
chmod +x run-command.sh

~/infra/docker/docker-run.sh \
  -a "-v /home/teamcity/.ccache:/root/.ccache -v /home/teamcity/.abc-depends:/root/abc-depends -v "${RESULTS_DIR}":/results" \
  -c run-command.sh /work/run-command.sh abc-base-image-"${HASH}" ./run-command.sh

[00:21:00] :	 [Step 1/1] Starting: /home/teamcity/buildAgent/temp/agentTmp/custom_script5016478172529715348
[00:21:00] :	 [Step 1/1] in directory: /home/teamcity/buildAgent/work/jailed-build
[00:21:00] :	 [Step 1/1] ~/buildAgent/work/jailed-build/bitcoin-abc ~/buildAgent/work/jailed-build
[00:21:00] :	 [Step 1/1] ~/buildAgent/work/jailed-build
[00:21:00] :	 [Step 1/1] Building base image for: eb0957adb...
[00:21:00] :	 [Step 1/1] ~/buildAgent/work/jailed-build/bitcoin-abc ~/buildAgent/work/jailed-build
[00:21:01] :	 [Step 1/1] ~/buildAgent/work/jailed-build
[00:21:01] :	 [Step 1/1] Tag name: abc-base-image-eb0957adb
[00:21:11]W:	 [Step 1/1] Traceback (most recent call last):
[00:21:11]W:	 [Step 1/1]   File "./contrib/teamcity/build-configurations.py", line 570, in <module>
[00:21:11]W:	 [Step 1/1]     main()
[00:21:11]W:	 [Step 1/1]   File "./contrib/teamcity/build-configurations.py", line 559, in main
[00:21:11]W:	 [Step 1/1]     script_dir, config_path, args.build)
[00:21:11]W:	 [Step 1/1]   File "./contrib/teamcity/build-configurations.py", line 59, in __init__
[00:21:11]W:	 [Step 1/1]     self.load(build_name)
[00:21:11]W:	 [Step 1/1]   File "./contrib/teamcity/build-configurations.py", line 82, in load
[00:21:11]W:	 [Step 1/1]     self.name, list(config.keys())
[00:21:11]W:	 [Step 1/1] AssertionError: build-debug is not a valid build identifier. Valid identifiers are ['templates', 'builds']
[00:21:11]W:	 [Step 1/1] mv: missing destination file operand after '/results'
[00:21:11]W:	 [Step 1/1] Try 'mv --help' for more information.
[00:21:14]W:	 [Step 1/1] Process exited with code 1
[00:21:14]E:	 [Step 1/1] Process exited with code 1 (Step: Command Line)
[00:21:15]E:	 [Step 1/1] Step Command Line failed
[00:21:15]E: Ant JUnit report watcher
[00:21:15]E:	 [Ant JUnit report watcher] No reports found for paths:
[00:21:15]E:	 [Ant JUnit report watcher] +:results/test_bitcoin.xml
[00:21:15]E:	 [Ant JUnit report watcher] +:results/**/junit_results*.xml
[00:21:15] : Publishing internal artifacts (1s)
[00:21:17] :	 [Publishing internal artifacts] Publishing 1 file using [WebPublisher]
[00:21:17] :	 [Publishing internal artifacts] Publishing 1 file using [ArtifactsCachePublisher]
[00:21:15]W: Publishing artifacts (1s)
[00:21:15] :	 [Publishing artifacts] Collecting files to publish: [+:results/**/junit_results*.xml]
[00:21:15]W:	 [Publishing artifacts] Artifacts path 'results/**/junit_results*.xml' not found
[00:21:17] : Build finished
This revision is now accepted and ready to land.Oct 20 2020, 01:08