Page MenuHomePhabricator

[backport#18637] coins: allow cache resize after init
ClosedPublic

Authored by majcosta on Feb 23 2021, 10:46.

Details

Reviewers
PiRK
Group Reviewers
Restricted Project
Commits
rABC8af6d9df957b: [backport#18637] coins: allow cache resize after init
Summary

f19fdd47a6371dcbe0760ef6f3c3c5adb31b1bb4 test: add test for CChainState::ResizeCoinsCaches() (James O'Beirne)
8ac3ef46999ed676ca3775f7b2f461d92f09a542 add ChainstateManager::MaybeRebalanceCaches() (James O'Beirne)
f36aaa6392fdbdac6891d92202d3efeff98754f4 Add CChainState::ResizeCoinsCaches (James O'Beirne)
b223111da2e0e9ceccef75df8a20252b0094b7bc txdb: add CCoinsViewDB::ChangeCacheSize (James O'Beirne)

Pull request description:

This is part of the [assumeutxo project](https://github.com/bitcoin/bitcoin/projects/11):

Parent PR: #15606
Issue: #15605
Specification: https://github.com/jamesob/assumeutxo-docs/tree/master/proposal

---

In the assumeutxo implementation draft (#15056), once a UTXO snapshot is loaded, a new chainstate object is created after initialization. This means that we have to reclaim some of the cache that we've allocated to the original chainstate (per `dbcache=`) to repurpose for the snapshot chainstate.

Furthermore, it makes sense to have different cache allocations depending on which chainstate is more active. While the snapshot chainstate is working to get to the network tip (and the background validation chainstate is idle), it makes sense that the snapshot chainstate should have the majority of cache allocation. And contrariwise once the snapshot has reached network tip, most of the cache should be given to the background validation chainstate.

This set of changes (detailed in the commit messages) allows us to dynamically resize the various coins caches. None of the functionality introduced here is used at the moment, but will be in the next AU PR (which introduces `ActivateSnapshot`).

`ChainstateManager::MaybeRebalanceCaches()` defines the (somewhat normative) cache allocations between the snapshot and background validation chainstates. I'd be interested in feedback if anyone has thoughts on the proportions I've set there.

Backport of core#18637

Test Plan
ninja all check check-functional

Event Timeline

Tail of the build log:

wallet_import_rescan.py                          | ○ Skipped | 0 s
wallet_import_with_label.py                      | ○ Skipped | 0 s
wallet_importdescriptors.py                      | ○ Skipped | 0 s
wallet_importmulti.py                            | ○ Skipped | 0 s
wallet_importprunedfunds.py                      | ○ Skipped | 0 s
wallet_keypool.py                                | ○ Skipped | 0 s
wallet_keypool_topup.py                          | ○ Skipped | 0 s
wallet_labels.py                                 | ○ Skipped | 0 s
wallet_listreceivedby.py                         | ○ Skipped | 0 s
wallet_listsinceblock.py                         | ○ Skipped | 0 s
wallet_listtransactions.py                       | ○ Skipped | 0 s
wallet_multiwallet.py                            | ○ Skipped | 0 s
wallet_multiwallet.py --usecli                   | ○ Skipped | 0 s
wallet_reorgsrestore.py                          | ○ Skipped | 0 s
wallet_resendwallettransactions.py               | ○ Skipped | 0 s
wallet_txn_clone.py                              | ○ Skipped | 0 s
wallet_txn_clone.py --mineblock                  | ○ Skipped | 0 s
wallet_txn_doublespend.py                        | ○ Skipped | 0 s
wallet_txn_doublespend.py --mineblock            | ○ Skipped | 0 s
wallet_watchonly.py                              | ○ Skipped | 0 s
wallet_watchonly.py --usecli                     | ○ Skipped | 0 s
wallet_zapwallettxes.py                          | ○ Skipped | 0 s

ALL                                              | ✓ Passed  | 437 s (accumulated) 
Runtime: 102 s

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

[148/404] Running avalanche test suite
PASSED: avalanche test suite
[157/404] Test Bitcoin RPC authentication...
...
----------------------------------------------------------------------
Ran 3 tests in 0.005s

OK
[158/404] cd /work/contrib/devtools/chainparams && /usr/bin/python3.7 ./test_make_chainparams.py
.....
----------------------------------------------------------------------
Ran 5 tests in 0.007s

OK
[172/404] Running seeder test suite
PASSED: seeder test suite
[175/404] Running pow test suite
PASSED: pow test suite
[183/404] Running bitcoin-qt test suite
PASSED: bitcoin-qt test suite
[185/404] Building CXX object src/test/CMakeFiles/test_bitcoin.dir/script_tests.cpp.o
In file included from /usr/include/boost/test/unit_test.hpp:19,
                 from ../../src/test/script_tests.cpp:30:
../../src/test/script_tests.cpp: In member function ‘void script_tests::script_build::test_method()’:
../../src/test/script_tests.cpp:540:22: note: variable tracking size limit exceeded with -fvar-tracking-assignments, retrying without
 BOOST_AUTO_TEST_CASE(script_build) {
                      ^~~~~~~~~~~~
ninja: build stopped: cannot make progress due to previous errors.
Build build-without-wallet failed with exit code 1

Tail of the build log:

wallet_createwallet.py --usecli                  | ✓ Passed  | 3 s
wallet_descriptor.py                             | ✓ Passed  | 12 s
wallet_disable.py                                | ✓ Passed  | 1 s
wallet_dump.py                                   | ✓ Passed  | 5 s
wallet_encryption.py                             | ✓ Passed  | 6 s
wallet_groups.py                                 | ✓ Passed  | 57 s
wallet_hd.py                                     | ✓ Passed  | 6 s
wallet_import_rescan.py                          | ✓ Passed  | 9 s
wallet_import_with_label.py                      | ✓ Passed  | 1 s
wallet_importdescriptors.py                      | ✓ Passed  | 5 s
wallet_importmulti.py                            | ✓ Passed  | 5 s
wallet_importprunedfunds.py                      | ✓ Passed  | 1 s
wallet_keypool.py                                | ✓ Passed  | 3 s
wallet_keypool_topup.py                          | ✓ Passed  | 3 s
wallet_labels.py                                 | ✓ Passed  | 2 s
wallet_listreceivedby.py                         | ✓ Passed  | 19 s
wallet_listsinceblock.py                         | ✓ Passed  | 6 s
wallet_listtransactions.py                       | ✓ Passed  | 9 s
wallet_multiwallet.py                            | ✓ Passed  | 11 s
wallet_multiwallet.py --usecli                   | ✓ Passed  | 13 s
wallet_reorgsrestore.py                          | ✓ Passed  | 3 s
wallet_resendwallettransactions.py               | ✓ Passed  | 10 s
wallet_txn_clone.py                              | ✓ Passed  | 3 s
wallet_txn_clone.py --mineblock                  | ✓ Passed  | 4 s
wallet_txn_doublespend.py                        | ✓ Passed  | 3 s
wallet_txn_doublespend.py --mineblock            | ✓ Passed  | 4 s
wallet_watchonly.py                              | ✓ Passed  | 1 s
wallet_watchonly.py --usecli                     | ✓ Passed  | 1 s
wallet_zapwallettxes.py                          | ✓ Passed  | 6 s

ALL                                              | ✓ Passed  | 1249 s (accumulated) 
Runtime: 250 s

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

[167/444] Test Bitcoin RPC authentication...
...
----------------------------------------------------------------------
Ran 3 tests in 0.004s

OK
[169/444] cd /work/contrib/devtools/chainparams && /usr/bin/python3.7 ./test_make_chainparams.py
.....
----------------------------------------------------------------------
Ran 5 tests in 0.001s

OK
[176/444] Running avalanche test suite
PASSED: avalanche test suite
[192/444] Running seeder test suite
PASSED: seeder test suite
[200/444] Running pow test suite
PASSED: pow test suite
[207/444] Running bitcoin-qt test suite
PASSED: bitcoin-qt test suite
ninja: build stopped: cannot make progress due to previous errors.
Build build-debug failed with exit code 1

Tail of the build log:

rpc_psbt.py                                      | ✓ Passed  | 29 s
rpc_rawtransaction.py                            | ✓ Passed  | 38 s
rpc_scantxoutset.py                              | ✓ Passed  | 3 s
rpc_setban.py                                    | ✓ Passed  | 2 s
rpc_signmessage.py                               | ✓ Passed  | 1 s
rpc_signrawtransaction.py                        | ✓ Passed  | 1 s
rpc_txoutproof.py                                | ✓ Passed  | 2 s
rpc_uptime.py                                    | ✓ Passed  | 1 s
rpc_users.py                                     | ✓ Passed  | 5 s
rpc_whitelist.py                                 | ✓ Passed  | 1 s
tool_wallet.py                                   | ✓ Passed  | 4 s
wallet_abandonconflict.py                        | ✓ Passed  | 8 s
wallet_address_types.py                          | ✓ Passed  | 12 s
wallet_avoidreuse.py                             | ✓ Passed  | 5 s
wallet_backup.py                                 | ✓ Passed  | 24 s
wallet_balance.py                                | ✓ Passed  | 18 s
wallet_basic.py                                  | ✓ Passed  | 26 s
wallet_coinbase_category.py                      | ✓ Passed  | 1 s
wallet_create_tx.py                              | ✓ Passed  | 6 s
wallet_createwallet.py                           | ✓ Passed  | 2 s
wallet_createwallet.py --usecli                  | ✓ Passed  | 2 s
wallet_descriptor.py                             | ✓ Passed  | 7 s
wallet_disable.py                                | ✓ Passed  | 1 s
wallet_dump.py                                   | ✓ Passed  | 5 s
wallet_encryption.py                             | ✓ Passed  | 5 s
wallet_groups.py                                 | ✓ Passed  | 40 s
wallet_hd.py                                     | ✓ Passed  | 5 s
wallet_import_rescan.py                          | ✓ Passed  | 5 s
wallet_import_with_label.py                      | ✓ Passed  | 1 s
wallet_importdescriptors.py                      | ✓ Passed  | 5 s
wallet_importmulti.py                            | ✓ Passed  | 3 s
wallet_importprunedfunds.py                      | ✓ Passed  | 2 s
wallet_keypool.py                                | ✓ Passed  | 3 s
wallet_keypool_topup.py                          | ✓ Passed  | 2 s
wallet_labels.py                                 | ✓ Passed  | 1 s
wallet_listreceivedby.py                         | ✓ Passed  | 29 s
wallet_listsinceblock.py                         | ✓ Passed  | 3 s
wallet_listtransactions.py                       | ✓ Passed  | 23 s
wallet_multiwallet.py                            | ✓ Passed  | 11 s
wallet_multiwallet.py --usecli                   | ✓ Passed  | 12 s
wallet_reorgsrestore.py                          | ✓ Passed  | 3 s
wallet_resendwallettransactions.py               | ✓ Passed  | 15 s
wallet_txn_clone.py                              | ✓ Passed  | 1 s
wallet_txn_clone.py --mineblock                  | ✓ Passed  | 3 s
wallet_txn_doublespend.py                        | ✓ Passed  | 1 s
wallet_txn_doublespend.py --mineblock            | ✓ Passed  | 3 s
wallet_watchonly.py                              | ✓ Passed  | 1 s
wallet_watchonly.py --usecli                     | ✓ Passed  | 1 s
wallet_zapwallettxes.py                          | ✓ Passed  | 3 s

ALL                                              | ✓ Passed  | 974 s (accumulated) 
Runtime: 195 s

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

ninja: build stopped: cannot make progress due to previous errors.
Build build-diff failed with exit code 1

fix -Werror=ignored-qualifiers

PiRK added a subscriber: PiRK.

I'm not blocking on this, but FYI there are a bunch of same line comments in validation_chainstate_tests.cpp

This revision is now accepted and ready to land.Feb 23 2021, 14:08

modified comments pertaining to function arguments to not be reliant on a particular formatting (// to /* */)