diff --git a/doc/files.md b/doc/files.md index a8912f60e..34229d875 100644 --- a/doc/files.md +++ b/doc/files.md @@ -1,109 +1,110 @@ # Bitcoin ABC file system **Contents** - [Data directory location](#data-directory-location) - [Data directory layout](#data-directory-layout) - [Multi-wallet environment](#multi-wallet-environment) - [GUI settings](#gui-settings) - [Legacy subdirectories and files](#legacy-subdirectories-and-files) - [Notes](#notes) ## Data directory location The data directory is the default location where the Bitcoin ABC files are stored. 1. The default data directory paths for supported platforms are: Platform | Data directory path ---------|-------------------- Linux | `$HOME/.bitcoin/` macOS | `$HOME/Library/Application Support/Bitcoin/` Windows | `%APPDATA%\Bitcoin\` [\[1\]](#note1) 2. The non-default data directory path can be specified by `-datadir` option. 3. All content of the data directory, except for `bitcoin.conf` file, is chain-specific. This means the actual data directory paths for non-mainnet cases differ: Chain option | Data directory path --------------------|-------------------- no option (mainnet) | *path_to_datadir*`/` `-testnet` | *path_to_datadir*`/testnet3/` `-regtest` | *path_to_datadir*`/regtest/` ## Data directory layout Subdirectory | File(s) | Description -------------------|-----------------------|------------ `blocks/` | | Blocks directory; can be specified by `-blocksdir` option (except for `blocks/index/`) `blocks/index/` | LevelDB database | Block index; `-blocksdir` option does not affect this path `blocks/` | `blkNNNNN.dat`[\[2\]](#note2) | Actual Bitcoin blocks (in network format, dumped in raw on disk, 128 MiB per file) `blocks/` | `revNNNNN.dat`[\[2\]](#note2) | Block undo data (custom format) `chainstate/` | LevelDB database | Blockchain state (a compact representation of all currently unspent transaction outputs and some metadata about the transactions they are from) `indexes/txindex/` | LevelDB database | Transaction index; *optional*, used if `-txindex=1` `indexes/blockfilter/basic/db/` | LevelDB database | Blockfilter index LevelDB database for the basic filtertype; *optional*, used if `-blockfilterindex=basic` `indexes/blockfilter/basic/` | `fltrNNNNN.dat`[\[2\]](#note2) | Blockfilter index filters for the basic filtertype; *optional*, used if `-blockfilterindex=basic` +`indexes/coinstats/db/` | LevelDB database | Coinstats index; *optional*, used if `-coinstatsindex=1` `wallets/` | | [Contains wallets](#multi-wallet-environment); can be specified by `-walletdir` option; if `wallets/` subdirectory does not exist, a wallet resides in the data directory `./` | `anchors.dat` | Anchor IP address database, created on shutdown and deleted at startup. Anchors are last known outgoing block-relay-only peers that are tried to re-connect to on startup `./` | `banlist.dat` | Stores the IPs/subnets of banned nodes `./` | `bitcoin.conf` | User-defined [configuration settings](bitcoin-conf.md) for `bitcoind` or `bitcoin-qt`. File is not written to by the software and must be created manually. Path can be specified by `-conf` option `./` | `bitcoind.pid` | Stores the process ID (PID) of `bitcoind` or `bitcoin-qt` while running; created at start and deleted on shutdown; can be specified by `-pid` option `./` | `debug.log` | Contains debug information and general logging generated by `bitcoind` or `bitcoin-qt`; can be specified by `-debuglogfile` option `./` | `guisettings.ini.bak` | Backup of former [GUI settings](#gui-settings) after `-resetguisettings` option is used `./` | `mempool.dat` | Dump of the mempool's transactions `./` | `onion_v3_private_key` | Cached Tor onion service private key for `-listenonion` option `./` | `i2p_private_key` | Private key that corresponds to our I2P address. When `-i2psam=` is specified the contents of this file is used to identify ourselves for making outgoing connections to I2P peers and possibly accepting incoming ones. Automatically generated if it does not exist. `./` | `peers.dat` | Peer IP address database (custom format) `./` | `settings.json` | Read-write settings set through GUI or RPC interfaces, augmenting manual settings from [bitcoin.conf](bitcoin-conf.md). File is created automatically if read-write settings storage is not disabled with `-nosettings` option. Path can be specified with `-settings` option `./` | `.cookie` | Session RPC authentication cookie; if used, created at start and deleted on shutdown; can be specified by `-rpccookiefile` option `./` | `.lock` | Data directory lock file ## Multi-wallet environment Wallets are Berkeley DB (BDB) databases: Subdirectory | File(s) | Description -------------|-------------------|------------ `database/` | BDB logging files | Part of BDB environment; created at start and deleted on shutdown; a user *must keep it as safe* as personal wallet `wallet.dat` `./` | `db.log` | BDB error file `./` | `wallet.dat` | Personal wallet (BDB) with keys and transactions `./` | `.walletlock` | Wallet lock file 1. Each user-defined wallet named "wallet_name" resides in `wallets/wallet_name/` subdirectory. 2. The default (unnamed) wallet resides in `wallets/` subdirectory; if the latter does not exist, the wallet resides in the data directory. 3. A wallet database path can be specified by `-wallet` option. 4. `wallet.dat` files must not be shared across different node instances, as that can result in key-reuse and double-spends due the lack of synchronization between instances. 5. Any copy or backup of the wallet should be done through a `backupwallet` call in order to update and lock the wallet, preventing any file corruption caused by updates during the copy. ## GUI settings `bitcoin-qt` uses [`QSettings`](https://doc.qt.io/qt-5/qsettings.html) class; this implies platform-specific [locations where application settings are stored](https://doc.qt.io/qt-5/qsettings.html#locations-where-application-settings-are-stored). ## Legacy subdirectories and files These subdirectories and files are no longer used by the Bitcoin ABC: Path | Description | Repository notes ---------------|-------------|----------------- `blktree/` | Blockchain index; replaced by `blocks/index/` in [0.8.0](https://github.com/bitcoin/bitcoin/blob/master/doc/release-notes/release-notes-0.8.0.md#improvements) | [PR #2231](https://github.com/bitcoin/bitcoin/pull/2231), [`8fdc94cc`](https://github.com/bitcoin/bitcoin/commit/8fdc94cc8f0341e96b1edb3a5b56811c0b20bd15) `coins/` | Unspent transaction output database; replaced by `chainstate/` in 0.8.0 | [PR #2231](https://github.com/bitcoin/bitcoin/pull/2231), [`8fdc94cc`](https://github.com/bitcoin/bitcoin/commit/8fdc94cc8f0341e96b1edb3a5b56811c0b20bd15) `blkindex.dat` | Blockchain index BDB database; replaced by {`chainstate/`, `blocks/index/`, `blocks/revNNNNN.dat`[\[2\]](#note2)} in 0.8.0 | [PR #1677](https://github.com/bitcoin/bitcoin/pull/1677) `blk000?.dat` | Block data (custom format, 2 GiB per file); replaced by `blocks/blkNNNNN.dat`[\[2\]](#note2) in 0.8.0 | [PR #1677](https://github.com/bitcoin/bitcoin/pull/1677) `addr.dat` | Peer IP address BDB database; replaced by `peers.dat` in [0.7.0](https://github.com/bitcoin/bitcoin/blob/master/doc/release-notes/release-notes-0.7.0.md) | [PR #1198](https://github.com/bitcoin/bitcoin/pull/1198), [`928d3a01`](https://github.com/bitcoin/bitcoin/commit/928d3a011cc66c7f907c4d053f674ea77dc611cc) `onion_private_key` | Cached Tor onion service private key for `-listenonion` option. Was used for Tor v2 services; replaced by `onion_v3_private_key` in [0.22.14](https://github.com/Bitcoin-ABC/bitcoin-abc/blob/master/doc/release-notes/release-notes-0.22.14.md) | [D9201](https://reviews.bitcoinabc.org/D9201) ## Notes 1. The `/` (slash, U+002F) is used as the platform-independent path component separator in this paper. 2. `NNNNN` matches `[0-9]{5}` regex. diff --git a/test/functional/rpc_blockchain.py b/test/functional/rpc_blockchain.py index c13fa515d..d560f76e0 100755 --- a/test/functional/rpc_blockchain.py +++ b/test/functional/rpc_blockchain.py @@ -1,454 +1,454 @@ #!/usr/bin/env python3 # Copyright (c) 2014-2019 The Bitcoin Core developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Test RPCs related to blockchainstate. Test the following RPCs: - getblockchaininfo - gettxoutsetinfo - getdifficulty - getbestblockhash - getblockhash - getblockheader - getchaintxstats - getnetworkhashps - verifychain Tests correspond to code in rpc/blockchain.cpp. """ import http.client import subprocess from decimal import Decimal from test_framework.blocktools import ( TIME_GENESIS_BLOCK, create_block, create_coinbase, ) from test_framework.messages import CBlockHeader, FromHex, msg_block from test_framework.p2p import P2PInterface from test_framework.test_framework import BitcoinTestFramework from test_framework.util import ( assert_equal, assert_greater_than, assert_greater_than_or_equal, assert_is_hash_string, assert_is_hex_string, assert_raises, assert_raises_rpc_error, ) class BlockchainTest(BitcoinTestFramework): def set_test_params(self): self.setup_clean_chain = True self.num_nodes = 1 # TODO: remove -txindex. Currently required for getrawtransaction call. self.extra_args = [["-txindex"]] self.supports_cli = False def run_test(self): self.mine_chain() # Set extra args with pruning after rescan is complete self.restart_node(0, extra_args=['-stopatheight=207', '-prune=1']) self._test_getblockchaininfo() self._test_getchaintxstats() self._test_gettxoutsetinfo() self._test_getblockheader() self._test_getdifficulty() self._test_getnetworkhashps() self._test_stopatheight() self._test_waitforblockheight() if self.is_wallet_compiled(): self._test_getblock() assert self.nodes[0].verifychain(4, 0) def mine_chain(self): self.log.info('Create some old blocks') address = self.nodes[0].get_deterministic_priv_key().address for t in range(TIME_GENESIS_BLOCK, TIME_GENESIS_BLOCK + 200 * 600, 600): # ten-minute steps from genesis block time self.nodes[0].setmocktime(t) self.nodes[0].generatetoaddress(1, address) assert_equal(self.nodes[0].getblockchaininfo()['blocks'], 200) def _test_getblockchaininfo(self): self.log.info("Test getblockchaininfo") keys = [ 'bestblockhash', 'blocks', 'chain', 'chainwork', 'difficulty', 'headers', 'initialblockdownload', 'mediantime', 'pruned', 'size_on_disk', 'softforks', 'verificationprogress', 'warnings', ] res = self.nodes[0].getblockchaininfo() # result should have these additional pruning keys if manual pruning is # enabled assert_equal(sorted(res.keys()), sorted( ['pruneheight', 'automatic_pruning'] + keys)) # size_on_disk should be > 0 assert_greater_than(res['size_on_disk'], 0) assert_equal(res['softforks'], { 'testdummy': { 'type': 'bip9', 'bip9': { 'status': 'started', 'bit': 28, 'start_time': 0, # testdummy does not have a timeout so is set to the max # int64 value 'timeout': 0x7fffffffffffffff, 'since': 144, 'statistics': { 'period': 144, 'threshold': 108, 'elapsed': 57, 'count': 57, 'possible': True, }, }, 'active': False, }, }) # pruneheight should be greater or equal to 0 assert_greater_than_or_equal(res['pruneheight'], 0) # check other pruning fields given that prune=1 assert res['pruned'] assert not res['automatic_pruning'] self.restart_node(0, ['-stopatheight=207']) res = self.nodes[0].getblockchaininfo() # should have exact keys assert_equal(sorted(res.keys()), keys) self.restart_node(0, ['-stopatheight=207', '-prune=550']) res = self.nodes[0].getblockchaininfo() # result should have these additional pruning keys if prune=550 assert_equal(sorted(res.keys()), sorted( ['pruneheight', 'automatic_pruning', 'prune_target_size'] + keys)) # check related fields assert res['pruned'] assert_equal(res['pruneheight'], 0) assert res['automatic_pruning'] assert_equal(res['prune_target_size'], 576716800) assert_greater_than(res['size_on_disk'], 0) def _test_getchaintxstats(self): self.log.info("Test getchaintxstats") # Test `getchaintxstats` invalid extra parameters assert_raises_rpc_error( -1, 'getchaintxstats', self.nodes[0].getchaintxstats, 0, '', 0) # Test `getchaintxstats` invalid `nblocks` assert_raises_rpc_error( -1, "JSON value is not an integer as expected", self.nodes[0].getchaintxstats, '') assert_raises_rpc_error( -8, "Invalid block count: should be between 0 and the block's height - 1", self.nodes[0].getchaintxstats, -1) assert_raises_rpc_error(-8, "Invalid block count: should be between 0 and the block's height - 1", self.nodes[ 0].getchaintxstats, self.nodes[0].getblockcount()) # Test `getchaintxstats` invalid `blockhash` assert_raises_rpc_error( -1, "JSON value is not a string as expected", self.nodes[0].getchaintxstats, blockhash=0) assert_raises_rpc_error(-8, "blockhash must be of length 64 (not 1, for '0')", self.nodes[0].getchaintxstats, blockhash='0') assert_raises_rpc_error( -8, "blockhash must be hexadecimal string (not 'ZZZ0000000000000000000000000000000000000000000000000000000000000')", self.nodes[0].getchaintxstats, blockhash='ZZZ0000000000000000000000000000000000000000000000000000000000000') assert_raises_rpc_error( -5, "Block not found", self.nodes[0].getchaintxstats, blockhash='0000000000000000000000000000000000000000000000000000000000000000') blockhash = self.nodes[0].getblockhash(200) self.nodes[0].invalidateblock(blockhash) assert_raises_rpc_error( -8, "Block is not in main chain", self.nodes[0].getchaintxstats, blockhash=blockhash) self.nodes[0].reconsiderblock(blockhash) chaintxstats = self.nodes[0].getchaintxstats(nblocks=1) # 200 txs plus genesis tx assert_equal(chaintxstats['txcount'], 201) # tx rate should be 1 per 10 minutes, or 1/600 # we have to round because of binary math assert_equal(round(chaintxstats['txrate'] * 600, 10), Decimal(1)) b1_hash = self.nodes[0].getblockhash(1) b1 = self.nodes[0].getblock(b1_hash) b200_hash = self.nodes[0].getblockhash(200) b200 = self.nodes[0].getblock(b200_hash) time_diff = b200['mediantime'] - b1['mediantime'] chaintxstats = self.nodes[0].getchaintxstats() assert_equal(chaintxstats['time'], b200['time']) assert_equal(chaintxstats['txcount'], 201) assert_equal(chaintxstats['window_final_block_hash'], b200_hash) assert_equal(chaintxstats['window_final_block_height'], 200) assert_equal(chaintxstats['window_block_count'], 199) assert_equal(chaintxstats['window_tx_count'], 199) assert_equal(chaintxstats['window_interval'], time_diff) assert_equal( round(chaintxstats['txrate'] * time_diff, 10), Decimal(199)) chaintxstats = self.nodes[0].getchaintxstats(blockhash=b1_hash) assert_equal(chaintxstats['time'], b1['time']) assert_equal(chaintxstats['txcount'], 2) assert_equal(chaintxstats['window_final_block_hash'], b1_hash) assert_equal(chaintxstats['window_final_block_height'], 1) assert_equal(chaintxstats['window_block_count'], 0) assert 'window_tx_count' not in chaintxstats assert 'window_interval' not in chaintxstats assert 'txrate' not in chaintxstats def _test_gettxoutsetinfo(self): node = self.nodes[0] res = node.gettxoutsetinfo() assert_equal(res['total_amount'], Decimal('8725000000.00')) assert_equal(res['transactions'], 200) assert_equal(res['height'], 200) assert_equal(res['txouts'], 200) assert_equal(res['bogosize'], 15000), assert_equal(res['bestblock'], node.getblockhash(200)) size = res['disk_size'] assert size > 6400 assert size < 64000 assert_equal(len(res['bestblock']), 64) assert_equal(len(res['hash_serialized']), 64) self.log.info( "Test that gettxoutsetinfo() works for blockchain with just the genesis block") b1hash = node.getblockhash(1) node.invalidateblock(b1hash) res2 = node.gettxoutsetinfo() assert_equal(res2['transactions'], 0) assert_equal(res2['total_amount'], Decimal('0')) assert_equal(res2['height'], 0) assert_equal(res2['txouts'], 0) assert_equal(res2['bogosize'], 0), assert_equal(res2['bestblock'], node.getblockhash(0)) assert_equal(len(res2['hash_serialized']), 64) self.log.info( "Test that gettxoutsetinfo() returns the same result after invalidate/reconsider block") node.reconsiderblock(b1hash) res3 = node.gettxoutsetinfo() # The field 'disk_size' is non-deterministic and can thus not be # compared between res and res3. Everything else should be the same. del res['disk_size'], res3['disk_size'] assert_equal(res, res3) self.log.info("Test hash_type option for gettxoutsetinfo()") # Adding hash_type 'hash_serialized', which is the default, should # not change the result. res4 = node.gettxoutsetinfo(hash_type='hash_serialized') del res4['disk_size'] assert_equal(res, res4) # hash_type none should not return a UTXO set hash. res5 = node.gettxoutsetinfo(hash_type='none') assert 'hash_serialized' not in res5 # hash_type muhash should return a different UTXO set hash. res6 = node.gettxoutsetinfo(hash_type='muhash') assert 'muhash' in res6 assert(res['hash_serialized'] != res6['muhash']) - # muhash should not be included in gettxoutset unless requested. + # muhash should not be returned unless requested. for r in [res, res2, res3, res4, res5]: assert 'muhash' not in r # Unknown hash_type raises an error assert_raises_rpc_error(-8, "foohash is not a valid hash_type", node.gettxoutsetinfo, "foohash") def _test_getblockheader(self): node = self.nodes[0] assert_raises_rpc_error(-8, "hash must be of length 64 (not 8, for 'nonsense')", node.getblockheader, "nonsense") assert_raises_rpc_error( -8, "hash must be hexadecimal string (not 'ZZZ7bb8b1697ea987f3b223ba7819250cae33efacb068d23dc24859824a77844')", node.getblockheader, "ZZZ7bb8b1697ea987f3b223ba7819250cae33efacb068d23dc24859824a77844") assert_raises_rpc_error(-5, "Block not found", node.getblockheader, "0cf7bb8b1697ea987f3b223ba7819250cae33efacb068d23dc24859824a77844") besthash = node.getbestblockhash() secondbesthash = node.getblockhash(199) header = node.getblockheader(blockhash=besthash) assert_equal(header['hash'], besthash) assert_equal(header['height'], 200) assert_equal(header['confirmations'], 1) assert_equal(header['previousblockhash'], secondbesthash) assert_is_hex_string(header['chainwork']) assert_equal(header['nTx'], 1) assert_is_hash_string(header['hash']) assert_is_hash_string(header['previousblockhash']) assert_is_hash_string(header['merkleroot']) assert_is_hash_string(header['bits'], length=None) assert isinstance(header['time'], int) assert isinstance(header['mediantime'], int) assert isinstance(header['nonce'], int) assert isinstance(header['version'], int) assert isinstance(int(header['versionHex'], 16), int) assert isinstance(header['difficulty'], Decimal) # Test with verbose=False, which should return the header as hex. header_hex = node.getblockheader(blockhash=besthash, verbose=False) assert_is_hex_string(header_hex) header = FromHex(CBlockHeader(), header_hex) header.calc_sha256() assert_equal(header.hash, besthash) assert 'previousblockhash' not in node.getblockheader( node.getblockhash(0)) assert 'nextblockhash' not in node.getblockheader( node.getbestblockhash()) def _test_getdifficulty(self): difficulty = self.nodes[0].getdifficulty() # 1 hash in 2 should be valid, so difficulty should be 1/2**31 # binary => decimal => binary math is why we do this check assert abs(difficulty * 2**31 - 1) < 0.0001 def _test_getnetworkhashps(self): hashes_per_second = self.nodes[0].getnetworkhashps() # This should be 2 hashes every 10 minutes or 1/300 assert abs(hashes_per_second * 300 - 1) < 0.0001 def _test_stopatheight(self): assert_equal(self.nodes[0].getblockcount(), 200) self.nodes[0].generatetoaddress( 6, self.nodes[0].get_deterministic_priv_key().address) assert_equal(self.nodes[0].getblockcount(), 206) self.log.debug('Node should not stop at this height') assert_raises(subprocess.TimeoutExpired, lambda: self.nodes[0].process.wait(timeout=3)) try: self.nodes[0].generatetoaddress( 1, self.nodes[0].get_deterministic_priv_key().address) except (ConnectionError, http.client.BadStatusLine): pass # The node already shut down before response self.log.debug('Node should stop at this height...') self.nodes[0].wait_until_stopped() self.start_node(0) assert_equal(self.nodes[0].getblockcount(), 207) def _test_waitforblockheight(self): self.log.info("Test waitforblockheight") node = self.nodes[0] peer = node.add_p2p_connection(P2PInterface()) current_height = node.getblock(node.getbestblockhash())['height'] # Create a fork somewhere below our current height, invalidate the tip # of that fork, and then ensure that waitforblockheight still # works as expected. # # (Previously this was broken based on setting # `rpc/blockchain.cpp:latestblock` incorrectly.) # b20hash = node.getblockhash(20) b20 = node.getblock(b20hash) def solve_and_send_block(prevhash, height, time): b = create_block(prevhash, create_coinbase(height), time) b.solve() peer.send_and_ping(msg_block(b)) return b b21f = solve_and_send_block(int(b20hash, 16), 21, b20['time'] + 1) b22f = solve_and_send_block(b21f.sha256, 22, b21f.nTime + 1) node.invalidateblock(b22f.hash) def assert_waitforheight(height, timeout=2): assert_equal( node.waitforblockheight( height=height, timeout=timeout)['height'], current_height) assert_waitforheight(0) assert_waitforheight(current_height - 1) assert_waitforheight(current_height) assert_waitforheight(current_height + 1) def _test_getblock(self): # Checks for getblock verbose outputs node = self.nodes[0] (blockhash, nextblockhash) = node.generate(2) blockinfo = node.getblock(blockhash, 2) transactioninfo = node.gettransaction(blockinfo['tx'][0]['txid']) blockheaderinfo = node.getblockheader(blockhash, True) assert_equal(blockinfo['hash'], transactioninfo['blockhash']) assert_equal( blockinfo['confirmations'], transactioninfo['confirmations']) assert_equal(blockinfo['height'], blockheaderinfo['height']) assert_equal(blockinfo['versionHex'], blockheaderinfo['versionHex']) assert_equal(blockinfo['version'], blockheaderinfo['version']) assert_equal(blockinfo['size'], 181) assert_equal(blockinfo['merkleroot'], blockheaderinfo['merkleroot']) # Verify transaction data by check the hex values for tx in blockinfo['tx']: rawtransaction = node.getrawtransaction(tx['txid'], True) assert_equal(tx['hex'], rawtransaction['hex']) assert_equal(blockinfo['time'], blockheaderinfo['time']) assert_equal(blockinfo['mediantime'], blockheaderinfo['mediantime']) assert_equal(blockinfo['nonce'], blockheaderinfo['nonce']) assert_equal(blockinfo['bits'], blockheaderinfo['bits']) assert_equal(blockinfo['difficulty'], blockheaderinfo['difficulty']) assert_equal(blockinfo['chainwork'], blockheaderinfo['chainwork']) assert_equal( blockinfo['previousblockhash'], blockheaderinfo['previousblockhash']) assert_equal(blockinfo['nextblockhash'], nextblockhash) assert_equal( blockinfo['nextblockhash'], blockheaderinfo['nextblockhash']) assert 'previousblockhash' not in node.getblock(node.getblockhash(0)) assert 'nextblockhash' not in node.getblock(node.getbestblockhash()) if __name__ == '__main__': BlockchainTest().main()