diff --git a/test/functional/README.md b/test/functional/README.md index 651b01f18..e6c484970 100644 --- a/test/functional/README.md +++ b/test/functional/README.md @@ -1,108 +1,108 @@ Regression tests ================ ### [test_framework/authproxy.py](test_framework/authproxy.py) Taken from the [python-bitcoinrpc repository](https://github.com/jgarzik/python-bitcoinrpc). ### [test_framework/test_framework.py](test_framework/test_framework.py) Base class for new regression tests. ### [test_framework/util.py](test_framework/util.py) Generally useful functions. ### [test_framework/mininode.py](test_framework/mininode.py) Basic code to support p2p connectivity to a bitcoind. ### [test_framework/comptool.py](test_framework/comptool.py) Framework for comparison-tool style, p2p tests. ### [test_framework/script.py](test_framework/script.py) Utilities for manipulating transaction scripts (originally from python-bitcoinlib) ### [test_framework/blockstore.py](test_framework/blockstore.py) Implements disk-backed block and tx storage. ### [test_framework/key.py](test_framework/key.py) Wrapper around OpenSSL EC_Key (originally from python-bitcoinlib) ### [test_framework/bignum.py](test_framework/bignum.py) Helpers for script.py ### [test_framework/blocktools.py](test_framework/blocktools.py) Helper functions for creating blocks and transactions. P2P test design notes --------------------- ## Mininode * ```mininode.py``` contains all the definitions for objects that pass over the network (```CBlock```, ```CTransaction```, etc, along with the network-level wrappers for them, ```msg_block```, ```msg_tx```, etc). * P2P tests have two threads. One thread handles all network communication with the bitcoind(s) being tested (using python's asyncore package); the other implements the test logic. * ```NodeConn``` is the class used to connect to a bitcoind. If you implement a callback class that derives from ```NodeConnCB``` and pass that to the ```NodeConn``` object, your code will receive the appropriate callbacks when events of interest arrive. * You can pass the same handler to multiple ```NodeConn```'s if you like, or pass different ones to each -- whatever makes the most sense for your test. * Call ```NetworkThread.start()``` after all ```NodeConn``` objects are created to start the networking thread. (Continue with the test logic in your existing thread.) * RPC calls are available in p2p tests. * Can be used to write free-form tests, where specific p2p-protocol behavior -is tested. Examples: ```p2p-accept-block.py```, ```maxblocksinflight.py```. +is tested. Examples: ```p2p-accept-block.py```, ```p2p-compactblocks.py```. ## Comptool * Testing framework for writing tests that compare the block/tx acceptance behavior of a bitcoind against 1 or more other bitcoind instances, or against known outcomes, or both. * Set the ```num_nodes``` variable (defined in ```ComparisonTestFramework```) to start up 1 or more nodes. If using 1 node, then ```--testbinary``` can be used as a command line option to change the bitcoind binary used by the test. If using 2 or more nodes, then ```--refbinary``` can be optionally used to change the bitcoind that will be used on nodes 2 and up. * Implement a (generator) function called ```get_tests()``` which yields ```TestInstance```s. Each ```TestInstance``` consists of: - a list of ```[object, outcome, hash]``` entries * ```object``` is a ```CBlock```, ```CTransaction```, or ```CBlockHeader```. ```CBlock```'s and ```CTransaction```'s are tested for acceptance. ```CBlockHeader```s can be used so that the test runner can deliver complete headers-chains when requested from the bitcoind, to allow writing tests where blocks can be delivered out of order but still processed by headers-first bitcoind's. * ```outcome``` is ```True```, ```False```, or ```None```. If ```True``` or ```False```, the tip is compared with the expected tip -- either the block passed in, or the hash specified as the optional 3rd entry. If ```None``` is specified, then the test will compare all the bitcoind's being tested to see if they all agree on what the best tip is. * ```hash``` is the block hash of the tip to compare against. Optional to specify; if left out then the hash of the block passed in will be used as the expected tip. This allows for specifying an expected tip while testing the handling of either invalid blocks or blocks delivered out of order, which complete a longer chain. - ```sync_every_block```: ```True/False```. If ```False```, then all blocks are inv'ed together, and the test runner waits until the node receives the last one, and tests only the last block for tip acceptance using the outcome and specified tip. If ```True```, then each block is tested in sequence and synced (this is slower when processing many blocks). - ```sync_every_transaction```: ```True/False```. Analogous to ```sync_every_block```, except if the outcome on the last tx is "None", then the contents of the entire mempool are compared across all bitcoind connections. If ```True``` or ```False```, then only the last tx's acceptance is tested against the given outcome. * For examples of tests written in this framework, see ```invalidblockrequest.py``` and ```p2p-fullblocktest.py```. diff --git a/test/functional/maxblocksinflight.py b/test/functional/maxblocksinflight.py deleted file mode 100755 index c8896afb2..000000000 --- a/test/functional/maxblocksinflight.py +++ /dev/null @@ -1,99 +0,0 @@ -#!/usr/bin/env python3 -# Copyright (c) 2015-2016 The Bitcoin Core developers -# Distributed under the MIT software license, see the accompanying -# file COPYING or http://www.opensource.org/licenses/mit-license.php. - -from test_framework.mininode import * -from test_framework.test_framework import BitcoinTestFramework -from test_framework.util import * - -''' -In this test we connect to one node over p2p, send it numerous inv's, and -compare the resulting number of getdata requests to a max allowed value. We -test for exceeding 128 blocks in flight, which was the limit an 0.9 client will -reach. [0.10 clients shouldn't request more than 16 from a single peer.] -''' -MAX_REQUESTS = 128 - - -class TestManager(NodeConnCB): - def on_getdata(self, conn, message): - self.log.debug("got getdata %s" % repr(message)) - # Log the requests - for inv in message.inv: - if inv.hash not in self.blockReqCounts: - self.blockReqCounts[inv.hash] = 0 - self.blockReqCounts[inv.hash] += 1 - - def on_close(self, conn): - if not self.disconnectOkay: - raise EarlyDisconnectError(0) - - def add_new_connection(self, connection): - super().add_connection(connection) - self.blockReqCounts = {} - self.disconnectOkay = False - - def run(self): - self.connection.rpc.generate(1) # Leave IBD - - numBlocksToGenerate = [8, 16, 128, 1024] - for count in range(len(numBlocksToGenerate)): - current_invs = [] - for i in range(numBlocksToGenerate[count]): - current_invs.append(CInv(2, random.randrange(0, 1 << 256))) - if len(current_invs) >= 50000: - self.connection.send_message(msg_inv(current_invs)) - current_invs = [] - if len(current_invs) > 0: - self.connection.send_message(msg_inv(current_invs)) - - # Wait and see how many blocks were requested - time.sleep(2) - - total_requests = 0 - with mininode_lock: - for key in self.blockReqCounts: - total_requests += self.blockReqCounts[key] - if self.blockReqCounts[key] > 1: - raise AssertionError( - "Error, test failed: block %064x requested more than once" % key) - if total_requests > MAX_REQUESTS: - raise AssertionError( - "Error, too many blocks (%d) requested" % total_requests) - self.log.info("Round %d: success (total requests: %d)" % - (count, total_requests)) - - self.disconnectOkay = True - self.connection.disconnect_node() - - -class MaxBlocksInFlightTest(BitcoinTestFramework): - - def add_options(self, parser): - parser.add_option("--testbinary", dest="testbinary", - default=os.getenv("BITCOIND", "bitcoind"), - help="Binary to test max block requests behavior") - - def __init__(self): - super().__init__() - self.setup_clean_chain = True - self.num_nodes = 1 - - def setup_network(self): - self.nodes = start_nodes(self.num_nodes, self.options.tmpdir, - extra_args=[['-whitelist=127.0.0.1']], - binary=[self.options.testbinary]) - - def run_test(self): - test = TestManager() - # pass log handler through to the test manager object - test.log = self.log - test.add_new_connection( - NodeConn('127.0.0.1', p2p_port(0), self.nodes[0], test)) - NetworkThread().start() # Start up network handling in another thread - test.run() - - -if __name__ == '__main__': - MaxBlocksInFlightTest().main() diff --git a/test/functional/test_runner.py b/test/functional/test_runner.py index c36c0ba5a..9c2e75e22 100755 --- a/test/functional/test_runner.py +++ b/test/functional/test_runner.py @@ -1,433 +1,432 @@ #!/usr/bin/env python3 # Copyright (c) 2014-2016 The Bitcoin Core developers # Copyright (c) 2017 The Bitcoin developers # Distributed under the MIT software license, see the accompanying # file COPYING or http://www.opensource.org/licenses/mit-license.php. """Run regression test suite. This module calls down into individual test cases via subprocess. It will forward all unrecognized arguments onto the individual test scripts. Functional tests are disabled on Windows by default. Use --force to run them anyway. For a description of arguments recognized by test scripts, see `test/functional/test_framework/test_framework.py:BitcoinTestFramework.main`. """ import argparse import configparser import os import time import shutil import signal import sys import subprocess import tempfile import re BOLD = ("", "") RED = ("", "") GREEN = ("", "") if os.name == 'posix': # primitive formatting on supported # terminal via ANSI escape sequences: BOLD = ('\033[0m', '\033[1m') RED = ("\033[0m", "\033[31m") GREEN = ("\033[0m", "\033[32m") TEST_EXIT_PASSED = 0 TEST_EXIT_SKIPPED = 77 BASE_SCRIPTS = [ # Longest test should go first, to favor running tests in parallel 'wallet-hd.py', 'walletbackup.py', # vv Tests less than 5m vv 'p2p-fullblocktest.py', 'fundrawtransaction.py', 'p2p-compactblocks.py', # vv Tests less than 2m vv 'wallet.py', 'wallet-accounts.py', 'wallet-dump.py', 'listtransactions.py', # vv Tests less than 60s vv 'sendheaders.py', 'zapwallettxes.py', 'importmulti.py', 'mempool_limit.py', 'merkle_blocks.py', 'receivedby.py', 'abandonconflict.py', 'bip68-112-113-p2p.py', 'rawtransactions.py', 'reindex.py', # vv Tests less than 30s vv 'zmq_test.py', 'mempool_resurrect_test.py', 'txn_doublespend.py --mineblock', 'txn_clone.py', 'getchaintips.py', 'rest.py', 'mempool_spendcoinbase.py', 'mempool_reorg.py', 'mempool_persist.py', 'httpbasics.py', 'multi_rpc.py', 'proxy_test.py', 'signrawtransactions.py', 'disconnect_ban.py', 'decodescript.py', 'blockchain.py', 'disablewallet.py', 'net.py', 'keypool.py', 'p2p-mempool.py', 'prioritise_transaction.py', 'high_priority_transaction.py', 'invalidblockrequest.py', 'invalidtxrequest.py', 'p2p-versionbits-warning.py', 'preciousblock.py', 'importprunedfunds.py', 'signmessages.py', 'nulldummy.py', 'import-rescan.py', 'rpcnamedargs.py', 'listsinceblock.py', 'p2p-leaktests.py', 'abc-cmdline.py', 'abc-p2p-fullblocktest.py', 'abc-rpc.py', 'mempool-accept-txn.py', ] EXTENDED_SCRIPTS = [ # Longest test should go first, to favor running tests in parallel 'pruning.py', # vv Tests less than 20m vv 'smartfees.py', # vv Tests less than 5m vv 'maxuploadtarget.py', 'mempool_packages.py', # vv Tests less than 2m vv 'bip68-sequence.py', 'getblocktemplate_longpoll.py', 'p2p-timeouts.py', # vv Tests less than 60s vv 'bip9-softforks.py', 'p2p-feefilter.py', 'rpcbind_test.py', # vv Tests less than 30s vv 'assumevalid.py', 'bip65-cltv.py', 'bip65-cltv-p2p.py', 'bipdersig-p2p.py', 'bipdersig.py', 'getblocktemplate_proposals.py', 'txn_doublespend.py', 'txn_clone.py --mineblock', 'forknotify.py', 'invalidateblock.py', - 'maxblocksinflight.py', 'p2p-acceptblock.py', ] # Place EXTENDED_SCRIPTS first since it has the 3 longest running tests ALL_SCRIPTS = EXTENDED_SCRIPTS + BASE_SCRIPTS def main(): # Parse arguments and pass through unrecognised args parser = argparse.ArgumentParser(add_help=False, usage='%(prog)s [test_runner.py options] [script options] [scripts]', description=__doc__, epilog=''' Help text and arguments for individual test script:''', formatter_class=argparse.RawTextHelpFormatter) parser.add_argument('--coverage', action='store_true', help='generate a basic coverage report for the RPC interface') parser.add_argument( '--exclude', '-x', help='specify a comma-seperated-list of scripts to exclude. Do not include the .py extension in the name.') parser.add_argument('--extended', action='store_true', help='run the extended test suite in addition to the basic tests') parser.add_argument('--force', '-f', action='store_true', help='run tests even on platforms where they are disabled by default (e.g. windows).') parser.add_argument('--help', '-h', '-?', action='store_true', help='print help text and exit') parser.add_argument('--jobs', '-j', type=int, default=4, help='how many test scripts to run in parallel. Default=4.') args, unknown_args = parser.parse_known_args() # Create a set to store arguments and create the passon string tests = set(arg for arg in unknown_args if arg[:2] != "--") passon_args = [arg for arg in unknown_args if arg[:2] == "--"] # Read config generated by configure. config = configparser.ConfigParser() configfile = os.path.abspath(os.path.dirname(__file__)) + "/../config.ini" config.read_file(open(configfile)) passon_args.append("--configfile=%s" % configfile) enable_wallet = config["components"].getboolean("ENABLE_WALLET") enable_utils = config["components"].getboolean("ENABLE_UTILS") enable_bitcoind = config["components"].getboolean("ENABLE_BITCOIND") if config["environment"]["EXEEXT"] == ".exe" and not args.force: # https://github.com/bitcoin/bitcoin/commit/d52802551752140cf41f0d9a225a43e84404d3e9 # https://github.com/bitcoin/bitcoin/pull/5677#issuecomment-136646964 print( "Tests currently disabled on Windows by default. Use --force option to enable") sys.exit(0) if not (enable_wallet and enable_utils and enable_bitcoind): print( "No functional tests to run. Wallet, utils, and bitcoind must all be enabled") print( "Rerun `configure` with -enable-wallet, -with-utils and -with-daemon and rerun make") sys.exit(0) # Build list of tests if tests: # Individual tests have been specified. Run specified tests that exist # in the ALL_SCRIPTS list. Accept the name with or without .py # extension. test_list = [t for t in ALL_SCRIPTS if (t in tests or re.sub(".py$", "", t) in tests)] else: # No individual tests have been specified. # Run all base tests, and optionally run extended tests. test_list = BASE_SCRIPTS if args.extended: test_list += EXTENDED_SCRIPTS # TODO: BASE_SCRIPTS and EXTENDED_SCRIPTS are sorted by runtime # (for parallel running efficiency). This combined list will is no # longer sorted. # Remove the test cases that the user has explicitly asked to exclude. if args.exclude: for exclude_test in args.exclude.split(','): if exclude_test + ".py" in test_list: test_list.remove(exclude_test + ".py") if not test_list: print("No valid test scripts specified. Check that your test is in one " "of the test lists in test_runner.py, or run test_runner.py with no arguments to run all tests") sys.exit(0) if args.help: # Print help for test_runner.py, then print help of the first script # and exit. parser.print_help() subprocess.check_call( (config["environment"]["SRCDIR"] + '/test/functional/' + test_list[0]).split() + ['-h']) sys.exit(0) run_tests( test_list, config["environment"][ "SRCDIR"], config["environment"]["BUILDDIR"], config["environment"]["EXEEXT"], args.jobs, args.coverage, passon_args) def run_tests(test_list, src_dir, build_dir, exeext, jobs=1, enable_coverage=False, args=[]): # Set env vars if "BITCOIND" not in os.environ: os.environ["BITCOIND"] = build_dir + '/src/bitcoind' + exeext tests_dir = src_dir + '/test/functional/' flags = ["--srcdir={}".format(src_dir)] + args flags.append("--cachedir=%s/test/cache" % build_dir) if enable_coverage: coverage = RPCCoverage() flags.append(coverage.flag) print("Initializing coverage directory at {dir}\n".format( dir=coverage.dir)) else: coverage = None if len(test_list) > 1 and jobs > 1: # Populate cache subprocess.check_output([tests_dir + 'create_cache.py'] + flags) # Run Tests all_passed = True time_sum = 0 time0 = time.time() job_queue = TestHandler(jobs, tests_dir, test_list, flags) max_len_name = len(max(test_list, key=len)) results = BOLD[1] + "%s | %s | %s\n\n" % ( "TEST".ljust(max_len_name), "STATUS ", "DURATION") + BOLD[0] for _ in range(len(test_list)): (name, stdout, stderr, status, duration) = job_queue.get_next() all_passed = all_passed and status != "Failed" time_sum += duration print('\n' + BOLD[1] + name + BOLD[0] + ":") print('' if status == "Passed" else stdout + '\n', end='') print('' if stderr == '' else 'stderr:\n' + stderr + '\n', end='') print("Pass: {bold}{result}{unbold}, Duration: {duration}s\n".format( bold=BOLD[1], result=status, unbold=BOLD[0], duration=duration)) result = "{name} | {passed} | {duration}s\n".format(name=name.ljust( max_len_name), passed=str(status).ljust(6), duration=duration) if status == "Passed": results += GREEN[1] + result + GREEN[0] else: results += RED[1] + result + RED[0] results += BOLD[1] + "\n{name} | {passed} | {duration}s (accumulated)".format( name="ALL".ljust(max_len_name), passed=str(all_passed).ljust(6), duration=time_sum) + BOLD[0] print(results) print("\nRuntime: {} s".format(int(time.time() - time0))) if coverage: coverage.report_rpc_coverage() print("Cleaning up coverage data") coverage.cleanup() sys.exit(not all_passed) class TestHandler: """ Trigger the testscrips passed in via the list. """ def __init__(self, num_tests_parallel, tests_dir, test_list=None, flags=None): assert(num_tests_parallel >= 1) self.num_jobs = num_tests_parallel self.tests_dir = tests_dir self.test_list = test_list self.flags = flags self.num_running = 0 # In case there is a graveyard of zombie bitcoinds, we can apply a # pseudorandom offset to hopefully jump over them. # (625 is PORT_RANGE/MAX_NODES) self.portseed_offset = int(time.time() * 1000) % 625 self.jobs = [] def get_next(self): while self.num_running < self.num_jobs and self.test_list: # Add tests self.num_running += 1 t = self.test_list.pop(0) port_seed = ["--portseed={}".format( len(self.test_list) + self.portseed_offset)] log_stdout = tempfile.SpooledTemporaryFile(max_size=2**16) log_stderr = tempfile.SpooledTemporaryFile(max_size=2**16) self.jobs.append((t, time.time(), subprocess.Popen( (self.tests_dir + t).split() + self.flags + port_seed, universal_newlines=True, stdout=log_stdout, stderr=log_stderr), log_stdout, log_stderr)) if not self.jobs: raise IndexError('pop from empty list') while True: # Return first proc that finishes time.sleep(.5) for j in self.jobs: (name, time0, proc, log_out, log_err) = j if os.getenv('TRAVIS') == 'true' and int(time.time() - time0) > 20 * 60: # In travis, timeout individual tests after 20 minutes (to stop tests hanging and not # providing useful output. proc.send_signal(signal.SIGINT) if proc.poll() is not None: log_out.seek(0), log_err.seek(0) [stdout, stderr] = [l.read().decode('utf-8') for l in (log_out, log_err)] log_out.close(), log_err.close() if proc.returncode == TEST_EXIT_PASSED and stderr == "": status = "Passed" elif proc.returncode == TEST_EXIT_SKIPPED: status = "Skipped" else: status = "Failed" self.num_running -= 1 self.jobs.remove(j) return name, stdout, stderr, status, int( time.time() - time0) print('.', end='', flush=True) class RPCCoverage(object): """ Coverage reporting utilities for test_runner. Coverage calculation works by having each test script subprocess write coverage files into a particular directory. These files contain the RPC commands invoked during testing, as well as a complete listing of RPC commands per `bitcoin-cli help` (`rpc_interface.txt`). After all tests complete, the commands run are combined and diff'd against the complete list to calculate uncovered RPC commands. See also: test/functional/test_framework/coverage.py """ def __init__(self): self.dir = tempfile.mkdtemp(prefix="coverage") self.flag = '--coveragedir={}'.format(self.dir) def report_rpc_coverage(self): """ Print out RPC commands that were unexercised by tests. """ uncovered = self._get_uncovered_rpc_commands() if uncovered: print("Uncovered RPC commands:") print("".join((" - {}\n".format(i)) for i in sorted(uncovered))) else: print("All RPC commands covered.") def cleanup(self): return shutil.rmtree(self.dir) def _get_uncovered_rpc_commands(self): """ Return a set of currently untested RPC commands. """ # This is shared from `test/functional/test-framework/coverage.py` reference_filename = 'rpc_interface.txt' coverage_file_prefix = 'coverage.' coverage_ref_filename = os.path.join(self.dir, reference_filename) coverage_filenames = set() all_cmds = set() covered_cmds = set() if not os.path.isfile(coverage_ref_filename): raise RuntimeError("No coverage reference found") with open(coverage_ref_filename, 'r') as f: all_cmds.update([i.strip() for i in f.readlines()]) for root, dirs, files in os.walk(self.dir): for filename in files: if filename.startswith(coverage_file_prefix): coverage_filenames.add(os.path.join(root, filename)) for filename in coverage_filenames: with open(filename, 'r') as f: covered_cmds.update([i.strip() for i in f.readlines()]) return all_cmds - covered_cmds if __name__ == '__main__': main()