Page MenuHomePhabricator
Differential D9900 Diff 29518 doc/functional-tests.md

Changeset View

Standalone View

View Options

doc/functional-tests.md

Show First 20 Lines • Show All 60 Lines • ▼ Show 20 Lines
``` ```
test/functional/test_runner.py --extended test/functional/test_runner.py --extended
``` ```
By default, up to 4 tests will be run in parallel by test_runner. To specify By default, up to 4 tests will be run in parallel by test_runner. To specify
how many jobs to run, append `--jobs=n` how many jobs to run, append `--jobs=n`
The individual tests and the test_runner harness have many command-line The individual tests and the test_runner harness have many command-line
options. Run `test_runner.py -h` to see them all. options. Run `test/functional/test_runner.py -h` to see them all.
#### Troubleshooting and debugging test failures #### Troubleshooting and debugging test failures
##### Resource contention ##### Resource contention
The P2P and RPC ports used by the bitcoind nodes-under-test are chosen to make The P2P and RPC ports used by the bitcoind nodes-under-test are chosen to make
conflicts with other processes unlikely. However, if there is another bitcoind conflicts with other processes unlikely. However, if there is another bitcoind
process running on the system (perhaps from a previous test which hasn't successfully process running on the system (perhaps from a previous test which hasn't successfully
killed all its bitcoind nodes), then there may be a port conflict which will killed all its bitcoind nodes), then there may be a port conflict which will
cause the test to fail. It is recommended that you run the tests on a system cause the test to fail. It is recommended that you run the tests on a system
where no other bitcoind processes are running. where no other bitcoind processes are running.
On linux, the test_framework will warn if there is another On linux, the test framework will warn if there is another
bitcoind process running when the tests are started. bitcoind process running when the tests are started.
If there are zombie bitcoind processes after test failure, you can kill them If there are zombie bitcoind processes after test failure, you can kill them
by running the following commands. **Note that these commands will kill all by running the following commands. **Note that these commands will kill all
bitcoind processes running on the system, so should not be used if any non-test bitcoind processes running on the system, so should not be used if any non-test
bitcoind processes are being run.** bitcoind processes are being run.**
```bash ```bash
Show All 11 Lines
A pre-mined blockchain with 200 blocks is generated the first time a A pre-mined blockchain with 200 blocks is generated the first time a
functional test is run and is stored in test/cache. This speeds up functional test is run and is stored in test/cache. This speeds up
test startup times since new blockchains don't need to be generated for test startup times since new blockchains don't need to be generated for
each test. However, the cache may get into a bad state, in which case each test. However, the cache may get into a bad state, in which case
tests will fail. If this happens, remove the cache directory (and make tests will fail. If this happens, remove the cache directory (and make
sure bitcoind processes are stopped as above): sure bitcoind processes are stopped as above):
```bash ```bash
rm -rf cache rm -rf test/cache
killall bitcoind killall bitcoind
``` ```
##### Test logging ##### Test logging
The tests contain logging at different levels (debug, info, warning, etc). By The tests contain logging at different levels (debug, info, warning, etc). By
default: default:
- When run through the test_runner harness, *all* logs are written to - When run through the test_runner harness, *all* logs are written to
`test_framework.log` and no logs are output to the console. `test_framework.log` and no logs are output to the console.
- When run directly, *all* logs are written to `test_framework.log` and INFO - When run directly, *all* logs are written to `test_framework.log` and INFO
level and above are output to the console. level and above are output to the console.
- When run on Travis, no logs are output to the console. However, if a test - When run on Travis, no logs are output to the console. However, if a test
fails, the `test_framework.log` and bitcoind `debug.log`s will all be dumped fails, the `test_framework.log` and bitcoind `debug.log`s will all be dumped
to the console to help troubleshooting. to the console to help troubleshooting.
These log files can be located under the test data directory (which is always
printed in the first line of test output):
- `<test data directory>/test_framework.log`
- `<test data directory>/node<node number>/regtest/debug.log`.
The node number identifies the relevant test node, starting from `node0`, which
corresponds to its position in the nodes list of the specific test,
e.g. `self.nodes[0]`.
To change the level of logs output to the console, use the `-l` command line To change the level of logs output to the console, use the `-l` command line
argument. argument.
`test_framework.log` and bitcoind `debug.log`s can be combined into a single `test_framework.log` and bitcoind `debug.log`s can be combined into a single
aggregate log by running the `combine_logs.py` script. The output can be plain aggregate log by running the `combine_logs.py` script. The output can be plain
text, colorized text or html. For example: text, colorized text or html. For example:
``` ```
combine_logs.py -c <test data directory> | less -r test/functional/combine_logs.py -c <test data directory> | less -r
``` ```
will pipe the colorized logs from the test into less. will pipe the colorized logs from the test into less.
Use `--tracerpc` to trace out all the RPC calls and responses to the console. Use `--tracerpc` to trace out all the RPC calls and responses to the console.
For some tests (eg any that use `submitblock` to submit a full block over RPC), For some tests (eg any that use `submitblock` to submit a full block over RPC),
this can result in a lot of screen output. this can result in a lot of screen output.
▲ Show 20 Lines • Show All 102 Lines • ▼ Show 20 Lines
Util tests can be run locally by running `test/util/bitcoin-util-test.py`. Util tests can be run locally by running `test/util/bitcoin-util-test.py`.
Use the `-v` option for verbose output. Use the `-v` option for verbose output.
# Writing functional tests # Writing functional tests
#### Example test #### Example test
The [example_test.py](/test/functional/example_test.py) is a heavily commented The file [test/functional/example_test.py](/test/functional/example_test.py) is a heavily commented
example of a test case that uses both the RPC and P2P interfaces. If you are example of a test case that uses both the RPC and P2P interfaces. If you are
writing your first test, copy that file and modify to fit your needs. writing your first test, copy that file and modify to fit your needs.
#### Coverage #### Coverage
Running `test_runner.py` with the `--coverage` argument tracks which RPCs are Running `test/functional/test_runner.py` with the `--coverage` argument tracks which RPCs are
called by the tests and prints a report of uncovered RPCs in the summary. This called by the tests and prints a report of uncovered RPCs in the summary. This
can be used (along with the `--extended` argument) to find out which RPCs we can be used (along with the `--extended` argument) to find out which RPCs we
don't have test cases for. don't have test cases for.
#### Style guidelines #### Style guidelines
- Where possible, try to adhere to - Where possible, try to adhere to
[PEP-8 guidelines](https://www.python.org/dev/peps/pep-0008/) [PEP-8 guidelines](https://www.python.org/dev/peps/pep-0008/)
▲ Show 20 Lines • Show All 51 Lines • ▼ Show 20 Lines
P2P messages. These can be found in the following source files: P2P messages. These can be found in the following source files:
- `/src/rpc/*` for RPCs - `/src/rpc/*` for RPCs
- `/src/wallet/rpc*` for wallet RPCs - `/src/wallet/rpc*` for wallet RPCs
- `ProcessMessage()` in `/src/net_processing.cpp` for parsing P2P messages - `ProcessMessage()` in `/src/net_processing.cpp` for parsing P2P messages
#### Using the P2P interface #### Using the P2P interface
- `messages.py` contains all the definitions for objects that pass - [messages.py](/test/functional/test_framework/messages.py) contains all the
over the network (`CBlock`, `CTransaction`, etc, along with the network-level definitions for objects that pass over the network (`CBlock`, `CTransaction`,
wrappers for them, `msg_block`, `msg_tx`, etc). etc, along with the network-level wrappers for them, `msg_block`, `msg_tx`, etc).
- P2P tests have two threads. One thread handles all network communication - P2P tests have two threads. One thread handles all network communication
with the bitcoind(s) being tested in a callback-based event loop; the other with the bitcoind(s) being tested in a callback-based event loop; the other
implements the test logic. implements the test logic.
- `P2PConnection` is the class used to connect to a bitcoind. `P2PInterface` - `P2PConnection` is the class used to connect to a bitcoind. `P2PInterface`
contains the higher level logic for processing P2P payloads and connecting to contains the higher level logic for processing P2P payloads and connecting to
the Bitcoin Core node application logic. For custom behaviour, subclass the the Bitcoin Core node application logic. For custom behaviour, subclass the
P2PInterface object and override the callback methods. P2PInterface object and override the callback methods.
- Can be used to write tests where specific P2P protocol behavior is tested. - Can be used to write tests where specific P2P protocol behavior is tested.
Examples tests are `p2p-acceptblock.py`, `p2p-compactblocks.py`. Examples tests are [p2p_unrequested_blocks.py](/test/functional/p2p_unrequested_blocks.py),
[p2p_compactblocks.py](/test/functional/p2p_compactblocks.py).
#### Prototyping tests #### Prototyping tests
The [`TestShell`](test-shell.md) class exposes the BitcoinTestFramework The [`TestShell`](test-shell.md) class exposes the BitcoinTestFramework
functionality to interactive Python3 environments and can be used to prototype functionality to interactive Python3 environments and can be used to prototype
tests. This may be especially useful in a REPL environment with session logging tests. This may be especially useful in a REPL environment with session logging
utilities, such as utilities, such as
[IPython](https://ipython.readthedocs.io/en/stable/interactive/reference.html#session-logging-and-restoring). [IPython](https://ipython.readthedocs.io/en/stable/interactive/reference.html#session-logging-and-restoring).
The logs of such interactive sessions can later be adapted into permanent test The logs of such interactive sessions can later be adapted into permanent test
cases. cases.
### test-framework modules ### Test framework modules
The following are useful modules for test developers. They are located in
[test/functional/test_framework/](/test/functional/test_framework).
#### [test_framework/authproxy.py](/test/functional/test_framework/authproxy.py) #### [authproxy.py](/test/functional/test_framework/authproxy.py)
Taken from the [python-bitcoinrpc repository](https://github.com/jgarzik/python-bitcoinrpc). Taken from the [python-bitcoinrpc repository](https://github.com/jgarzik/python-bitcoinrpc).
#### [test_framework/test_framework.py](/test/functional/test_framework/test_framework.py) #### [test_framework.py](/test/functional/test_framework/test_framework.py)
Base class for functional tests. Base class for functional tests.
#### [test_framework/util.py](/test/functional/test_framework/util.py) #### [util.py](/test/functional/test_framework/util.py)
Generally useful functions. Generally useful functions.
#### [test_framework/p2p.py](/test/functional/test_framework/p2p.py) #### [p2p.py](/test/functional/test_framework/p2p.py)
Test objects for interacting with a bitcoind node over the p2p interface. Test objects for interacting with a bitcoind node over the p2p interface.
#### [test_framework/script.py](/test/functional/test_framework/script.py) #### [script.py](/test/functional/test_framework/script.py)
Utilities for manipulating transaction scripts (originally from python-bitcoinlib) Utilities for manipulating transaction scripts (originally from python-bitcoinlib)
#### [test_framework/key.py](/test/functional/test_framework/key.py) #### [key.py](/test/functional/test_framework/key.py)
Test-only secp256k1 elliptic curve implementation Test-only secp256k1 elliptic curve implementation
#### [test_framework/blocktools.py](/test/functional/test_framework/blocktools.py) #### [blocktools.py](/test/functional/test_framework/blocktools.py)
Helper functions for creating blocks and transactions. Helper functions for creating blocks and transactions.