Page MenuHomePhabricator

interface_rpc: improve reliability of test_work_queue_exceeded with multiprocessing
ClosedPublic

Authored by PiRK on Mar 1 2023, 16:45.

Details

Summary

Threading in python does not work very well, everything runs on a single thread most of the time because of the global interpreter lock]. Using multiple threads only makes sense when doing I/O operations or sleeping, in this case Python can emulate a multi-threading behavior by switching threads when the current one becomes idle.

This causes much variability in the interface_rpc test, after D13182. The test duration varies most of the time between 1 and 30 seconds, but a duration of 282 seconds was recorded at least by one user. The test struggles to spam the work queue faster than the RPC calls are handled by bitcoind.

Fix this by using multiprocessing instead of threading. With this, I consistently get a test duration of 1 or 2 seconds.

The trade-off is that the code is slightly more complicated because processes need to use queue objects to commnunicate, as they don't share their variables.

A minor behavior improvement is that now all errror messages are checked if multiple processes manage to trigger the error at roughly the same time.

Test Plan

seq 10 | while read i;do test/functional/test_runner.py interface_rpc;done

Diff Detail

Event Timeline

PiRK requested review of this revision.Mar 1 2023, 16:45
PiRK edited the summary of this revision. (Show Details)

minor improvement: less variables

behavior improvement: check all error messages (in case multiple processes trigger the error)

rebase (I accidentaly put this on top on an unrelated flynt branch)

This revision is now accepted and ready to land.Mar 1 2023, 18:06