Page MenuHomePhabricator

[Chronik] Plugins: Add a bare-bones Python `PluginContext`
ClosedPublic

Authored by tobias_ruck on Jan 10 2024, 23:42.

Details

Reviewers
Fabien
Group Reviewers
Restricted Project
Commits
rABCd53d6c757027: [Chronik] Plugins: Add a bare-bones Python `PluginContext`
Summary

This adds a some scaffolding for Python-based plugins using PyO3 for Chronik.

Currently, this only finds the Python version and logs it. The purpose is to setup the build and CI so we can optionally add plugins using the BUILD_BITCOIN_CHRONIK_PLUGINS flag.

The goal is to load Python scripts from a plugins folder in the datadir, which specify which LOKAD ID to index, and provide functions that will be called by Chronik on txs that match the LOKAD ID, which will then be indexed using GroupHistoryWriter and GroupUtxosWriter into either column families for that plugin, or into column families shared by all plugins (and prefixed appropriately).

Users can then call /plugin/<plugin name/id>/history/confirmed-txs/unconfirmed-txs/utoxs and get the txs of the protocol specified by the plugin.

One obvious application would be the SLPAgora or SLPDEX contracts, or something similar for ALP, allowing users to fetch all available trades for a given token ID or pubkey.

Test Plan
  1. cmake .. -GNinja -DBUILD_BITCOIN_CHRONIK=on -DBUILD_BITCOIN_CHRONIK_PLUGINS=on && ninja
  2. ./src/bitcoind -chronik -regtest => prints Plugin context initialized Python 3.9 (or whichever version)

Diff Detail

Repository
rABC Bitcoin ABC
Branch
chronik-init-python
Lint
Lint Passed
Unit
No Test Coverage
Build Status
Buildable 26252
Build 52076: Build Diffchronik-client-integration-tests · build-chronik
Build 52075: arc lint + arc unit

Event Timeline

Tail of the build log:

cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik-plugins/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/cargo --locked --features plugins clippy --package bitcoinsuite-* -- -D warnings
error: unexpected argument '--features' found

Usage: cargo [OPTIONS] [COMMAND]
       cargo [OPTIONS] -Zscript <MANIFEST> [ARGS]...

For more information, try '--help'.
[2/6] cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik-plugins/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/cargo --locked --features plugins clippy --package abc-rust-* -- -D warnings
FAILED: chronik/CMakeFiles/check-crate-abc-rust-clippy 
cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik-plugins/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/cargo --locked --features plugins clippy --package abc-rust-* -- -D warnings
error: unexpected argument '--features' found

Usage: cargo [OPTIONS] [COMMAND]
       cargo [OPTIONS] -Zscript <MANIFEST> [ARGS]...

For more information, try '--help'.
[3/6] cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik-plugins/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/cargo --locked --features plugins test --package bitcoinsuite-*
FAILED: chronik/CMakeFiles/check-crate-bitcoinsuite-test 
cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik-plugins/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/cargo --locked --features plugins test --package bitcoinsuite-*
error: unexpected argument '--features' found

  tip: 'test --features' exists

Usage: cargo [OPTIONS] [COMMAND]
       cargo [OPTIONS] -Zscript <MANIFEST> [ARGS]...

For more information, try '--help'.
[4/6] cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik-plugins/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/cargo --locked --features plugins test --package abc-rust-*
FAILED: chronik/CMakeFiles/check-crate-abc-rust-test 
cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik-plugins/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/cargo --locked --features plugins test --package abc-rust-*
error: unexpected argument '--features' found

  tip: 'test --features' exists

Usage: cargo [OPTIONS] [COMMAND]
       cargo [OPTIONS] -Zscript <MANIFEST> [ARGS]...

For more information, try '--help'.
[5/6] cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik-plugins/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/cargo --locked --features plugins test --package chronik-*
FAILED: chronik/CMakeFiles/check-crate-chronik-test 
cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik-plugins/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/cargo --locked --features plugins test --package chronik-*
error: unexpected argument '--features' found

  tip: 'test --features' exists

Usage: cargo [OPTIONS] [COMMAND]
       cargo [OPTIONS] -Zscript <MANIFEST> [ARGS]...

For more information, try '--help'.
[6/6] cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik-plugins/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/cargo --locked --features plugins clippy --package chronik-* -- -D warnings
FAILED: chronik/CMakeFiles/check-crate-chronik-clippy 
cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik-plugins/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/cargo --locked --features plugins clippy --package chronik-* -- -D warnings
error: unexpected argument '--features' found

Usage: cargo [OPTIONS] [COMMAND]
       cargo [OPTIONS] -Zscript <MANIFEST> [ARGS]...

For more information, try '--help'.
ninja: build stopped: cannot make progress due to previous errors.
Build build-chronik-plugins failed with exit code 1

only provide features to rustc, not clippy etc.

activate lints for chronik-plugin

Tail of the build log:

-- Looking for C++ include atomic
-- Looking for C++ include atomic - found
-- Performing Test ENABLE_HWCRC32
-- Performing Test ENABLE_HWCRC32 - Success
-- Performing Test have_C__pedantic
-- Performing Test have_C__pedantic - Success
-- Performing Test have_C__Wshadow
-- Performing Test have_C__Wshadow - Success
-- Performing Test have_C__Wundef
-- Performing Test have_C__Wundef - Success
-- Performing Test have_C__Wno_unused_function
-- Performing Test have_C__Wno_unused_function - Success
-- Performing Test have_C__Wno_overlength_strings
-- Performing Test have_C__Wno_overlength_strings - Success
-- Performing Test have_C__std_c89
-- Performing Test have_C__std_c89 - Success
-- Performing Test have_C__Wno_long_long
-- Performing Test have_C__Wno_long_long - Success
-- Performing Test have_C__Wno_duplicated_branches
-- Performing Test have_C__Wno_duplicated_branches - Success
-- Performing Test USE_ASM_X86_64
-- Performing Test USE_ASM_X86_64 - Success
-- Found Event component event: /work/depends/x86_64-linux-gnu/lib/libevent.a
-- Found Event: /work/depends/x86_64-linux-gnu/include (found suitable version "99.99.99", minimum required is "2.0.22") found components: event 
-- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.74.0/BoostConfig.cmake (found suitable version "1.74.0", minimum required is "1.64")  
-- Found Event component pthreads: /work/depends/x86_64-linux-gnu/lib/libevent_pthreads.a
-- Found Event: /work/depends/x86_64-linux-gnu/include (found suitable version "99.99.99", minimum required is "2.0.22") found components: pthreads 
-- Found MiniUPnPc component miniupnpc: /work/depends/x86_64-linux-gnu/lib/libminiupnpc.a
-- Found MiniUPnPc: /work/depends/x86_64-linux-gnu/include/miniupnpc (found suitable version "2.0.20180203", minimum required is "1.9")  
-- Found NATPMP component natpmp: /work/depends/x86_64-linux-gnu/lib/libnatpmp.a
-- Found NATPMP: /work/depends/x86_64-linux-gnu/include   
-- Found BerkeleyDB component CXX: /work/depends/x86_64-linux-gnu/lib/libdb_cxx-5.3.a
-- Found BerkeleyDB: /work/depends/x86_64-linux-gnu/include (found suitable version "5.3.28", minimum required is "5.3") found components: CXX 
-- Found SQLite3: /work/depends/x86_64-linux-gnu/include (found suitable version "3.32.1", minimum required is "3.7.17") 
-- Found ZeroMQ component zmq: /work/depends/x86_64-linux-gnu/lib/libzmq.a
-- Found ZeroMQ: /work/depends/x86_64-linux-gnu/include (found suitable version "4.3.1", minimum required is "4.1.5")  
-- Could NOT find Protobuf (missing: Protobuf_DIR)
-- Found Protobuf: /work/depends/x86_64-linux-gnu/lib/libprotobuf.a;-pthread (found version "3.21.12") 
-- Found OpenSSL: /work/depends/x86_64-linux-gnu/lib/libcrypto.a (found version "1.1.1t")  
-- Looking for EVP_MD_CTX_new
-- Looking for EVP_MD_CTX_new - found
-- Found QREncode component qrencode: /work/depends/x86_64-linux-gnu/lib/libqrencode.a
-- Found QREncode: /work/depends/x86_64-linux-gnu/include   
-- Rust Toolchain: 1.72.0-x86_64-unknown-linux-gnu
-- Found Rust: /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc (found version "1.72.0") 
-- Could NOT find Python (missing: Python_INCLUDE_DIRS Python_LIBRARIES Interpreter Development Development.Module Development.Embed) 
Adding Python_LIBRARIES: Python_LIBRARY-NOTFOUND
-- Performing Test have_linker__latomic
-- Performing Test have_linker__latomic - Success
-- Found Protobuf: /work/depends/x86_64-linux-gnu/lib/libprotobuf.a;-pthread (found suitable version "3.21.12", minimum required is "3.0.0") 
-- Configuring native build in /work/abc-ci-builds/build-linux64/native
-- Configuring done
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
Python_LIBRARY
    linked by target "chronik" in directory /work/chronik

-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.
Build build-linux64 failed with exit code 1

remove -DBUILD_BITCOIN_CHRONIK_PLUGINS for build-linux64 as it doesn't have Python

tobias_ruck edited the summary of this revision. (Show Details)
This comment was removed by Fabien.
chronik/chronik-plugin/src/context.rs
21 ↗(On Diff #44045)

Can't you load from a file instead ? This as several benefits like linting and syntax coloring.
Maybe it's simply too early at this stage.

contrib/teamcity/build-configurations.yml
217 ↗(On Diff #44045)

This doesn't do anything more than building right ? There is no real test for the feature ?

chronik/chronik-plugin/src/context.rs
21 ↗(On Diff #44045)

Yes; but PyO3 doesn't provide that, we'd have to load via Rust with std::fs::File.

contrib/teamcity/build-configurations.yml
217 ↗(On Diff #44045)

I can add a functional test that looks for the Plugin context log, or add a /plugin/info endpoint or so; the biggest challenge though initially was building it

Fabien added inline comments.
contrib/teamcity/build-configurations.yml
217 ↗(On Diff #44045)

it's fine I just wanted to clarify, no point testing a placeholder

This revision is now accepted and ready to land.Jan 11 2024, 16:36