Page MenuHomePhabricator
Differential D15133

[Chronik] Plugins: Add a bare-bones Python `PluginContext`
ClosedPublic
Actions

Authored by tobias_ruck on Jan 10 2024, 23:42.

Details

Reviewers
Fabien
Group Reviewers
Restricted Project
Commits
rABCd53d6c757027: [Chronik] Plugins: Add a bare-bones Python `PluginContext`
Summary

This adds a some scaffolding for Python-based plugins using PyO3 for Chronik.

Currently, this only finds the Python version and logs it. The purpose is to setup the build and CI so we can optionally add plugins using the BUILD_BITCOIN_CHRONIK_PLUGINS flag.

The goal is to load Python scripts from a plugins folder in the datadir, which specify which LOKAD ID to index, and provide functions that will be called by Chronik on txs that match the LOKAD ID, which will then be indexed using GroupHistoryWriter and GroupUtxosWriter into either column families for that plugin, or into column families shared by all plugins (and prefixed appropriately).

Users can then call /plugin/<plugin name/id>/history/confirmed-txs/unconfirmed-txs/utoxs and get the txs of the protocol specified by the plugin.

One obvious application would be the SLPAgora or SLPDEX contracts, or something similar for ALP, allowing users to fetch all available trades for a given token ID or pubkey.

Test Plan
  1. cmake .. -GNinja -DBUILD_BITCOIN_CHRONIK=on -DBUILD_BITCOIN_CHRONIK_PLUGINS=on && ninja
  2. ./src/bitcoind -chronik -regtest => prints Plugin context initialized Python 3.9 (or whichever version)

Diff Detail

Repository
rABC Bitcoin ABC
Branch
chronik-init-python
Lint
Lint Passed
Unit
No Test Coverage
Build Status
Buildable 26252
Build 52076: Build Diffchronik-client-integration-tests · build-chronik
Build 52075: arc lint + arc unit

Event Timeline

tobias_ruck created this revision.Jan 10 2024, 23:42
Herald added a reviewer: Restricted Project. · View Herald TranscriptJan 10 2024, 23:42
tobias_ruck added a comment.Jan 10 2024, 23:55

@bot build-chronik-plugins

teamcity awarded a token.Jan 10 2024, 23:55
Build Bitcoin ABC Diffs / Diff Testing (build-chronik-plugins) failed.

Tail of the build log:

cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik-plugins/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/cargo --locked --features plugins clippy --package bitcoinsuite-* -- -D warnings
error: unexpected argument '--features' found

Usage: cargo [OPTIONS] [COMMAND]
       cargo [OPTIONS] -Zscript <MANIFEST> [ARGS]...

For more information, try '--help'.
[2/6] cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik-plugins/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/cargo --locked --features plugins clippy --package abc-rust-* -- -D warnings
FAILED: chronik/CMakeFiles/check-crate-abc-rust-clippy 
cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik-plugins/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/cargo --locked --features plugins clippy --package abc-rust-* -- -D warnings
error: unexpected argument '--features' found

Usage: cargo [OPTIONS] [COMMAND]
       cargo [OPTIONS] -Zscript <MANIFEST> [ARGS]...

For more information, try '--help'.
[3/6] cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik-plugins/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/cargo --locked --features plugins test --package bitcoinsuite-*
FAILED: chronik/CMakeFiles/check-crate-bitcoinsuite-test 
cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik-plugins/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/cargo --locked --features plugins test --package bitcoinsuite-*
error: unexpected argument '--features' found

  tip: 'test --features' exists

Usage: cargo [OPTIONS] [COMMAND]
       cargo [OPTIONS] -Zscript <MANIFEST> [ARGS]...

For more information, try '--help'.
[4/6] cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik-plugins/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/cargo --locked --features plugins test --package abc-rust-*
FAILED: chronik/CMakeFiles/check-crate-abc-rust-test 
cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik-plugins/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/cargo --locked --features plugins test --package abc-rust-*
error: unexpected argument '--features' found

  tip: 'test --features' exists

Usage: cargo [OPTIONS] [COMMAND]
       cargo [OPTIONS] -Zscript <MANIFEST> [ARGS]...

For more information, try '--help'.
[5/6] cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik-plugins/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/cargo --locked --features plugins test --package chronik-*
FAILED: chronik/CMakeFiles/check-crate-chronik-test 
cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik-plugins/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/cargo --locked --features plugins test --package chronik-*
error: unexpected argument '--features' found

  tip: 'test --features' exists

Usage: cargo [OPTIONS] [COMMAND]
       cargo [OPTIONS] -Zscript <MANIFEST> [ARGS]...

For more information, try '--help'.
[6/6] cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik-plugins/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/cargo --locked --features plugins clippy --package chronik-* -- -D warnings
FAILED: chronik/CMakeFiles/check-crate-chronik-clippy 
cd /work && /usr/bin/cmake -E env CARGO_TARGET_DIR="/work/abc-ci-builds/build-chronik-plugins/cargo/build" CARGO_BUILD_RUSTC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc" CARGO_BUILD_RUSTDOC="/root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustdoc" /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/cargo --locked --features plugins clippy --package chronik-* -- -D warnings
error: unexpected argument '--features' found

Usage: cargo [OPTIONS] [COMMAND]
       cargo [OPTIONS] -Zscript <MANIFEST> [ARGS]...

For more information, try '--help'.
ninja: build stopped: cannot make progress due to previous errors.
Build build-chronik-plugins failed with exit code 1
tobias_ruck updated this revision to Diff 44042.Jan 11 2024, 00:02

only provide features to rustc, not clippy etc.

tobias_ruck added a comment.Jan 11 2024, 00:02

@bot build-chronik-plugins

teamcity updated a token.Jan 11 2024, 00:02
Harbormaster completed remote builds in B26251: Diff 44041.Jan 11 2024, 00:05
Harbormaster completed remote builds in B26252: Diff 44042.Jan 11 2024, 00:19
tobias_ruck updated this revision to Diff 44043.Jan 11 2024, 00:20

activate lints for chronik-plugin

tobias_ruck added a comment.Jan 11 2024, 00:24

@bot build-linux64

teamcity updated a token.Jan 11 2024, 00:24
Build Bitcoin ABC Diffs / Diff Testing (build-linux64) failed.

Tail of the build log:

-- Looking for C++ include atomic
-- Looking for C++ include atomic - found
-- Performing Test ENABLE_HWCRC32
-- Performing Test ENABLE_HWCRC32 - Success
-- Performing Test have_C__pedantic
-- Performing Test have_C__pedantic - Success
-- Performing Test have_C__Wshadow
-- Performing Test have_C__Wshadow - Success
-- Performing Test have_C__Wundef
-- Performing Test have_C__Wundef - Success
-- Performing Test have_C__Wno_unused_function
-- Performing Test have_C__Wno_unused_function - Success
-- Performing Test have_C__Wno_overlength_strings
-- Performing Test have_C__Wno_overlength_strings - Success
-- Performing Test have_C__std_c89
-- Performing Test have_C__std_c89 - Success
-- Performing Test have_C__Wno_long_long
-- Performing Test have_C__Wno_long_long - Success
-- Performing Test have_C__Wno_duplicated_branches
-- Performing Test have_C__Wno_duplicated_branches - Success
-- Performing Test USE_ASM_X86_64
-- Performing Test USE_ASM_X86_64 - Success
-- Found Event component event: /work/depends/x86_64-linux-gnu/lib/libevent.a
-- Found Event: /work/depends/x86_64-linux-gnu/include (found suitable version "99.99.99", minimum required is "2.0.22") found components: event 
-- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.74.0/BoostConfig.cmake (found suitable version "1.74.0", minimum required is "1.64")  
-- Found Event component pthreads: /work/depends/x86_64-linux-gnu/lib/libevent_pthreads.a
-- Found Event: /work/depends/x86_64-linux-gnu/include (found suitable version "99.99.99", minimum required is "2.0.22") found components: pthreads 
-- Found MiniUPnPc component miniupnpc: /work/depends/x86_64-linux-gnu/lib/libminiupnpc.a
-- Found MiniUPnPc: /work/depends/x86_64-linux-gnu/include/miniupnpc (found suitable version "2.0.20180203", minimum required is "1.9")  
-- Found NATPMP component natpmp: /work/depends/x86_64-linux-gnu/lib/libnatpmp.a
-- Found NATPMP: /work/depends/x86_64-linux-gnu/include   
-- Found BerkeleyDB component CXX: /work/depends/x86_64-linux-gnu/lib/libdb_cxx-5.3.a
-- Found BerkeleyDB: /work/depends/x86_64-linux-gnu/include (found suitable version "5.3.28", minimum required is "5.3") found components: CXX 
-- Found SQLite3: /work/depends/x86_64-linux-gnu/include (found suitable version "3.32.1", minimum required is "3.7.17") 
-- Found ZeroMQ component zmq: /work/depends/x86_64-linux-gnu/lib/libzmq.a
-- Found ZeroMQ: /work/depends/x86_64-linux-gnu/include (found suitable version "4.3.1", minimum required is "4.1.5")  
-- Could NOT find Protobuf (missing: Protobuf_DIR)
-- Found Protobuf: /work/depends/x86_64-linux-gnu/lib/libprotobuf.a;-pthread (found version "3.21.12") 
-- Found OpenSSL: /work/depends/x86_64-linux-gnu/lib/libcrypto.a (found version "1.1.1t")  
-- Looking for EVP_MD_CTX_new
-- Looking for EVP_MD_CTX_new - found
-- Found QREncode component qrencode: /work/depends/x86_64-linux-gnu/lib/libqrencode.a
-- Found QREncode: /work/depends/x86_64-linux-gnu/include   
-- Rust Toolchain: 1.72.0-x86_64-unknown-linux-gnu
-- Found Rust: /root/.rustup/toolchains/1.72.0-x86_64-unknown-linux-gnu/bin/rustc (found version "1.72.0") 
-- Could NOT find Python (missing: Python_INCLUDE_DIRS Python_LIBRARIES Interpreter Development Development.Module Development.Embed) 
Adding Python_LIBRARIES: Python_LIBRARY-NOTFOUND
-- Performing Test have_linker__latomic
-- Performing Test have_linker__latomic - Success
-- Found Protobuf: /work/depends/x86_64-linux-gnu/lib/libprotobuf.a;-pthread (found suitable version "3.21.12", minimum required is "3.0.0") 
-- Configuring native build in /work/abc-ci-builds/build-linux64/native
-- Configuring done
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
Python_LIBRARY
    linked by target "chronik" in directory /work/chronik

-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.
Build build-linux64 failed with exit code 1
Harbormaster completed remote builds in B26253: Diff 44043.Jan 11 2024, 00:38
tobias_ruck updated this revision to Diff 44045.Jan 11 2024, 00:40

remove -DBUILD_BITCOIN_CHRONIK_PLUGINS for build-linux64 as it doesn't have Python

tobias_ruck added a comment.Jan 11 2024, 00:41

@bot build-chronik-plugins

teamcity updated a token.Jan 11 2024, 00:41
tobias_ruck published this revision for review.Jan 11 2024, 00:54
tobias_ruck edited the summary of this revision. (Show Details)
Harbormaster completed remote builds in B26255: Diff 44045.Jan 11 2024, 00:56
tobias_ruck edited the test plan for this revision. (Show Details)Jan 11 2024, 00:58
Fabien added a subscriber: Fabien.Jan 11 2024, 15:34
This comment was removed by Fabien.
Fabien added inline comments.Jan 11 2024, 15:43
chronik/chronik-plugin/src/context.rs
21 ↗(On Diff #44045)

Can't you load from a file instead ? This as several benefits like linting and syntax coloring.
Maybe it's simply too early at this stage.

contrib/teamcity/build-configurations.yml
217 ↗(On Diff #44045)

This doesn't do anything more than building right ? There is no real test for the feature ?

tobias_ruck added inline comments.Jan 11 2024, 15:53
chronik/chronik-plugin/src/context.rs
21 ↗(On Diff #44045)

Yes; but PyO3 doesn't provide that, we'd have to load via Rust with std::fs::File.

contrib/teamcity/build-configurations.yml
217 ↗(On Diff #44045)

I can add a functional test that looks for the Plugin context log, or add a /plugin/info endpoint or so; the biggest challenge though initially was building it

Fabien accepted this revision.Jan 11 2024, 16:36
Fabien added inline comments.
contrib/teamcity/build-configurations.yml
217 ↗(On Diff #44045)

it's fine I just wanted to clarify, no point testing a placeholder

This revision is now accepted and ready to land.Jan 11 2024, 16:36
Closed by commit rABCd53d6c757027: [Chronik] Plugins: Add a bare-bones Python `PluginContext` (authored by tobias_ruck). · Explain WhyJan 11 2024, 18:18
This revision was automatically updated to reflect the committed changes.
tobias_ruck added a commit: rABCd53d6c757027: [Chronik] Plugins: Add a bare-bones Python `PluginContext`.
Kronkmeister added a subscriber: Kronkmeister.Jan 12 2024, 23:11

Revision Contents
Changeset List

PathSize
1 line
chronik/
17 lines
chronik-lib/
7 lines
src/
4 lines
chronik-plugin/
15 lines
src/
27 lines
1 line
contrib/
teamcity/
14 lines

Diff 44042

View Options

Cargo.lock

Loading...
View Options

Cargo.toml

Loading...
View Options

chronik/CMakeLists.txt

Loading...
View Options

chronik/chronik-lib/Cargo.toml

Loading...
View Options

chronik/chronik-lib/src/bridge.rs

Loading...
View Options

chronik/chronik-plugin/Cargo.toml

Loading...
View Options

chronik/chronik-plugin/src/context.rs

Loading...
View Options

chronik/chronik-plugin/src/lib.rs

Loading...
View Options

contrib/teamcity/build-configurations.yml

Loading...