Change Details

The purpose of this endpoint is to allow users an easy way to explore the groups indexed by a plugin. Under normal usage, this API will be extremely fast, as it just slices exactly the requested info from the DB. This is because the way users query data is very close to how it's layed out in the DB. There's one edge case where adjacent groups with lots of UTXOs that are all spent in the mempool could require a bigger DB scan; we prevent this by setting an upper limit on the number of skipped groups, and use a heuristic where groups with more than 500 UTXOs in the DB are assumed to have some unspent UTXOs (even if all are spent in the mempool). Worst case, an attacker could create a setup that makes us waste around 500k UTXO reads per request (~20MB), which will be cached and adjacent in storage anyway, which worse case should be around 100ms without cache. This will make the API return incorrect results (i.e. too many groups) in certain edge cases, but this is entirely acceptable: 1. There's no guarantee that the returned groups will still be correct after the request returned anyway, as someone could've made transactions spending the UTXOs of the group, so users of the API will have to expect this behavior anyway for different reasons. 2. The inconsistency will be resolved once the txs are mined. 3. The API will never incorrectly leave out groups, only incorrectly keep them. 4. The danger of having an API spin for a long time is much bigger, as it could "stun lock" Chronik by keeping read access on the RwLock for a long time. The `prefix` param is intended to make plugins prefix different kinds of groups with different prefixes (e.g. "T" for token IDs, and "P" for public keys, etc.), so this allows a simple selection of "sub-groups" of a plugin. The `start` params is intended to allow a form of pagination; users can put the last group of the last query here to get the next page. For convenience, the API returns a `next_start` value which, if provided as `start`, will guarantee to make progress on the next call. If empty, it indicates if there's no more values. Note that this diff changes how PluginMember is serialized (and therefore is a DB schema update), but since currently nobody is using the system yet, this is fine. The change is needed as postcard prefixes the length of the group, which makes it impossible to use for a prefix search. An example how this endpoint would be useful is for the Agora plugin: 1. It allows the user to get every token ID that has an offer in a paginated way, so they can have an overview what tokens even exists. 2. Then, when the user clicks on a token ID, the app queries all the offered UTXOs for this token ID and can display them. Depends on D16551.

The purpose of this endpoint is to allow users an easy way to explore the groups indexed by a plugin. Under normal usage, this API will be extremely fast, as it just slices exactly the requested info from the DB. This is because the way users query data is very close to how it's layed out in the DB. There's one edge case where adjacent groups with lots of UTXOs that are all spent in the mempool could require a bigger DB scan; we prevent this by setting an upper limit on the number of skipped groups, and use a heuristic where groups with more than 500 UTXOs in the DB are assumed to have some unspent UTXOs (even if all are spent in the mempool). Worst case, an attacker could create a setup that makes us waste around 500k UTXO reads per request (~20MB), which will be cached and adjacent in storage anyway, which worse case should be around 100ms without cache. This will make the API return incorrect results (i.e. too many groups) in certain edge cases, but this is entirely acceptable: 1. There's no guarantee that the returned groups will still be correct after the request returned anyway, as someone could've made transactions spending the UTXOs of the group, so users of the API will have to expect this behavior anyway for different reasons. 2. The inconsistency will be resolved once the txs are mined. 3. The API will never incorrectly leave out groups, only incorrectly keep them. 4. The danger of having an API spin for a long time is much bigger, as it could "stun lock" Chronik by keeping read access on the RwLock for a long time. The `prefix` param is intended to make plugins prefix different kinds of groups with different prefixes (e.g. "T" for token IDs, and "P" for public keys, etc.), so this allows a simple selection of "sub-groups" of a plugin. The `start` params is intended to allow a form of pagination; users can put the last group of the last query here to get the next page. For convenience, the API returns a `next_start` value which, if provided as `start`, will guarantee to make progress on the next call. If empty, it indicates if there's no more values. Note that this diff changes how PluginMember is serialized (and therefore is a DB schema update), but since currently nobody is using the system yet, this is fine. The change is needed as postcard prefixes the length of the group, which makes it impossible to use for a prefix search. An example how this endpoint would be useful is for the Agora plugin: 1. It allows the user to get every token ID that has an offer in a paginated way, so they can have an overview what tokens even exists. 2. Then, when the user clicks on a token ID, the app queries all the offered UTXOs for this token ID and can display them. This is especially interesting for NFTs, as one sub-group could have all the NFT collections ("NFT Groups"), another sub-group could have all the NFTs of a specific collection, etc., allowing for a drill-down without any indexing outside of the Chronik plugin. This would make integration of Agora into other wallets very convenient. Depends on D16551.

The purpose of this endpoint is to allow users an easy way to explore the groups indexed by a plugin. Under normal usage, this API will be extremely fast, as it just slices exactly the requested info from the DB. This is because the way users query data is very close to how it's layed out in the DB. There's one edge case where adjacent groups with lots of UTXOs that are all spent in the mempool could require a bigger DB scan; we prevent this by setting an upper limit on the number of skipped groups, and use a heuristic where groups with more than 500 UTXOs in the DB are assumed to have some unspent UTXOs (even if all are spent in the mempool). Worst case, an attacker could create a setup that makes us waste around 500k UTXO reads per request (~20MB), which will be cached and adjacent in storage anyway, which worse case should be around 100ms without cache. This will make the API return incorrect results (i.e. too many groups) in certain edge cases, but this is entirely acceptable: 1. There's no guarantee that the returned groups will still be correct after the request returned anyway, as someone could've made transactions spending the UTXOs of the group, so users of the API will have to expect this behavior anyway for different reasons. 2. The inconsistency will be resolved once the txs are mined. 3. The API will never incorrectly leave out groups, only incorrectly keep them. 4. The danger of having an API spin for a long time is much bigger, as it could "stun lock" Chronik by keeping read access on the RwLock for a long time. The `prefix` param is intended to make plugins prefix different kinds of groups with different prefixes (e.g. "T" for token IDs, and "P" for public keys, etc.), so this allows a simple selection of "sub-groups" of a plugin. The `start` params is intended to allow a form of pagination; users can put the last group of the last query here to get the next page. For convenience, the API returns a `next_start` value which, if provided as `start`, will guarantee to make progress on the next call. If empty, it indicates if there's no more values. Note that this diff changes how PluginMember is serialized (and therefore is a DB schema update), but since currently nobody is using the system yet, this is fine. The change is needed as postcard prefixes the length of the group, which makes it impossible to use for a prefix search. An example how this endpoint would be useful is for the Agora plugin: 1. It allows the user to get every token ID that has an offer in a paginated way, so they can have an overview what tokens even exists. 2. Then, when the user clicks on a token ID, the app queries all the offered UTXOs for this token ID and can display them. This is especially interesting for NFTs, as one sub-group could have all the NFT collections ("NFT Groups"), another sub-group could have all the NFTs of a specific collection, etc., allowing for a drill-down without any indexing outside of the Chronik plugin. This would make integration of Agora into other wallets very convenient. Depends on D16551.