Page MenuHomePhabricator

[Chronik-client] Implement new UTXO APIs
AbandonedPublic

Authored by emack on Jul 12 2023, 00:16.

Details

Reviewers
bytesofman
tobias_ruck
Fabien
Group Reviewers
Restricted Project
Summary

As observed in recent attempts to develop a simple tx building demo for new devs looking to build on eCash, there are a number of chronik specific nuances that the app dev needs to handle in their app.

One such nuance is where chronik-client's .script().utxos() api returns a mixture of both XEC and SLP UTXOs, which meant the app developer would have to explicitly filter in their app.

This diff:
- Keeps the existing .script().utxos() function to maintain backwards compatibility

  • Refactors the existing .script().utxos() function to return all UTXOs however neatly separated into two separate XEC and SLP UTXO arrays. eToken compatible wallets can use this to refresh its XEC and SLP utxo sets without additional parsing logic.
  • Adds new .script().xecUtxos() and .script().slpUtxos() functions to only return XEC or SLP UTXOs. Wallets which don't support eTokens can use this to easily retrieve UTXOs to spend without worrying about accidentally burning SLP UTXOs.

Note: The XEC/SLP UTXO separation logic is essentially a filter applied on the utxos map array based on whether the slpToken param exists or not. This slpToken param determines whether the utxo is a token utxo or not, however worth noting that this only applies to the Chronik indexer, hence why it's worth abstracting this indexer specific knowledge from app devs.

Test Plan

npm test

Diff Detail

Repository
rABC Bitcoin ABC
Branch
utxoSplit
Lint
Lint Passed
Unit
No Test Coverage
Build Status
Buildable 24494
Build 48588: Build Diff
Build 48587: arc lint + arc unit

Event Timeline

emack requested review of this revision.Jul 12 2023, 00:16
Fabien requested changes to this revision.Jul 12 2023, 05:40

This is duplicating each utxo, which can be very costly (imagine your RAM dying with the IFP address !)

This revision now requires changes to proceed.Jul 12 2023, 05:40

Removed the duplicated utxo set in the existing utxos array.

Code looks good, but I'm not sure about the API. What is the chance an app has the need for both native and slp utxos ? If only one is needed then only returning the desired subset could lead to a much lighter returned object and thus better performance. What do you think ?

  • Can we keep the original functionality of the script.utxos() call so that this is not a breaking change?
  • Possible to add new method to get the utxos split out, as diff currently does?
  • Possible to add new method to get either one or the other, as Fabien suggests?
  • Need to bump the version in package.json and update the README
This revision now requires changes to proceed.Jul 12 2023, 18:39
  • Reverted the existing .script().utxos() function to maintain backwards compatibility
  • Added a new .script().utxosByType(type) function to only return XEC or SLP UTXOs based on supplied type. Wallets which don't support eTokens can use this to easily retrieve UTXOs to spend without worrying about accidentally burning SLP UTXOs.
  • Added a new .script().utxosSeparated() function to return all UTXOs however neatly separated into two separate XEC and SLP UTXO arrays. Cashtab and other eToken compatible wallets can use this to refresh its XEC and SLP utxo sets without additional parsing logic.
  • updated README and bumped package.json
emack retitled this revision from [Chronik-client] Update the .script().utxos() API to return separated UTXOs to [Chronik-client] Implement new UTXO APIs.Jul 13 2023, 05:17
Fabien requested changes to this revision.Jul 13 2023, 06:54

You're not taking advantage of your own API: calling utxos() from the filtering functions let you deduplicate the chronik call.

modules/chronik-client/index.ts
246 ↗(On Diff #41433)

The only thing that these 2 branches have in common is the call to chronik, and lowering the case for the selection input string. This is equivalent to using a bool to select how a function works, which is generally a bad practice. In this case it's even worth since you can pass 'foobar' in the type and get something but you don't know what without looking at the implementation detail.

There is a simple fix: create 2 functions, named after what they return.

This revision now requires changes to proceed.Jul 13 2023, 06:54
emack marked an inline comment as done.

Good point,

  • utxosByType now split into separate xecUtxos and slpUtxos functions that leverage the existing utxos API for chronik interaction and specifically returns XEC or SLP utxos.
  • Updated utxosSeparated to leverage the existing utxos API as well, returning an object containing XEC and SLP utxo sets.

Removed redundant array size checks

modules/chronik-client/README.md
74 ↗(On Diff #41436)

IMO either this one or the legacy utxos() should be removed, they are quite redondant. It's the same data with a different way to present the output. I don't see why an application would have a strong preference for one or the other, and most likely the legacy one should be deprecated. Do you know of any app that would have use for the legacy call ? If not let's update it to return whatever we want.

Does JS has union types ? That would be another option that makes it backward compatible, and since there is (per doc) no guarantee on ordering it is still providing the new output as well if needed, all of that with no duplicated data.

bytesofman added inline comments.
modules/chronik-client/README.md
74 ↗(On Diff #41436)

my bad for requesting this

Can we keep the original functionality of the script.utxos() call so that this is not a breaking change?

On the one hand, it would be nice to ensure that no current users need to implement the new format returned. On the other hand, the ecosystem is small at the moment, and having an indexer that returns split utxos by default is much better than having an indexer that puts this burden on app developers.

Users that implement the new API would just see their whole app break if they don't upgrade (since utxos.utxos no longer exists), which imo is prob an acceptable way for it to fail (vs, say, they suddenly start getting slp utxos where they expected xec utxos).

modules/chronik-client/package.json
3 ↗(On Diff #41436)

0.9.0 (added features in a backward-compatible way)

or, more likely per above comments

1.0.0, for breaking change

modules/chronik-client/test/test.ts
444 ↗(On Diff #41436)

avoid formatting changes unrelated to the diff

This revision now requires changes to proceed.Jul 13 2023, 17:32
emack marked 4 inline comments as done.
  • deprecated the legacy utxos() function
  • the utxosSeparated() function is now refactored as the new utxos() function which interacts with chronik
  • the xecUtxos() and slpUtxos() functions now calls upon the refactored utxos() function for chronik interaction
  • bumped version to 1.0.0 to reflect breaking change
  • updated README
modules/chronik-client/index.ts
218 ↗(On Diff #41444)

Why is there a [0] here ?

224 ↗(On Diff #41444)

Same question

emack marked 2 inline comments as done.Jul 14 2023, 09:22
emack added inline comments.
modules/chronik-client/index.ts
218 ↗(On Diff #41444)

If an address has utxos, chronik returns an array with one entry, an object that contains the outputScript and the utxos array.

[

{
     outputScript: '<hash160 used to call this routine>',
     utxos: [
         [Object], [Object], [Object], [Object], [Object], [Object],
     ]
 }

]

Tobias designed it this way so that it could handle multiple different output scripts in one query, particularly for the taproot with state on Lotus, which can have an arbitrary payload along with the pubkey. In that scenario you provide the pubkey and it gives all the outputs with the different paylods each having one object in the outer array.

224 ↗(On Diff #41444)

as above

modules/chronik-client/index.ts
218 ↗(On Diff #41444)

OK thanks. In our case the array length will always be 1 (maybe we should assert it?), so we should abstract this and just returns the single object instead of expecting the callsite to know about this and use [0] everyhwere.

emack marked 3 inline comments as done.
  • updated utxos API to return the single entry utxo object directly
  • updated utxo response in xecUtxos and slpUtxos
Fabien added inline comments.
modules/chronik-client/index.ts
217 ↗(On Diff #41446)

Nice

223–224 ↗(On Diff #41446)
230 ↗(On Diff #41446)

Same

modules/chronik-client/index.ts
201 ↗(On Diff #41446)

Is the double capitalization of "U" and "T" here a typo? Why legacyScriptUTxos and not legacyScriptUtxos?

218 ↗(On Diff #41446)

legacyScriptUtxos[0] ?

emack marked 5 inline comments as done.

Updated nits and typo

@bytesofman - does this diff need to wait till D14271 lands, followed by running npm publish?

modules/chronik-client/index.ts
201 ↗(On Diff #41446)

yes to be corrected.

218 ↗(On Diff #41446)

yup typo, to be corrected.

Rebased to D14271 and executed npm run build

Also verified via local npm installation of this diff on a test script
npm install ./chronik-client

const { ChronikClient } = require('./chronik-client');
const chronik = new ChronikClient('https://chronik.fabien.cash');
const ecashaddr = require('ecashaddrjs');

async function getUtxosFromAddress() {
    const { type, hash } = ecashaddr.decode('ecash:qq9h6d0a5q65fgywv4ry64x04ep906mdku8f0gxfgx', true);
    const utxos = await chronik.script(type, hash).utxos();
    console.log(utxos);
};

(async () => {
    await getUtxosFromAddress();
})();

Diff looks good. Talking with @tobias_ruck -- he has some reservations about merging this due to upcoming have-to-happen changes in chronik-client, lack of backward compatibility, and overcomplicating the API.

imo having a way to get only xec utxos or only slp utxos is a net win, since otherwise you must get all the utxos from a server and filter out what you don't need.

In the meantime, @emack, let's just get the examples diffs rolling again without this. May have a window for landing this in the series of chronik-client upgrades needed to support the in-node version.

modules/chronik-client/README.md
55 ↗(On Diff #41558)

nit: we are also now returning an object instead of an array

emack planned changes to this revision.Jul 28 2023, 12:13

Per feedback from @tobias_ruck , we can abandon this one for now. It simplifies the examples/ scripts but is not necessary for wallet devs. Can be revived if we later decide the functionality is required.

For reference, here's some points I raised with Joey and Fabien (redacted somewhat):

I’m not really convinced this is an improvement, why would you group it by XEC vs SLP? Wouldn’t you also want to group the SLP tokens by token ID? The intention behind chronik-client was mostly to provide a easy to use client to access the data coming from Chronik. Wouldn’t it make more sense to add a auxiliary library for this, which can provide a wrapper around chronik-client?

Frankly I don’t really need this splitting (I filter by token ID), and I think there’s a danger that we add too much post processing to chronik-client, kinda inviting the devil.

An even better approach for the future might be to have Chronik filter by XEC-only/token ID A/token ID B, etc. server side.

Right now I’m just unhappy with the proposed API because the application isn’t very clear. It’s still not clear to me why this is really needed. It’s such a simple operation; but it often has to be tailored to the specific use-case. The scope of this improvement is very narrow (it doesn’t benefit me for example), and for most SLP wallets it’s not sufficient either. Really the only place where it’s useful is XEC wallets that are SLP aware, which is very narrow.

What’s wrong with adding the filtering to e.g. a xec-js library? That way we can keep it decoupled. chronik-client can focus on what it’s best at (a thin reliable client for Chronik), and eg xec-js can focus on providing a simple to use API to the XEC world.

The issue is not with this particular change, it’s with the future changes that will be added. Those will always be opinionated, and have tradeoffs. And those should be done. They will simplify the dev UX massively. But I think it’s better to have each library focus on what they’re best at.

Also, considering that there will be slight changes when we migrate to the native Chronik version, plus changes based on SLPv2, is why I’m a bit skeptical of adding stuff like this already, it’ll make upgrades harder as the design starts to ossify. I’d rather take a holistic approach, look at the goals of the library and design it accordingly, instead of adding stuff as we see fit.

There’s also other issues; like the usage of xec in the name, it’ll make it awkward to use for XPI users (Stamp is already using the library).

IMO, chronik-client should stay thin, and there should be a library around it with all the fancy more specific stuff. Another option would be to have a fork with a new library name on npm for it where you can do the XEC specific stuff, but I like that less than the layered approach. There’s some inherent complexity here which I don’t think one can remove simply, for what you guys want to do, it either has to be a forked chronik-client or a layered approach.

As per feedback above.

Will handle XEC/SLP utxo splitting function via the ecash-coinselect module in D14227.