Page MenuHomePhabricator

[ecash-wallet] Add fluent API for building and broadcasting txs
Changes PlannedPublic

Authored by bytesofman on Mar 19 2025, 04:27.

Details

Reviewers
tobias_ruck
Group Reviewers
Restricted Project
Summary

We want a powerful and user-friendly API that exposes commonly used steps in tx processing.

Here we initialize such an API. The MVP includes basic support for SLP and ALP txs.

  • We can get the utxos required to build an arbitary tx without actually building the tx by calling wallet.tx(<instructions>).utxos()
  • We can build a tx (and get info about it) without broadcasting it, by calling wallet.tx(<instructions>).build().fee() or wallet.tx(<instructions>).build().size(); or the actual TxBuilder with wallet.tx(<instructions>).build().tx
  • We can build and broadcast a tx with one call with await wallet.tx(<instructions>).build().broadcast()

    From here,
  • We should add a websocket so the user does not need to remember to call await wallet.sync() before sending every tx
  • Support other token types
  • Support Agora actions
  • Add support for chained txs, i.e. await wallet.tx(<instructions>).build().broadcast() should "just work" even if the instructions are asking to send more token outputs than a single tx can hold, or more XEC outputs than the tx broadcast limit of 100kb will tolerate.
Test Plan

npm test

integration tests:
build node in build dir, then, from bitcoin-abc/modules/ecash-wallet/, BUILD_DIR="${PWD}/../../build" npm run integration-tests

Review API, suggest different names, shapes of objects, anything that should go in now before we start to add more features

Diff Detail

Repository
rABC Bitcoin ABC
Branch
wallet-txs-regtest
Lint
Lint Passed
Unit
No Test Coverage
Build Status
Buildable 33350
Build 66181: Build Diffecash-wallet-tests
Build 66180: arc lint + arc unit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Pushing up this last round of feedback-based changes to support incremental review, but this is not ready to land yet

  • Confirm general path forward re: tokens and Action shape, imo we need to get a loose token API in this diff (make sure we aren't preventing anything about what we want to build), but we will need to really work out the token API as we add those features and confirm with regtest
  • Iterate on API changes with action and WalletTxs class; is this approach ok? Are we allowing enough granular control?
  • On confirming approach, get rid of what we don't need (i.e. with the approach up now, we no longer need the getRequiredUtxos function or the helpers.ts functions (e.g. getSignedDummyTx).
modules/ecash-wallet/src/wallet.ts
57 ↗(On Diff #53214)

nice yeah this makes sense / is simplest

69 ↗(On Diff #53214)

Difficult to get this right at the abstract level, i.e. before we are actually building these txs, getting the utxos for these transactions

High level -- should probably either

  1. define all token actions using only Output; design the interface of Output to accomodate this
  2. define all token actions where outputs must be calculated by token spec separately from Output in e.g. MintAction, BurnAction.

Imo there is a stronger case for 2.

For example if we have two genesis txs in Action ... we can't just have a bunch of outputs with no tokenId and isGenesis at the outputs key. We need to group the specific genesis outputs we want with each request. Would make sense to do this in distinct GenesisAction objects.

For this diff, I think we need a reasonably-acceptable framework + tentative dev direction. Will need to focus on the specific logic in diffs that add + test this functionality.

80 ↗(On Diff #53214)

If we go this way, do we need MintAction at all? could be a useful interface for GenesisAction but I don't see why we need it as a tokenAction; mint outputs could be defined by isMintBaton and tokenId in Output ....

With this approach, we would also need some way in Output to distinguish between MINT outputs and SEND outputs.

It seems pretty complicated if we specify Outputs for MINT actions in the outputs: key but then also have to "color" these outputs with a MintAction. Want to limit as much as possible the need for a spec just to create the input for ecash-wallet txs.

As above tho, it's hard to get this right in a diff that is not building and testing mint txs. For now we should get a loose placeholder and general agreement on the dev direction, likely to change as we implement.

100 ↗(On Diff #53214)

what if we have more than one GenesisAction? How do we know which isGenesis outputs belong to which GenesisAction?

imo it makes sense to define action-specific genesis outputs in an action interface. Or we could make a rule about only one genesis action allowed.

109 ↗(On Diff #53214)

yes, this is simpler. Since Action will need to be some kind of Interface anyway, might as well add config there as keys. Can still get the desired behavior of "if these keys are not specified, ecash-wallet will use sensible default values".

NB will need to add more tests but needs review now to confirm API and dev direction ... extensively testing the wrong API won't help anyone

tobias_ruck added inline comments.
modules/ecash-wallet/src/wallet.ts
55 ↗(On Diff #53365)

perfect! that should cover everything + is easy to use (I suspect)

80 ↗(On Diff #53365)

we should export these types

82–89 ↗(On Diff #53365)

this should be inferable from outputs

95 ↗(On Diff #53365)
96 ↗(On Diff #53365)

oh we also need this, otherwise we can't build the OP_RETURN

and this (modelled after ChronikClient.ts):

export interface TokenType {
  protocol: 'SLP' | 'ALP';
  number: number;
}
100 ↗(On Diff #53365)

(also, move this before MintAction. Canonical order is GENESIS, MINT, SEND, BURN)

102–109 ↗(On Diff #53365)

same here, MintData can be inferred, that's why we have isGenesis

215 ↗(On Diff #53365)
267 ↗(On Diff #53365)

need an extra function for tokens, so better make it clear already

297 ↗(On Diff #53365)

you actually need to derive the OP_RETURN from the outputs somewhere.

the process would look like this (steps might be melded):

  1. Collect all used token IDs from the outputs (plus one for genesis, if there's one. For this calculation, it's probably best to assign a dummy token ID for GENESIS)
  2. Determine the last output index for each token ID, including for genesis.
    1. So if token A sends to output 1, 2, 7, this is 7, and token B sends to 3, 4, this is 4
    2. If there's any mint batons, the last output index will be the index of the first mint baton - 1 (so if the token sends tokens to 1, 2, 7 and a mint baton to 9 and 10, the last output index will be 8).
    3. Also, count the mint batons of a token ID (and ensure they are consecutive (i.e. 9, 10, 11 is ok, but 9, 10, 12 is not). This is because mint batons are assigned by a single number ("numBatons"), and they come after the send amounts).
    4. At the end we have two numbers for each token ID: last output index, and number of mint batons.
  3. Allocate zero'd bigint[] arrays for each token ID ("atomsArray") with the number of items equal to the last output index from step 2.
  4. Go through each of the action's outputs again, and for output N, if it has any atoms, assign atomsArray[N] of the corresponding token ID the atoms of the output
  5. Now we can build all the ALP/SLP pushdata.
    1. For ALP, for each token ID (including genesis), we create a MINT if there's a MintAction in tokenActions, or a GENESIS if there's a GenesisAction, with the atomsArray and numBatons derived before. Otherwise, if there's at least 1 atom in the outputs, create a SEND. Independent of this, if there's a BurnAction, add a BURN (so (SEND|MINT) and BURN can co-exist).
    2. For SLP, this will be similar, but a GENESIS and MINT is more limited, also there can only be either a GENESIS, SEND, MINT or BURN, but not multiple.
  6. And then we add this in an OP_RETURN at the beginning of the tx's outputs (if there's any ALP or SLP data)
This revision now requires changes to proceed.Apr 4 2025, 18:37
bytesofman added inline comments.
modules/ecash-wallet/src/wallet.ts
82–89 ↗(On Diff #53365)

ok -- this makes sense for ALP, but mb not for SLP?

The issue here is I don't want the user to have to be familiar with the spec of the respective token protocol when building tx instructions

For ALP ... they can just go anywhere?

For SLP tho we can only have the 1 quantity, has to be at index 1, mint batons need to follow.

We can deal with the specifics of this problem when we get there, but in general I do like the principal of "any time there is some specific spec rule that needs to be worked out, ecash-wallet should do the lifting there and the user instructions should be as abstract as possible"

82–89 ↗(On Diff #53365)

Oh, well the user actions are not index-ordered outputs anyway, esp as we may or may not need to generate and insert an OP_RETURN output at index 0

102–109 ↗(On Diff #53365)

caveat as above of "maybe not for SLP" (or other as-yet-unknown protocols) -- but just something to keep in mind for when we get to that implementation

267 ↗(On Diff #53365)

good point

297 ↗(On Diff #53365)

imo we can do this in a separate diff; esp as, for now, we just throw an error if any tokenActions are specified.

bytesofman marked 4 inline comments as done.

add TODO for OP_RETURN parsing, update and export actions, use correct order for actions, typo patch

Failed tests logs:

====== Can sum value of utxos: Returns 0 if called with no utxos.static methods Can sum value of utxos Returns 0 if called with no utxos ======
TypeError: import_wallet.default.sumUtxos is not a function
    at Context.<anonymous> (src/wallet.test.ts:263:27)
    at process.processImmediate (node:internal/timers:483:21)
====== Can sum value of utxos: Can get total eCash satoshis held by all kinds of utxos.static methods Can sum value of utxos Can get total eCash satoshis held by all kinds of utxos ======
TypeError: import_wallet.default.sumUtxos is not a function
    at Context.<anonymous> (src/wallet.test.ts:266:27)
    at process.processImmediate (node:internal/timers:483:21)

Each failure log is accessible here:
Can sum value of utxos: Returns 0 if called with no utxos.static methods Can sum value of utxos Returns 0 if called with no utxos
Can sum value of utxos: Can get total eCash satoshis held by all kinds of utxos.static methods Can sum value of utxos Can get total eCash satoshis held by all kinds of utxos

update missed test function name

modules/ecash-wallet/src/wallet.ts
82–89 ↗(On Diff #53365)

this makes sense for ALP, but mb not for SLP?

For SLP tho we can only have the 1 quantity, has to be at index 1, mint batons need to follow.

For SLP it also makes sense—keep in mind there's v1 which only allows one mint output and v2 which allows multiple like ALP... and v2 has no mint batons

So better to just handle it in one API

The issue here is I don't want the user to have to be familiar with the spec of the respective token protocol when building tx instructions

Yes, totally agree—that's what I'm actually trying to accomplish also.

We can deal with the specifics of this problem when we get there, but in general I do like the principal of "any time there is some specific spec rule that needs to be worked out, ecash-wallet should do the lifting there and the user instructions should be as abstract as possible"

Exactly—the goal is that the user just specifies how the outputs should look like.

Point taken that it would be easier to just have a single "mint quantity to here, mint baton to there" API for SLP v1, but I think a user should still have a semblance of "output". And it's good to have a unified interface for all SLP+ALP.

For now, I think the correct behavior is to simply enforce for SLP that the mint quantity must be at output 1 and the mint baton somewhere after, and throw a helpful error if otherwise. ALP also needs to enforce this. If the user sees an error, he then simply reorders their outputs and the problem is solved—no need to read specs. Also minting with multiple amounts is kinda an advance thing anyway.

In the future, we might want to add more abstract helper methods (e.g. just wallet.mint(address, quantity), sending back the baton to the wallet), but for now I think this is fine.

297 ↗(On Diff #53365)

Definitely, but maybe it'd be a good idea to already have an implementation for this in a draft so we know it does work and the design isn't somehow really stupid.

This revision now requires changes to proceed.Apr 10 2025, 23:06
bytesofman added inline comments.
modules/ecash-wallet/src/wallet.ts
82–89 ↗(On Diff #53365)

ok got it, error behavior makes sense. The output order is not THAT comlplicated re: spec so should be fine.

It's also a bit of an issue with send txs, where we will likely automatically calculate and add change outputs (since these, and the OP_RETURN, are dependent on available utxos)

mb issue not really the right word -- but it is another case where the lib will handle spec and user input to "outputs" will not necessarily be explicit to what happens in the tx.

297 ↗(On Diff #53365)

the last output index will be the index of the first mint baton - 1

should probably just be the last output index of a non-mintbaton mint amount, no? since it is possible to mint with no mint batons, and have no index of the first mint baton -1.

297 ↗(On Diff #53365)

Update: I have been working on this for several days ... it's pretty complicated. I am making progress and working toward something that is reviewable.

It's a necessary exercise to get this API right but ... damn... I mean there really are lots of things to consider.

Before this lib tho, every token tx would need outputs to be generated "black box" style. It would be nice if the user were able to specify outputs with certain token actions, and this type of behavior is probably critical if we ever need "complex" token action txs, e.g. multiple token actions in the same tx in ALP.

bytesofman marked an inline comment as done.

coming up for air, taking a look at what has happened

this is not a great stopping point but I have done so many rewrites and changes now that wanted to get some kind of anchor in for reference

the "validateOutputs" function is I think the way to go. buildOpReturn here is now totally obsolete, needs to be rewritten.

General ideas that I think will stick

  • We do not need a mint action, instead we should make outputs specify mint qty. This makes outputs a bit more complicated but the whole point of designing this around outputs is to allow good granularity for user txs, so I think it's ok
  • The general approach of "just feed an array of token actions into tokenActions key and parse them" proved hard and complex to validate. Since the mint action was just a tokenId, and now we don't have that, really the only actions we need to specify are

genesis and burn. Instead of putting those into an array, keep genesis action at a key (there can be only one), and accept an array of burn actions

  • It is (probably) possible to validate and build the op return at the same time. But it is very hard to do this in your head and hit all the cases. I think readability is better if we validate first, then calculate everything. So, that is the approach started here

Next up

  • Clean up tests, comments for validateOutputs
  • Implement it in buildOpReturn
  • Finish buildOpReturn so it fulfillls the original inline comment vision, add tests
  • Integrate into the class with tests

At this point, mb need to just add all the token features, or at least enough to regtest token txs. It makes the diff bigger...but a major challenge with this diff is that so much depends on so much else, the diff is itself an opportunity to rework the API.

Add SLP actions to (working) regtest, NB still lots to do to clean up tests, implement ALP

bytesofman planned changes to this revision.EditedTue, Apr 29, 17:45

Pushed at a better-but-still-not-great stopping point, where regtest works for SLP actions

From here,

  • Get regtest to work for ALP
  • Clean up unit tests for the shape the helper functions took
  • Add unit tests for coverage
  • Clean up debug logs, TODO comments
  • Handle anticipated rework in reviews

Approach so far achieves the goal of allowing the user to specify outputs to construct complex token txs. A handful of design decisions to support this:

  • We always add a leftover output (bc this should be at the end, where it will not interfere with token outIdx constraints)
  • We allow directed unintentional burns for SLP only, bc intentional burns would require utxo preparation or selection

rebase, get regtest working with SLP and ALP. NB we still have lots of cleaning up and other tests to add.

regtest handles basic cases for SLP and ALP now. Next up is cleaning up debug logs, function names, adding unit tests for untested functions.

Failed tests logs:

====== ALP_TOKEN_TYPE_STANDARD: DOES NOT throw and adds adjusted change if BURN is specified and tx is otherwise valid, for 2 tokens.validateOutputs ALP_TOKEN_TYPE_STANDARD DOES NOT throw and adds adjusted change if BURN is specified and tx is otherwise valid, for 2 tokens ======
Error: Specified action results in OP_RETURN of 328 bytes, vs max allowed of 223.
    at validateOutputs (src/wallet.ts:1823:23)
    at Context.<anonymous> (src/wallet.test.ts:2478:28)
    at process.processImmediate (node:internal/timers:483:21)

Each failure log is accessible here:
ALP_TOKEN_TYPE_STANDARD: DOES NOT throw and adds adjusted change if BURN is specified and tx is otherwise valid, for 2 tokens.validateOutputs ALP_TOKEN_TYPE_STANDARD DOES NOT throw and adds adjusted change if BURN is specified and tx is otherwise valid, for 2 tokens

tobias_ruck added inline comments.
modules/ecash-wallet/src/wallet.ts
66 ↗(On Diff #53865)

generally the terms we use in the node are "policy" (for mempool acceptance rules) and "consensus" (for rules enforced on blocks), so let's use that here too :)

102 ↗(On Diff #53865)

shouldn't this be optional too?

for NonTokenOutput only: leftover XEC change
for TokenOutput: picks the DUST_LIMIT value by default

340 ↗(On Diff #53865)

shouldn't it be this for consistency?

This revision now requires changes to proceed.Sat, May 17, 12:27
bytesofman added inline comments.
modules/ecash-wallet/src/wallet.ts
102 ↗(On Diff #53865)

nice yeah makes a lot of sense to just default this to dust. wonder how many times I've typed 546 over the past few years 🤔

bytesofman marked an inline comment as done.

default to dust sats if unspecified, variable name improvements

tobias_ruck added inline comments.
modules/ecash-wallet/src/wallet.ts
72

If I understand correctly, when using tokens, the user is always expected to use an OpReturnOutput, and set script to undefined—I like that, but I think it should be made a bit clearer in the comment that it has to be specified and left empty for tokens.

Can you clarify what happens when using tokens (which will occupy the OP_RETURN) and a non-undefined script in OpReturnOutput? Will there be an error?

I think what you're proposing is a clean solution to have this as an "opaque OP_RETURN", for exceptional stuff that we can't/don't want to have special handling for in our system, and fail if the user tries to use a manual OP_RETURN and SLP/ALP simultaneously.

86

this is how OP_RETURN outputs are identified, right? the user sets sats to 0n?

Maybe add a clarifying comment that we know an output is OP_RETURN if its sats are 0n

124

I think this more clearly distinguishes the isMintBaton and isMinted variant

Note an alternative to this field that I suggest below; I think we should prefer this over the approach here

135

with my GENESIS_TOKEN_ID_PLACEHOLDER suggestion, I think it might be simpler to just have tokenId == 'GENESIS_TOKEN_ID_PLACEHOLDER' instead.

This way, we don't force people to see the isGenesis always. Not a big difference but I think it's a bit cleaner and simpler, and we do away with GenesisOutput

187

I suggest we have a generic list of token actions here (did we do this in a previous diff already and I lamented it? If not, I hope you don't go full Martin Luther with claiming contradictory councils, I'm not an infallible authority on code reviews)

I'm not sure if you're going to like this suggestion, but I think it solves a lot of problems simultaneously:

  • How to specify which token ID is a GENESIS, MINT, BURN, SEND
  • Allows us to add extra config data in the future for specific tx types (e.g. prevent a SEND (or MINT) to be split into multiple txs)
  • The order of the generated eMPP pushdata (maybe the BURN should be first, then the SEND, or vice versa)
  • Adding arbitrary eMPP pushdata in the middle

(comments below are part of the review, not suggested code comments)

type TokenAction = GenesisAction | BurnAction | MintAction | SendAction | DataAction;

// Naturally, the genesis action must be first, but this way we cleanly encode
// the empp pushdata in a list exactly in the order of how it's going
// to look like in the OP_RETURN
interface GenesisAction {
    type: 'GENESIS'; // to distinguish actions e.g. in a `switch (tokenAction.type)`
    tokenType: TokenType;
    genesisInfo: GenesisInfo;
}

interface BurnAction {
    type: 'BURN';
    tokenId: string;
    burnAtoms: bigint;
}

interface MintAction {
    type: 'MINT';
    tokenId: string;
}

interface SendAction {
    type: 'SEND';
    tokenId: string;
}
// in the future, can add more actions, e.g. to blacklist UTXOs (if the crown requires us)

// attaches an arbitrary eMPP pushdata
interface DataAction {
    type: 'DATA';
    data: Uint8Array; // or hex string...
}

an action then could look like this:

wallet.action({
  outputs: [
    { sats: 0n }, // OP_RETURN, filled in for us
    { sats: 10000n, script: Script.p2kh(...) },
    { tokenId: GENESIS_TOKEN_ID_PLACEHOLDER, atoms: 20n, script: ... },
    { tokenId: tokenId1, atoms: 1000n, script: ... },
    { tokenId: tokenId2, atoms: 300000n, script: ... },
    { tokenId: tokenId1, isMintBaton: true, script: ... },
    { script: ... }, // change output
  ],
  tokenActions: [
    { type: 'GENESIS', tokenType: ALP_STANDARD, genesisInfo: { ... } }, // must be first
    { type: 'MINT', tokenId: tokenId1 },
    { type: 'SEND', tokenId: tokenId2 },
    { type: 'DATA', data: fromHex("12345678") },
    { type: 'BURN', tokenId: tokenId2, burnAtoms: 12n },
  ],
})

SENDs and MINTs also have the special property that they can be split into multiple transactions (but we can implement this later)

202

BTW—we might want to consistently use this placeholder instead of of isGenesis, then we don't need a separate GenesisOutput; and TokenOutput is enough.

The user would have to set { tokenId: GENESIS_TOKEN_ID_PLACEHOLDER } for outputs coming from the GENESIS. Tbh it's a little makeshift and maybe confusing to newcomers, but worst-case we just provide a helper shorthand function to genesis a new token in a more "dumb" way.

This revision now requires changes to proceed.Sat, May 17, 16:51
bytesofman marked 3 inline comments as done.
  • Lose isGenesis key and instead require a genesis placeholder tokenId
  • Expand Action to accept an array of token actions per comment
  • Validate and implement these inputs (will change how we build EMPP OP_RETURNs)
  • (potentially) also lose the isMint key, since a MintAction will make this unnecessary
modules/ecash-wallet/src/wallet.ts
72

Yes, I ended up adding this because I ran into issues with slp spec requiring certain outputs to be at certain outIdx values. Because of this requirement, and because we are offering the user the granularity of setting exact outputs -- imo we need to make sure the outputs set by the user appear at the outIdx set by the user.

So, this leaves us with a bit of a weird externality in that the user must specify a blank OP_RETURN at outIdx 0. But the error msgs and typescript clarify the requirement.

Can you clarify what happens when using tokens (which will occupy the OP_RETURN) and a non-undefined script in OpReturnOutput? Will there be an error?

Looks like this test case was missed. Added. We get this error (from finalizeOutputs):

A token tx cannot specify any manual OP_RETURN outputs. Token txs can only include a blank OP_RETURN output (i.e. { sats: 0n} at index 0.
86

it's only the special case of a "to-be-defined OP_RETURN" that looks exactly like {sats:0n} where this is true

If the user is manually specifying an OP_RETURN, it would be identified by the script

For now, we have error handling so that any OP_RETURN must also include 0n at the sats: key (i.e. ecash-wallet will not support XEC burns in OP_RETURN outputs).

202

we might want to consistently use this placeholder instead of of isGenesis

... this is probably better yeah. a bit weird but we could handle it with the other weirdness of token specs, throwing an error if not implemented correctly. can take a look at this in isolation.