Page MenuHomePhabricator

[Cashtab] Implement websocket in OrderBook
ClosedPublic

Authored by bytesofman on Sun, Jan 12, 18:33.

Details

Reviewers
Fabien
Group Reviewers
Restricted Project
Commits
rABC5bc41ee4ec03: [Cashtab] Implement websocket in OrderBook
Summary

On busier OrderBooks, it's just like an exchange. Users are buying and selling.

Right now Cashtab only loads the state of the OrderBook once. If another user creates a new listing or buys an existing listing, the OrderBook does not refresh for users who already have it open. Then if they try to accept an offer, its utxo may no longer exist and they get an error.

Solution is to use websockets so that the OrderBook updates itself according to agora activity as it happens. It is possible to optimize this going forward for speed. For a first pass, we only refresh the OrderBook every time we see an agora action related to its token added to the mempool.

Test Plan

npm test to confirm we did not break anything

test site: https://cashtab-local-dev.netlify.app

Added debug log on the test site so you can see the msgs that come in. List a token. Observe that the OrderBook updates itself and the wsMsg is in the dev console.

This is quite a complicated piece of logic to test in react testing library. We could accomplish it with sinon and spies. However the tested behavior (for now) is very simple. If it gets more complex, perhaps OrderBook should be exported and tested by ecash-agora; may need regtest integration to get a "good" websocket test.

Here is what I did

  1. Navigate to test site
  2. Hard refresh to make sure you are on this version (ctrl + shift + r in brave/chrome)
  3. Go to a token page
  4. Buy or list that token, notice in the dev console that the offers are refreshed. Confirm the offers are refreshed by checking the offer your action changed.
  5. Manually enter another tokenId in the nav bar. Check the dev console to confirm we are closing the ws for this tokenId and loading one for a new tokenId. Also getting offers for the new tokenId.
  6. Navigate to the Agora screen. In the dev console, note that the whitelisted OrderBooks get a websocket.
  7. Buy some XECX or Star Crystal. In the dev console see the orderbooks refresh. Confirm the offer you bought has been updated.
  8. Toggle to Manage Offers. Confirm websockets are presesnt.
  9. Load all offers. In the dev console, see that all offers are loaded with websocket disabled.

It's quite a test plan and a bit manual. But testing websocket actions without actually firing transactions quickly becomes an exercise in testing the testing libraries. imo the solution is to not make the websocket more complicated in Cashtab. If we need more optimized orderbook websockets (updating indivdual offers as they are partially accepted, removing individual offers as they are canceled) -- then this component should get its own tested library. That way we could use regtest on it with ABC monorepo CI, like we do with ecash-lib and ecash-agora.

Diff Detail

Repository
rABC Bitcoin ABC
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

back out formatting changes handled in separate diff

Failed tests logs:

====== CashTab Unit Tests: <CreateTokenForm /> User can create an SLP1 token with a mint baton ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <CreateTokenForm /> User can create an ALP token ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <Token /> Accepts a valid ecash: prefixed address ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <Token /> Accepts a valid etoken: prefixed address ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <Token /> Displays a validation error for an invalid address ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <Token /> Displays a validation error if the user includes any query string ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <Token /> Renders the send token notification upon successful broadcast ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <Token /> Renders the burn token success notification upon successful burn tx broadcast ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <Token /> We can mint an slpv1 token if we have a mint baton ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <Token /> We can mint an slpv1 token if we have a mint baton and confirm modals enabled ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at listOnTimeout (node:internal/timers:545:9)
    at processTimers (node:internal/timers:519:7)
====== CashTab Unit Tests: <Token /> For an uncached token with no balance, we show a spinner while loading the token info, then show an info screen and open agora offers ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <Configure /> Setting "ABSOLUTE MINIMUM fees" settings will reduce fees to absolute min ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at listOnTimeout (node:internal/timers:545:9)
    at processTimers (node:internal/timers:519:7)
====== CashTab Unit Tests: <Token /> available actions rendered SLP1 fixed supply token ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <Token /> available actions rendered SLP1 variable supply token with mint baton ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <Token /> available actions rendered ALP token ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)

Each failure log is accessible here:
CashTab Unit Tests: <CreateTokenForm /> User can create an SLP1 token with a mint baton
CashTab Unit Tests: <CreateTokenForm /> User can create an ALP token
CashTab Unit Tests: <Token /> Accepts a valid ecash: prefixed address
CashTab Unit Tests: <Token /> Accepts a valid etoken: prefixed address
CashTab Unit Tests: <Token /> Displays a validation error for an invalid address
CashTab Unit Tests: <Token /> Displays a validation error if the user includes any query string
CashTab Unit Tests: <Token /> Renders the send token notification upon successful broadcast
CashTab Unit Tests: <Token /> Renders the burn token success notification upon successful burn tx broadcast
CashTab Unit Tests: <Token /> We can mint an slpv1 token if we have a mint baton
CashTab Unit Tests: <Token /> We can mint an slpv1 token if we have a mint baton and confirm modals enabled
CashTab Unit Tests: <Token /> For an uncached token with no balance, we show a spinner while loading the token info, then show an info screen and open agora offers
CashTab Unit Tests: <Configure /> Setting "ABSOLUTE MINIMUM fees" settings will reduce fees to absolute min
CashTab Unit Tests: <Token /> available actions rendered SLP1 fixed supply token
CashTab Unit Tests: <Token /> available actions rendered SLP1 variable supply token with mint baton
CashTab Unit Tests: <Token /> available actions rendered ALP token

Failed tests logs:

====== CashTab Unit Tests: <CreateTokenForm /> User can create an SLP1 token with a mint baton ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <CreateTokenForm /> User can create an ALP token ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <Token /> Accepts a valid ecash: prefixed address ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <Token /> Accepts a valid etoken: prefixed address ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <Token /> Displays a validation error for an invalid address ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <Token /> Displays a validation error if the user includes any query string ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <Token /> Renders the send token notification upon successful broadcast ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <Token /> Renders the burn token success notification upon successful burn tx broadcast ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <Token /> We can mint an slpv1 token if we have a mint baton ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <Token /> We can mint an slpv1 token if we have a mint baton and confirm modals enabled ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at listOnTimeout (node:internal/timers:545:9)
    at processTimers (node:internal/timers:519:7)
====== CashTab Unit Tests: <Token /> For an uncached token with no balance, we show a spinner while loading the token info, then show an info screen and open agora offers ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <Configure /> Setting "ABSOLUTE MINIMUM fees" settings will reduce fees to absolute min ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at listOnTimeout (node:internal/timers:545:9)
    at processTimers (node:internal/timers:519:7)
====== CashTab Unit Tests: <Token /> available actions rendered SLP1 fixed supply token ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <Token /> available actions rendered SLP1 variable supply token with mint baton ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)
====== CashTab Unit Tests: <Token /> available actions rendered ALP token ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:781:15)

Each failure log is accessible here:
CashTab Unit Tests: <CreateTokenForm /> User can create an SLP1 token with a mint baton
CashTab Unit Tests: <CreateTokenForm /> User can create an ALP token
CashTab Unit Tests: <Token /> Accepts a valid ecash: prefixed address
CashTab Unit Tests: <Token /> Accepts a valid etoken: prefixed address
CashTab Unit Tests: <Token /> Displays a validation error for an invalid address
CashTab Unit Tests: <Token /> Displays a validation error if the user includes any query string
CashTab Unit Tests: <Token /> Renders the send token notification upon successful broadcast
CashTab Unit Tests: <Token /> Renders the burn token success notification upon successful burn tx broadcast
CashTab Unit Tests: <Token /> We can mint an slpv1 token if we have a mint baton
CashTab Unit Tests: <Token /> We can mint an slpv1 token if we have a mint baton and confirm modals enabled
CashTab Unit Tests: <Token /> For an uncached token with no balance, we show a spinner while loading the token info, then show an info screen and open agora offers
CashTab Unit Tests: <Configure /> Setting "ABSOLUTE MINIMUM fees" settings will reduce fees to absolute min
CashTab Unit Tests: <Token /> available actions rendered SLP1 fixed supply token
CashTab Unit Tests: <Token /> available actions rendered SLP1 variable supply token with mint baton
CashTab Unit Tests: <Token /> available actions rendered ALP token

Offer a noWebsocket prop so that we do not use websockets when we load too many OrderBooks to make them usable, implement in Agora

Failed tests logs:

====== CashTab Unit Tests: <CreateTokenForm /> User can create an SLP1 token with a mint baton ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:791:15)
====== CashTab Unit Tests: <CreateTokenForm /> User can create an ALP token ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:791:15)
====== CashTab Unit Tests: <Token /> Accepts a valid ecash: prefixed address ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:791:15)
====== CashTab Unit Tests: <Token /> Accepts a valid etoken: prefixed address ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:791:15)
====== CashTab Unit Tests: <Token /> Displays a validation error for an invalid address ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:791:15)
====== CashTab Unit Tests: <Token /> Displays a validation error if the user includes any query string ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:791:15)
====== CashTab Unit Tests: <Token /> Renders the send token notification upon successful broadcast ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:791:15)
====== CashTab Unit Tests: <Token /> Renders the burn token success notification upon successful burn tx broadcast ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:791:15)
====== CashTab Unit Tests: <Token /> We can mint an slpv1 token if we have a mint baton ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:791:15)
====== CashTab Unit Tests: <Token /> We can mint an slpv1 token if we have a mint baton and confirm modals enabled ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:791:15)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at listOnTimeout (node:internal/timers:545:9)
    at processTimers (node:internal/timers:519:7)
====== CashTab Unit Tests: <Token /> For an uncached token with no balance, we show a spinner while loading the token info, then show an info screen and open agora offers ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:791:15)
====== CashTab Unit Tests: <Configure /> Setting "ABSOLUTE MINIMUM fees" settings will reduce fees to absolute min ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:791:15)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at listOnTimeout (node:internal/timers:545:9)
    at processTimers (node:internal/timers:519:7)
====== CashTab Unit Tests: <Token /> available actions rendered SLP1 fixed supply token ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:791:15)
====== CashTab Unit Tests: <Token /> available actions rendered SLP1 variable supply token with mint baton ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:791:15)
====== CashTab Unit Tests: <Token /> available actions rendered ALP token ======
TypeError: agora.subscribeWs is not a function
    at subscribeWs (/work/cashtab/src/components/Agora/OrderBook/index.tsx:791:15)

Each failure log is accessible here:
CashTab Unit Tests: <CreateTokenForm /> User can create an SLP1 token with a mint baton
CashTab Unit Tests: <CreateTokenForm /> User can create an ALP token
CashTab Unit Tests: <Token /> Accepts a valid ecash: prefixed address
CashTab Unit Tests: <Token /> Accepts a valid etoken: prefixed address
CashTab Unit Tests: <Token /> Displays a validation error for an invalid address
CashTab Unit Tests: <Token /> Displays a validation error if the user includes any query string
CashTab Unit Tests: <Token /> Renders the send token notification upon successful broadcast
CashTab Unit Tests: <Token /> Renders the burn token success notification upon successful burn tx broadcast
CashTab Unit Tests: <Token /> We can mint an slpv1 token if we have a mint baton
CashTab Unit Tests: <Token /> We can mint an slpv1 token if we have a mint baton and confirm modals enabled
CashTab Unit Tests: <Token /> For an uncached token with no balance, we show a spinner while loading the token info, then show an info screen and open agora offers
CashTab Unit Tests: <Configure /> Setting "ABSOLUTE MINIMUM fees" settings will reduce fees to absolute min
CashTab Unit Tests: <Token /> available actions rendered SLP1 fixed supply token
CashTab Unit Tests: <Token /> available actions rendered SLP1 variable supply token with mint baton
CashTab Unit Tests: <Token /> available actions rendered ALP token

tests that do not specify mockAgora now need mockAgora

Fabien requested changes to this revision.Mon, Jan 13, 09:40
Fabien added a subscriber: Fabien.

I'm not sure about the strategy here:

  • Does it make sense to have ws on the agora page (vs the token page) ?
  • If yes it would make sense to only subscribe to the rendered order books (the ones on screen) so you have a natural subscription limit
cashtab/src/components/Agora/OrderBook/index.tsx
139 ↗(On Diff #52158)
140 ↗(On Diff #52158)

why is that a setting if it's the correct behavior ? The user should not be able to trigger a dos with a setting

This revision now requires changes to proceed.Mon, Jan 13, 09:40

I'm not sure about the strategy here:

  • Does it make sense to have ws on the agora page (vs the token page) ?
  • If yes it would make sense to only subscribe to the rendered order books (the ones on screen) so you have a natural subscription limit

Does it make sense to have ws on the agora page (vs the token page) ?

Not if we show 100s of tokens -- the plan though is to engineer past that. This is being held up at the moment by my trying to figure out the best way to load limited orderbook data while still loading enough in the background to make search useful. But the interim solution of only loading whitelisted tokens at first -- well, it's nice to have ws subscriptions just for those. This is what we do in this diff.

It would be nice from the user perspective to be subscribed for every orderbook that is rendered. Esp going forward, we could do things like show notifications on buys and listings -- this would be crazy noisy for 700 offers but interesting for say 5 whitelisted offers.

If yes it would make sense to only subscribe to the rendered order books (the ones on screen) so you have a natural subscription limit

agreed -- having the ws subscription toggle as a setting in the param is a way to do this. Now OrderBook is a controlled component with this param -- this param can be a variable in the parent component and turn the websocket "on" or "off in orderbook depending on logic determined in the parent component, in this case the agora screen.

The way the app behaves now, having noWebsocket as a setting is useful because it makes OrderBook a controlled component with this prop -- i.e. we can switch the websocket on or off from its parent component.

So, on the Token page, where we have only one OrderBook, the websocket is on (default setting)

On the Agora page, where we sometimes have manageable OrderBooks, and sometimes have too many, the websocket is on when they are manageable (whitelist only, user managing own offers) and off when they are not (user has elected to load all the OrderBooks).

bytesofman marked 2 inline comments as done.
bytesofman added inline comments.
cashtab/src/components/Agora/OrderBook/index.tsx
140 ↗(On Diff #52158)

Still iterating through how this component will be used. imo OrderBook should never be rendered beyond the user-reviewable limit; say something like 5-12, whatever would fit on a screen.

However, it is useful to render 100s of this component as this provides a way to get info about the entire agora market in a parallel way. My plan is to add some way to render a "headless" orderbook ... ie you can include an OrderBook for a given tokenId and have it populate orderBookInfoMap to build searchable data quickly, but not actually render anything on the screen.

It makes sense that default OrderBook behavior is to have a websocket, since there is always trading activity and the OrderBook is not usable if it only presents stale offers.

noWebsocket is an optional prop that allows us to render the orderbook with no websocket if we are using it in a way that we do not need this feature ... i.e. loading 100s of orderbooks just to populate orderBookInfoMap to get searchable data.

I think a sensible next step would be to have a prop headless or infoOnly that replaces noWebsocket -- then we associate turning the websocket on or off with whether or not we are actually presenting the component to the user. but I still have some work to do to get that done, and I think the websocket is high enough impact to put in prod first.

It's also possible I discover that headless is still too much memory, and I need to pursue some other strategy to load agora info that is useful and searchable (like handling it in `useWallet.ts, so it is always happening in the background ... but that seems overkill since I don't know if I can assume all users want to load this info).

It is a bit messy iterating to this type of solution. But I get lots of useful info from user feedback (and my own personal use) of the component in the production app. There is enough complexity involved that it is difficult to try and work it all out "in theory" before implementing. Iterating through implementations is part of the design process.

bytesofman marked an inline comment as done.

typo fix "component" not comonent

This revision is now accepted and ready to land.Mon, Jan 13, 15:00

minor version bump, some code tweaks from additional testing, test site redeployed with this version with debug logs

remove unrelated line break, preserve offer refresh if ws is null