Page MenuHomePhabricator

[alias-server] implement new db functions in handleBlockConnected to fetch and store aliases
ClosedPublic

Authored by bytesofman on Apr 13 2023, 18:28.

Details

Summary

T3060

Modify handleBlockConnected function in events.js to update the database with aliases.

Modify mocks and unit tests for new logic.

Test Plan

npm test

Diff Detail

Repository
rABC Bitcoin ABC
Branch
make-the-app-work
Lint
Lint Passed
Unit
No Test Coverage
Build Status
Buildable 23483
Build 46585: Build Diffalias-server-tests
Build 46584: arc lint + arc unit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Fabien requested changes to this revision.Apr 25 2023, 15:05
Fabien added inline comments.
apps/alias-server/src/events.js
145 ↗(On Diff #39912)

Does processedAliases evaluates to false if there is no processed alias (empty array) ? That would be a bug

153 ↗(On Diff #39912)

I'm not sure that's the expected behavior. If your indexer restarted and is connecting older blocks that doesn't mean there is any error on the alias server

174 ↗(On Diff #39912)

You probably want to bail early if unprocessedAliasTxs.length == 0

184 ↗(On Diff #39912)

That's a very bad way to achieve the feature, as it requires you to fetch the whole DB for each alias.
You should rather loop through your unprocessed alias and see if you get a match in the db for each one. It is very likely that there are less alias to register than there are alias already registered. It's also more likely to not saturate the db cache.

187 ↗(On Diff #39912)

Same here, just bail out if there is nothing to do

196 ↗(On Diff #39912)

return false

201 ↗(On Diff #39912)

That's wrong. If there is no alias in the block you should still update the server state to that block.

226 ↗(On Diff #39912)

So this should really be 2 functions:

  1. A function to register the new aliases, that simply returns true if it succeeds (even if there is nothing to do) and false in case of a failue
  2. A function to update the server state, only called if the above returned true.
apps/alias-server/test/aliasTests.js
199 ↗(On Diff #39912)

That's unrelated to this diff

This revision now requires changes to proceed.Apr 25 2023, 15:05
bytesofman marked an inline comment as done.

need to refactor to not get the whole db on every block

apps/alias-server/src/events.js
145 ↗(On Diff #39912)

No -- a unit test has been added for this in this diff (see change to dbTests.js)

It evaluates to []

153 ↗(On Diff #39912)

In this case, processedBlockheight will be 0

We probably don't need this check. I'm not sure how this situation would happen.

184 ↗(On Diff #39912)

good point. this will take a bit of refactoring.

201 ↗(On Diff #39912)

That's what's happening here (since we don't bail above if no aliases, at least, not with the current logic).

confirmedUnprocessedTxs are any confirmed txs at the registration address not yet seen by the server, alias or not.

So at this point -- "even if we found no aliases, if we did have previously unseen confirmed txs, update the server state to reflect this"

The return on line 196 handles the case of "we found aliases, but could not add them to the db for some reason". In this case, we do not want to update the server state, because the app still needs to process those aliases from the last server state.

apps/alias-server/test/aliasTests.js
199 ↗(On Diff #39912)

This was added to make sure startup case (no valid aliases in db) would not crash the app

refactoring getValidAliasRegistrations to check individual db calls

Fabien requested changes to this revision.Apr 26 2023, 09:30
Fabien added inline comments.
apps/alias-server/src/db.js
173 ↗(On Diff #39943)

I don't think this makes sense and this is a very brittle strategy.
I tried to look a bit at how mongo deals with insert errors but the doc is more or less providing a stupid simple working example only, which is not very helpful;

So here is the issue:

  1. For each aliases: a. You check if it is in the db or in the list of aliases to be registered b. If not already there, you add it to the list of aliases to be registered
  2. You insert the list of aliases to be registered to the db.

This is brittle because it assumes that the state doesn't change between 1. and 2. AFAICT that's the case due to the reentrency check but that will become less obvious after 0-conf tx is available. Also I have zero idea what happens if 2. fails and how to recover from that.

IMO this is a better solution:

  1. For each aliases, attempt to insert it into the db

That's it. Because the aliases are ordered, you will insert the correct one and duplicates will error out. You don't even need the local list of aliases to be registered to remove the duplicates from the supplied list. If needed, you can even return the list of success/failures for later use or tests.

apps/alias-server/src/events.js
145 ↗(On Diff #39912)

TIL that's super confusing. So in js if([]) is true, but if([] == false) is also true because then it converts to a string, and an empty string is (as opposed to empty array or object) false.

For clarity I suggest you use if (processedAliases === false) here.

153 ↗(On Diff #39912)

I mean chronik could reindex while the alias server keeps running. In this case processedBlockheight will not be 0 !

I think the check is good to avoid running the whole logic for each past block, the only question being whether it should return true of false since it's not a failure.
Also this raises the question of how do you handle reorgs.

apps/alias-server/test/aliasTests.js
199 ↗(On Diff #39912)

OK but that's not what the test do. It tests that getAliasStringsFromValidAliasTxs returns an empty array if there is no alias. That's great but that's not testing any of the added logic. This is purely a test case for getAliasStringsFromValidAliasTxs that was missed before

This revision now requires changes to proceed.Apr 26 2023, 09:30
bytesofman marked 2 inline comments as done.

Use new database logic for checking validity of an alias while adding to database

lint

apps/alias-server/src/db.js
173 ↗(On Diff #39943)

Good point, will do it this way

apps/alias-server/src/events.js
145 ↗(On Diff #39912)

good point. however, we don't need to do this step anymore with the new logic, so code is removed.

153 ↗(On Diff #39912)

I don't think we need a syntax difference in "exit loop because of error" versus "exit loop because waiting for the node to index"

These types of distinctions can be handled by admin notifications. In this case, we want to tell the admin if the loop exited because of some kind of error. But we don't need to tell the admin if the loop exits because it's waiting for the node to sync. (or mb we do, but specify it is due to re-indexing).

the return false is just useful for unit tests to show that the loop does bail.

apps/alias-server/test/aliasTests.js
199 ↗(On Diff #39912)

good point. moot now though as with the new db approach, we no longer need the getAliasStringsFromValidAliasTxs helper function

remove changes unrelated to this diff

Fabien requested changes to this revision.Apr 27 2023, 12:39
Fabien added inline comments.
apps/alias-server/src/alias.js
149 ↗(On Diff #40004)

The name is no longer representative of what the function do

160 ↗(On Diff #40004)

Layout nit

189 ↗(On Diff #40004)

unconfirmedBlockheight does not belong to config, because it's clearly not a config option.

Also this function should not check for the tx confirmation, this is the job for some other code. You even removed such txs before the call, so you know that already.

Let's keep this function simple: call it registerAliases, order then store each alias to the db, and return the added ones.
The sorting function already exists and should have its unit tests.
This function can be added separately with its own tests.
Then you can wrap around the logic for actually handling the registrations upon block finalization.

191 ↗(On Diff #40004)

You don't need all these extra variables, just use if (await ddOneAliasToDb(db, thisAliasTx)) { and save 1 byte of RAM.

204 ↗(On Diff #40004)
  1. Why do you need to remove the _id key ? Is that a problem ?
  2. If it is, why do you need all the data and not only the alias (which is the unique key) ?
apps/alias-server/src/db.js
116 ↗(On Diff #40004)

Use a constant to avoid the magic number

apps/alias-server/src/events.js
166 ↗(On Diff #40004)

That's useless, you want to update the server state if there was no issue, even if there was no new alias

181 ↗(On Diff #40004)

And in fact there is no loop to exit

apps/alias-server/test/dbTests.js
116 ↗(On Diff #40004)

All of this (+the addOneAliasToDb function itself) could be its own diff

This revision now requires changes to proceed.Apr 27 2023, 12:39
bytesofman marked 6 inline comments as done.

Responding to review feedback

apps/alias-server/src/alias.js
204 ↗(On Diff #40004)

1 Why do you need to remove the _id key ? Is that a problem ?

  1. The _id key makes it difficult if not impossible to unit test, since the returned array of objects will always have unique _id keys.

2 If it is, why do you need all the data and not only the alias (which is the unique key) ?

  1. address, alias, blockheight, and txid are all info we want to store in the database about each tx. So, the unit tests should confirm that all this info is accurate.

We may not need all of this information in handleBlockConnected -- could omit blockheight (need the rest to make telegram announcement msgs). But, we might actually want the blockheight for the telegram msgs, and if we omit this info it is a significant complication to preparing unit test mocks.

apps/alias-server/test/dbTests.js
116 ↗(On Diff #40004)

Yes. These improvements were all learned through the review process on this diff. I could abandon and recreate. However, I think there is value in keeping the review history in one place.

apps/alias-server/src/alias.js
204 ↗(On Diff #40004)

OK

apps/alias-server/test/dbTests.js
116 ↗(On Diff #40004)

Don't recreate this diff: just extract these changes and rebase on top

apps/alias-server/test/dbTests.js
116 ↗(On Diff #40004)

This is a technical gap for me. I'm not sure how to do this.

The first diff will need to be just adding this function

The second diff will need to be all the logic changes and unit test changes in this diff

Fabien requested changes to this revision.Apr 27 2023, 14:38
Fabien added inline comments.
apps/alias-server/config.js
18 ↗(On Diff #40031)

that's not a config. It's a db error, it belongs in db.js

apps/alias-server/src/events.js
137 ↗(On Diff #40031)

Should be >=

172 ↗(On Diff #40031)

Why do you insist on this ? Just do processedBlockheight: tipHeight.

179 ↗(On Diff #40031)

That's still not a loop

This revision now requires changes to proceed.Apr 27 2023, 14:38
bytesofman marked 4 inline comments as done.

Responding to feedback

apps/alias-server/src/events.js
172 ↗(On Diff #40031)

lol that is much better. it's an oversight, not an insistence ;)

When I removed the gating, unit tests failed for cases with no confirmed txs

Fabien requested changes to this revision.Apr 27 2023, 15:20

It's getting there

apps/alias-server/test/eventsTests.js
233 ↗(On Diff #40035)

Check the equal case as well

This revision now requires changes to proceed.Apr 27 2023, 15:20

Add unit test covering handleBlockConnected called on same block height as server state

This revision is now accepted and ready to land.Apr 27 2023, 15:59