Details

Reviewers

emack

Group Reviewers

Restricted Project

Commits

rABC2d244d97c3eb: [token-server] Accept uploaded token icons

Summary

Provide all necessary handlers for migrating current token icon infra to token-server

Support for image uploads at new endpoint
Support in Cashtab for new server
Telegram bot for mod approval/rejection

The telegram bot features are not tested in the app, though I have extensively tested them manually and locally. I looked into adding tests for this but the process is not straightforward (emulate telegram, emulate clicking on rendered telegram messages). Since this feature is used only by the admin, imo it is ok to leave automatic testing out. Expected that we will always have some degree of possible manual work with token icons.

While this diff is ready for review now -- some steps I will need to complete before landing

Migrate existing token icons to token-server
Update CI deploy step with telegram bot info
Update token-server dir structure to support rejected icons
Need D15746 to land to support automated icon rejections
Land this one
Shut down legacy token icon server

Test Plan

npm test

Diff Detail

Repository

rABC Bitcoin ABC

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

bytesofman created this revision.Mar 20 2024, 16:00

Herald added a reviewer: Restricted Project. · View Herald TranscriptMar 20 2024, 16:00

Harbormaster completed remote builds in B27952: Diff 46325.Mar 20 2024, 16:05

Build Bitcoin ABC Diffs / Diff Testing (token-server-tests) failed.

Tail of the build log:


TSError: ⨯ Unable to compile TypeScript:
src/routes.ts(12,21): error TS2307: Cannot find module '../secrets' or its corresponding type declarations.

    at createTSError (/work/apps/token-server/node_modules/ts-node/src/index.ts:859:12)
    at reportTSError (/work/apps/token-server/node_modules/ts-node/src/index.ts:863:19)
    at getOutput (/work/apps/token-server/node_modules/ts-node/src/index.ts:1077:36)
    at Object.compile (/work/apps/token-server/node_modules/ts-node/src/index.ts:1433:41)
    at Module.m._compile (/work/apps/token-server/node_modules/ts-node/src/index.ts:1617:30)
    at module.exports (/usr/lib/node_modules/nyc/node_modules/default-require-extensions/js.js:7:9)
    at /usr/lib/node_modules/nyc/node_modules/append-transform/index.js:64:4
    at require.extensions.<computed> (/work/apps/token-server/node_modules/ts-node/src/index.ts:1621:12)
    at Object.<anonymous> (/usr/lib/node_modules/nyc/node_modules/append-transform/index.js:64:4)
    at Module.load (node:internal/modules/cjs/loader:1207:32)
    at Function.Module._load (node:internal/modules/cjs/loader:1023:12)
    at Module.require (node:internal/modules/cjs/loader:1235:19)
    at require (node:internal/modules/helpers:176:18)
    at Object.<anonymous> (/work/apps/token-server/test/routes.test.ts:9:1)
    at Module._compile (node:internal/modules/cjs/loader:1376:14)
    at Module.replacementCompile (/usr/lib/node_modules/nyc/node_modules/append-transform/index.js:60:13)
    at Module.m._compile (/work/apps/token-server/node_modules/ts-node/src/index.ts:1618:23)
    at module.exports (/usr/lib/node_modules/nyc/node_modules/default-require-extensions/js.js:7:9)
    at /usr/lib/node_modules/nyc/node_modules/append-transform/index.js:64:4
    at require.extensions.<computed> (/work/apps/token-server/node_modules/ts-node/src/index.ts:1621:12)
    at Object.<anonymous> (/usr/lib/node_modules/nyc/node_modules/append-transform/index.js:64:4)
    at Module.load (node:internal/modules/cjs/loader:1207:32)
    at Function.Module._load (node:internal/modules/cjs/loader:1023:12)
    at Module.require (node:internal/modules/cjs/loader:1235:19)
    at require (node:internal/modules/helpers:176:18)
    at Object.exports.requireOrImport (/work/apps/token-server/node_modules/mocha/lib/nodejs/esm-utils.js:53:16)
    at async Object.exports.loadFilesAsync (/work/apps/token-server/node_modules/mocha/lib/nodejs/esm-utils.js:100:20)
    at async singleRun (/work/apps/token-server/node_modules/mocha/lib/cli/run-helpers.js:125:3)
    at async Object.exports.handler (/work/apps/token-server/node_modules/mocha/lib/cli/run.js:370:5)
--------------------------|---------|----------|---------|---------|-------------------
File                      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
--------------------------|---------|----------|---------|---------|-------------------
All files                 |   22.58 |        0 |       0 |   20.33 |                   
 token-server             |     100 |      100 |     100 |     100 |                   
  config.ts               |     100 |      100 |     100 |     100 |                   
 token-server/src         |      20 |        0 |       0 |      20 |                   
  rewards.ts              |      20 |        0 |       0 |      20 | 38-60             
 token-server/src/chronik |      20 |        0 |       0 |   17.02 |                   
  clientHandler.ts        |   11.76 |        0 |       0 |    12.5 | 31-89             
  parse.ts                |   18.18 |        0 |       0 |   18.18 | 26-32,47-69,78-93 
  wsHandler.ts            |   36.36 |      100 |       0 |   22.22 | 23-33,41          
--------------------------|---------|----------|---------|---------|-------------------

##teamcity[blockOpened name='Code Coverage Summary']
##teamcity[buildStatisticValue key='CodeCoverageAbsBCovered' value='14']
##teamcity[buildStatisticValue key='CodeCoverageAbsBTotal' value='62']
##teamcity[buildStatisticValue key='CodeCoverageAbsRCovered' value='0']
##teamcity[buildStatisticValue key='CodeCoverageAbsRTotal' value='26']
##teamcity[buildStatisticValue key='CodeCoverageAbsMCovered' value='0']
##teamcity[buildStatisticValue key='CodeCoverageAbsMTotal' value='10']
##teamcity[buildStatisticValue key='CodeCoverageAbsLCovered' value='12']
##teamcity[buildStatisticValue key='CodeCoverageAbsLTotal' value='59']
##teamcity[blockClosed name='Code Coverage Summary']
mv: cannot stat 'test_results/token-server-junit.xml': No such file or directory
Build token-server-tests failed with exit code 1

fix build error, add secrets sample file, add test for cors

Harbormaster completed remote builds in B27953: Diff 46326.Mar 20 2024, 16:33

Build Bitcoin ABC Diffs / Diff Testing (token-server-tests) failed.

Tail of the build log:


TSError: ⨯ Unable to compile TypeScript:
src/routes.ts(12,21): error TS2307: Cannot find module '../secrets' or its corresponding type declarations.

    at createTSError (/work/apps/token-server/node_modules/ts-node/src/index.ts:859:12)
    at reportTSError (/work/apps/token-server/node_modules/ts-node/src/index.ts:863:19)
    at getOutput (/work/apps/token-server/node_modules/ts-node/src/index.ts:1077:36)
    at Object.compile (/work/apps/token-server/node_modules/ts-node/src/index.ts:1433:41)
    at Module.m._compile (/work/apps/token-server/node_modules/ts-node/src/index.ts:1617:30)
    at module.exports (/usr/lib/node_modules/nyc/node_modules/default-require-extensions/js.js:7:9)
    at /usr/lib/node_modules/nyc/node_modules/append-transform/index.js:64:4
    at require.extensions.<computed> (/work/apps/token-server/node_modules/ts-node/src/index.ts:1621:12)
    at Object.<anonymous> (/usr/lib/node_modules/nyc/node_modules/append-transform/index.js:64:4)
    at Module.load (node:internal/modules/cjs/loader:1207:32)
    at Function.Module._load (node:internal/modules/cjs/loader:1023:12)
    at Module.require (node:internal/modules/cjs/loader:1235:19)
    at require (node:internal/modules/helpers:176:18)
    at Object.<anonymous> (/work/apps/token-server/test/routes.test.ts:9:1)
    at Module._compile (node:internal/modules/cjs/loader:1376:14)
    at Module.replacementCompile (/usr/lib/node_modules/nyc/node_modules/append-transform/index.js:60:13)
    at Module.m._compile (/work/apps/token-server/node_modules/ts-node/src/index.ts:1618:23)
    at module.exports (/usr/lib/node_modules/nyc/node_modules/default-require-extensions/js.js:7:9)
    at /usr/lib/node_modules/nyc/node_modules/append-transform/index.js:64:4
    at require.extensions.<computed> (/work/apps/token-server/node_modules/ts-node/src/index.ts:1621:12)
    at Object.<anonymous> (/usr/lib/node_modules/nyc/node_modules/append-transform/index.js:64:4)
    at Module.load (node:internal/modules/cjs/loader:1207:32)
    at Function.Module._load (node:internal/modules/cjs/loader:1023:12)
    at Module.require (node:internal/modules/cjs/loader:1235:19)
    at require (node:internal/modules/helpers:176:18)
    at Object.exports.requireOrImport (/work/apps/token-server/node_modules/mocha/lib/nodejs/esm-utils.js:53:16)
    at async Object.exports.loadFilesAsync (/work/apps/token-server/node_modules/mocha/lib/nodejs/esm-utils.js:100:20)
    at async singleRun (/work/apps/token-server/node_modules/mocha/lib/cli/run-helpers.js:125:3)
    at async Object.exports.handler (/work/apps/token-server/node_modules/mocha/lib/cli/run.js:370:5)
--------------------------|---------|----------|---------|---------|-------------------
File                      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
--------------------------|---------|----------|---------|---------|-------------------
All files                 |   22.58 |        0 |       0 |   20.33 |                   
 token-server             |     100 |      100 |     100 |     100 |                   
  config.ts               |     100 |      100 |     100 |     100 |                   
 token-server/src         |      20 |        0 |       0 |      20 |                   
  rewards.ts              |      20 |        0 |       0 |      20 | 38-60             
 token-server/src/chronik |      20 |        0 |       0 |   17.02 |                   
  clientHandler.ts        |   11.76 |        0 |       0 |    12.5 | 31-89             
  parse.ts                |   18.18 |        0 |       0 |   18.18 | 26-32,47-69,78-93 
  wsHandler.ts            |   36.36 |      100 |       0 |   22.22 | 23-33,41          
--------------------------|---------|----------|---------|---------|-------------------

##teamcity[blockOpened name='Code Coverage Summary']
##teamcity[buildStatisticValue key='CodeCoverageAbsBCovered' value='14']
##teamcity[buildStatisticValue key='CodeCoverageAbsBTotal' value='62']
##teamcity[buildStatisticValue key='CodeCoverageAbsRCovered' value='0']
##teamcity[buildStatisticValue key='CodeCoverageAbsRTotal' value='26']
##teamcity[buildStatisticValue key='CodeCoverageAbsMCovered' value='0']
##teamcity[buildStatisticValue key='CodeCoverageAbsMTotal' value='10']
##teamcity[buildStatisticValue key='CodeCoverageAbsLCovered' value='12']
##teamcity[buildStatisticValue key='CodeCoverageAbsLTotal' value='59']
##teamcity[blockClosed name='Code Coverage Summary']
mv: cannot stat 'test_results/token-server-junit.xml': No such file or directory
Build token-server-tests failed with exit code 1

clean up logs, cashtab config to prod urls

handle secrets before tests

Harbormaster completed remote builds in B27954: Diff 46327.Mar 20 2024, 16:48

Build Bitcoin ABC Diffs / Diff Testing (token-server-tests) failed.

Tail of the build log:


TSError: ⨯ Unable to compile TypeScript:
src/routes.ts(12,21): error TS2307: Cannot find module '../secrets' or its corresponding type declarations.

    at createTSError (/work/apps/token-server/node_modules/ts-node/src/index.ts:859:12)
    at reportTSError (/work/apps/token-server/node_modules/ts-node/src/index.ts:863:19)
    at getOutput (/work/apps/token-server/node_modules/ts-node/src/index.ts:1077:36)
    at Object.compile (/work/apps/token-server/node_modules/ts-node/src/index.ts:1433:41)
    at Module.m._compile (/work/apps/token-server/node_modules/ts-node/src/index.ts:1617:30)
    at module.exports (/usr/lib/node_modules/nyc/node_modules/default-require-extensions/js.js:7:9)
    at /usr/lib/node_modules/nyc/node_modules/append-transform/index.js:64:4
    at require.extensions.<computed> (/work/apps/token-server/node_modules/ts-node/src/index.ts:1621:12)
    at Object.<anonymous> (/usr/lib/node_modules/nyc/node_modules/append-transform/index.js:64:4)
    at Module.load (node:internal/modules/cjs/loader:1207:32)
    at Function.Module._load (node:internal/modules/cjs/loader:1023:12)
    at Module.require (node:internal/modules/cjs/loader:1235:19)
    at require (node:internal/modules/helpers:176:18)
    at Object.<anonymous> (/work/apps/token-server/test/routes.test.ts:9:1)
    at Module._compile (node:internal/modules/cjs/loader:1376:14)
    at Module.replacementCompile (/usr/lib/node_modules/nyc/node_modules/append-transform/index.js:60:13)
    at Module.m._compile (/work/apps/token-server/node_modules/ts-node/src/index.ts:1618:23)
    at module.exports (/usr/lib/node_modules/nyc/node_modules/default-require-extensions/js.js:7:9)
    at /usr/lib/node_modules/nyc/node_modules/append-transform/index.js:64:4
    at require.extensions.<computed> (/work/apps/token-server/node_modules/ts-node/src/index.ts:1621:12)
    at Object.<anonymous> (/usr/lib/node_modules/nyc/node_modules/append-transform/index.js:64:4)
    at Module.load (node:internal/modules/cjs/loader:1207:32)
    at Function.Module._load (node:internal/modules/cjs/loader:1023:12)
    at Module.require (node:internal/modules/cjs/loader:1235:19)
    at require (node:internal/modules/helpers:176:18)
    at Object.exports.requireOrImport (/work/apps/token-server/node_modules/mocha/lib/nodejs/esm-utils.js:53:16)
    at async Object.exports.loadFilesAsync (/work/apps/token-server/node_modules/mocha/lib/nodejs/esm-utils.js:100:20)
    at async singleRun (/work/apps/token-server/node_modules/mocha/lib/cli/run-helpers.js:125:3)
    at async Object.exports.handler (/work/apps/token-server/node_modules/mocha/lib/cli/run.js:370:5)
--------------------------|---------|----------|---------|---------|-------------------
File                      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
--------------------------|---------|----------|---------|---------|-------------------
All files                 |   22.58 |        0 |       0 |   20.33 |                   
 token-server             |     100 |      100 |     100 |     100 |                   
  config.ts               |     100 |      100 |     100 |     100 |                   
 token-server/src         |      20 |        0 |       0 |      20 |                   
  rewards.ts              |      20 |        0 |       0 |      20 | 38-60             
 token-server/src/chronik |      20 |        0 |       0 |   17.02 |                   
  clientHandler.ts        |   11.76 |        0 |       0 |    12.5 | 31-89             
  parse.ts                |   18.18 |        0 |       0 |   18.18 | 26-32,47-69,78-93 
  wsHandler.ts            |   36.36 |      100 |       0 |   22.22 | 23-33,41          
--------------------------|---------|----------|---------|---------|-------------------

##teamcity[blockOpened name='Code Coverage Summary']
##teamcity[buildStatisticValue key='CodeCoverageAbsBCovered' value='14']
##teamcity[buildStatisticValue key='CodeCoverageAbsBTotal' value='62']
##teamcity[buildStatisticValue key='CodeCoverageAbsRCovered' value='0']
##teamcity[buildStatisticValue key='CodeCoverageAbsRTotal' value='26']
##teamcity[buildStatisticValue key='CodeCoverageAbsMCovered' value='0']
##teamcity[buildStatisticValue key='CodeCoverageAbsMTotal' value='10']
##teamcity[buildStatisticValue key='CodeCoverageAbsLCovered' value='12']
##teamcity[buildStatisticValue key='CodeCoverageAbsLTotal' value='59']
##teamcity[blockClosed name='Code Coverage Summary']
mv: cannot stat 'test_results/token-server-junit.xml': No such file or directory
Build token-server-tests failed with exit code 1

bytesofman edited the summary of this revision. (Show Details)Mar 20 2024, 16:53

Harbormaster completed remote builds in B27955: Diff 46328.Mar 20 2024, 16:54

bytesofman published this revision for review.Mar 20 2024, 16:58

bytesofman edited the summary of this revision. (Show Details)Mar 20 2024, 19:05

bytesofman edited the summary of this revision. (Show Details)

I'll ping you on tg to whitelist my IP. Given the changes it would be good if I can manually test the token creation process in case there's something not obvious to the int tests coverage.

apps/token-server/README.md
39 ↗	(On Diff #46328)	since this is off chain this could do with some guidance on an efficient process to backup the data on the server. I'm assuming you're doing this already with the current server?
apps/token-server/config.ts
37 ↗	(On Diff #46328)	Cashtab currently has no filter on this (tested with a 10MB file earlier) so may need to patch Cashtab to handle this size related rejection and render the appropriate notification. I think by default the user will see `'Not allowed by CORS'` which won't make much sense to them.
apps/token-server/src/routes.ts
161 ↗	(On Diff #46328)	I assume you'll port over our dev IPs as well so we can continue to hit this server from our localhost dev environments?
223 ↗	(On Diff #46328)	Cashtab also accepts jpegs in addition to png. Needs a jpg block here before the else statement. Your existing token server would have a mixture of jpg and png so wouldn't recommend carving out jpg.

This revision now requires changes to proceed.Mar 21 2024, 00:51

bytesofman requested review of this revision.Mar 21 2024, 02:16

bytesofman marked 4 inline comments as done.

bytesofman added inline comments.

apps/token-server/README.md
39 ↗	(On Diff #46328)	eh, I don't think the README is really the place for that. good sysadmin stuff is not done by the repo, i.e. this repo does not have auto backups. if the user wants them, they need to do it somehow. we have auto backups in place on the server.
apps/token-server/config.ts
37 ↗	(On Diff #46328)	Cashtab has some routines that compress and modify the file. the maxUploadSize here is unit tested in this diff.
apps/token-server/src/routes.ts
161 ↗	(On Diff #46328)	It's whitelisted by domain, not IP address (cashtab runs in the client, so we can't really whitelist everyone). It is probably still possible to hit this API as a developer. So, the whitelist is more about spam production than security.
223 ↗	(On Diff #46328)	cashtab converts jpg to png before hitting the endpoint -- existing token server is all png

serve images cross-origin

Harbormaster completed remote builds in B27962: Diff 46337.Mar 21 2024, 03:01

emack accepted this revision.Mar 21 2024, 03:50

This revision is now accepted and ready to land.Mar 21 2024, 03:50

bytesofman edited the summary of this revision. (Show Details)Mar 21 2024, 10:51

bytesofman edited the summary of this revision. (Show Details)Mar 21 2024, 11:18

Closed by commit rABC2d244d97c3eb: [token-server] Accept uploaded token icons (authored by bytesofman). · Explain WhyMar 21 2024, 11:22

This revision was automatically updated to reflect the committed changes.

bytesofman added a commit: rABC2d244d97c3eb: [token-server] Accept uploaded token icons.

[token-server] Accept uploaded token icons
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 46342

apps/token-server/.env

apps/token-server/.gitignore

apps/token-server/README.md

apps/token-server/config.ts

apps/token-server/index.ts

apps/token-server/package-lock.json

apps/token-server/package.json

apps/token-server/scripts/prepSecrets.ts

apps/token-server/scripts/sendTgIconNotice.ts

apps/token-server/secrets.sample.ts

apps/token-server/src/routes.ts

apps/token-server/src/telegram.ts

apps/token-server/test/routes.test.ts

cashtab/extension/public/manifest.json

cashtab/package-lock.json

cashtab/package.json

cashtab/src/components/Etokens/CreateTokenForm.js

cashtab/src/components/Etokens/TokenIcon.js

cashtab/src/components/Send/SendToken.js

cashtab/src/config/token.js

[token-server] Accept uploaded token iconsClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 46342

apps/token-server/.env

apps/token-server/.gitignore

apps/token-server/README.md

apps/token-server/config.ts

apps/token-server/index.ts

apps/token-server/package-lock.json

apps/token-server/package.json

apps/token-server/scripts/prepSecrets.ts

apps/token-server/scripts/sendTgIconNotice.ts

apps/token-server/secrets.sample.ts

apps/token-server/src/routes.ts

apps/token-server/src/telegram.ts

apps/token-server/test/routes.test.ts

cashtab/extension/public/manifest.json

cashtab/package-lock.json

cashtab/package.json

cashtab/src/components/Etokens/CreateTokenForm.js

cashtab/src/components/Etokens/TokenIcon.js

cashtab/src/components/Send/SendToken.js

cashtab/src/config/token.js

[token-server] Accept uploaded token icons
ClosedPublic
Actions

Revision Contents
Changeset List