Page MenuHomePhabricator
Differential D16109

[ecash-lib] [CashTab] Split out wasm into separate modules
AbandonedPublicDraft
Actions

Authored by tobias_ruck on Fri, May 3, 17:50.

Details

Reviewers
bytesofman
Group Reviewers
Restricted Project
Summary

Currently, integrating ecash-lib into CashTab is very difficult, due to the different environments between jest and webpack:

  • Jest can't handle ES modules, and either fails to import ecash-lib altogether (as it is a module), or, if we change it to be a CommonJS compatible, the generated JS from wasm-bindgen breaks it as import.meta.url can't be used, even if it's never executed.
  • Webpack can't find node:fs/promises, even though it's lazily loaded and never actually imported.

wasm-bindgen can build ffi modules that work for either NodeJS or web, but it can't build them for both. Since for CashTab, we need both in different occasions, one solution would be to split out the WASM part of ecash-lib, and allow users to specify which version should be used, by adding either ecash-lib-init-node for NodeJS or ecash-lib-init-web for web.

We use wasm-bindgen to build each version and then expose functions to mount either of them.

Test Plan

TBD

Diff Detail

Repository
rABC Bitcoin ABC
Branch
ecash-lib-fix-wasm
Lint
Lint Errors
SeverityLocationCodeMessage
Errorcashtab/src/transactions/__tests__/index.test.js:84ESLINTjest/no-focused-tests
Unit
No Test Coverage
Build Status
Buildable 28886
Build 57310: Build Diffecash-lib-integration-tests · ecash-agora-integration-tests · ecash-lib-tests · cashtab-tests
Build 57309: arc lint + arc unit

Event Timeline

tobias_ruck created this revision.Fri, May 3, 17:50
Herald added a reviewer: Restricted Project. · View Herald TranscriptFri, May 3, 17:50
Harbormaster completed remote builds in B28882: Diff 47604.Fri, May 3, 17:50
teamcity added a comment.Fri, May 3, 17:51
Build Bitcoin ABC Diffs / Diff Testing (ecash-lib-tests) failed.

Tail of the build log:

   Compiling wasm-bindgen v0.2.92
   Compiling abc-rust-lint v0.1.0 (/work/chronik/abc-rust-lint)
   Compiling generic-array v0.14.7
   Compiling quote v1.0.36
   Compiling syn v2.0.58
   Compiling secp256k1-sys-abc v0.4.1 (https://github.com/raipay/secp256k1-abc?rev=b23e742#b23e7421)
   Compiling crypto-common v0.1.6
   Compiling block-buffer v0.10.4
   Compiling digest v0.10.7
   Compiling ripemd v0.1.3
   Compiling sha2 v0.10.8
   Compiling wasm-bindgen-backend v0.2.92
   Compiling secp256k1-abc v0.20.3 (https://github.com/raipay/secp256k1-abc?rev=b23e742#b23e7421)
   Compiling thiserror-impl v1.0.58
   Compiling wasm-bindgen-macro-support v0.2.92
   Compiling wasm-bindgen-macro v0.2.92
   Compiling ecash-lib-wasm v0.1.0 (/work/modules/ecash-lib-wasm)
    Finished release-wasm [optimized] target(s) in 7.90s
Test depends on chronik-client. Building TypeScript...
/work/modules/chronik-client /work/modules/ecash-lib-wasm /work/abc-ci-builds/ecash-lib-tests

> chronik-client@0.26.2 prepublish
> npm run build


> chronik-client@0.26.2 build
> tsc


added 270 packages, and audited 271 packages in 6s

49 packages are looking for funding
  run `npm fund` for details

1 moderate severity vulnerability

To address all issues, run:
  npm audit fix

Run `npm audit` for details.
/work/modules/ecash-lib /work/modules/chronik-client /work/modules/ecash-lib-wasm /work/abc-ci-builds/ecash-lib-tests

added 363 packages, and audited 366 packages in 2s

60 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
CI configured to test build. Building...

> ecash-lib@0.1.0 build
> tsc && tsc -p ./tsconfig.build.json && cp -r ./src/ffi ./dist

src/ecc.test.ts(10,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
src/hash.test.ts(10,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
src/unsignedTx.test.ts(22,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
tests/alp.test.ts(28,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
tests/slp.test.ts(34,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
tests/txBuilder.test.ts(35,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
Build ecash-lib-tests failed with exit code 2
teamcity added a comment.Fri, May 3, 17:52
Build Bitcoin ABC Diffs / Diff Testing (cashtab-tests) failed.

Tail of the build log:

Installing mock-chronik-client dependencies...
/work/modules/mock-chronik-client /work/abc-ci-builds/cashtab-tests

added 236 packages, and audited 237 packages in 1s

35 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
/work/cashtab /work/modules/mock-chronik-client /work/abc-ci-builds/cashtab-tests

added 1678 packages, and audited 1682 packages in 28s

261 packages are looking for funding
  run `npm fund` for details

2 moderate severity vulnerabilities

To address all issues, run:
  npm audit fix

Run `npm audit` for details.

> cashtab@2.38.3 build
> node scripts/build.js

Creating an optimized production build...
Failed to compile.

Module not found: Error: Can't resolve 'ecash-lib' in '/work/cashtab/src'


Build cashtab-tests failed with exit code 1
teamcity added a comment.Fri, May 3, 17:54
Build Bitcoin ABC Diffs / Diff Testing (ecash-agora-integration-tests) failed.

Tail of the build log:

   Compiling thiserror v1.0.58
   Compiling abc-rust-lint v0.1.0 (/work/chronik/abc-rust-lint)
   Compiling generic-array v0.14.7
   Compiling quote v1.0.36
   Compiling syn v2.0.58
   Compiling secp256k1-sys-abc v0.4.1 (https://github.com/raipay/secp256k1-abc?rev=b23e742#b23e7421)
   Compiling block-buffer v0.10.4
   Compiling crypto-common v0.1.6
   Compiling digest v0.10.7
   Compiling sha2 v0.10.8
   Compiling ripemd v0.1.3
   Compiling wasm-bindgen-backend v0.2.92
   Compiling secp256k1-abc v0.20.3 (https://github.com/raipay/secp256k1-abc?rev=b23e742#b23e7421)
   Compiling thiserror-impl v1.0.58
   Compiling wasm-bindgen-macro-support v0.2.92
   Compiling wasm-bindgen-macro v0.2.92
   Compiling ecash-lib-wasm v0.1.0 (/work/modules/ecash-lib-wasm)
    Finished release-wasm [optimized] target(s) in 4.65s
Test depends on chronik-client. Building TypeScript...
/work/modules/chronik-client /work/modules/ecash-lib-wasm /work/abc-ci-builds/ecash-agora-integration-tests

> chronik-client@0.26.2 prepublish
> npm run build


> chronik-client@0.26.2 build
> tsc


added 270 packages, and audited 271 packages in 7s

49 packages are looking for funding
  run `npm fund` for details

1 moderate severity vulnerability

To address all issues, run:
  npm audit fix

Run `npm audit` for details.
Test depends on ecash-lib. Building TypeScript...
/work/modules/ecash-lib /work/modules/chronik-client /work/modules/ecash-lib-wasm /work/abc-ci-builds/ecash-agora-integration-tests

added 363 packages, and audited 366 packages in 2s

60 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

> ecash-lib@0.1.0 build
> tsc && tsc -p ./tsconfig.build.json && cp -r ./src/ffi ./dist

src/ecc.test.ts(10,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
src/hash.test.ts(10,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
src/unsignedTx.test.ts(22,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
tests/alp.test.ts(28,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
tests/slp.test.ts(34,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
tests/txBuilder.test.ts(35,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
Build ecash-agora-integration-tests failed with exit code 2
teamcity added a comment.Fri, May 3, 17:55
Build Bitcoin ABC Diffs / Diff Testing (ecash-lib-integration-tests) failed.

Tail of the build log:

    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:315:7)
    at ModuleJob.run (node:internal/modules/esm/module_job:222:25)
    at async ModuleLoader.import (node:internal/modules/esm/loader:323:24)
    at async formattedImport (/work/modules/ecash-lib/node_modules/mocha/lib/nodejs/esm-utils.js:9:14)
    at async exports.requireOrImport (/work/modules/ecash-lib/node_modules/mocha/lib/nodejs/esm-utils.js:42:28)
    at async exports.loadFilesAsync (/work/modules/ecash-lib/node_modules/mocha/lib/nodejs/esm-utils.js:100:20)
    at async singleRun (/work/modules/ecash-lib/node_modules/mocha/lib/cli/run-helpers.js:125:3)
    at async exports.handler (/work/modules/ecash-lib/node_modules/mocha/lib/cli/run.js:370:5) {
  code: 'MODULE_NOT_FOUND',
  path: '/work/modules/ecash-lib/node_modules/ecash-lib-init-node/package.json',
  requestPath: 'ecash-lib-init-node'
}
---------------------|---------|----------|---------|---------|-------------------------------------
File                 | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s                   
---------------------|---------|----------|---------|---------|-------------------------------------
All files            |    42.4 |    31.79 |   26.92 |   43.05 |                                     
 ecash-lib           |       0 |        0 |       0 |       0 |                                     
  eslint.config.js   |       0 |        0 |       0 |       0 |                                     
 ecash-lib/src       |   52.01 |    36.12 |   26.66 |   52.84 |                                     
  ecc.ts             |   42.85 |       80 |      20 |   42.85 | 23-31,36                            
  hash.ts            |   55.55 |       80 |      20 |   55.55 | 14,17,20,24                         
  index.ts           |     100 |       80 |     100 |     100 | 1                                   
  init.ts            |   66.66 |       80 |      50 |      80 | 15                                  
  op.ts              |   49.15 |    55.55 |      80 |   49.15 | ...,82-86,89,92,104,107,109,117-122 
  opcode.ts          |     100 |       80 |     100 |     100 | 1                                   
  script.ts          |   42.22 |    22.72 |   23.07 |   39.53 | 33-35,55-138,147-158                
  sigHashType.ts     |      75 |    37.03 |   71.42 |      75 | 26-58                               
  tx.ts              |   33.33 |    17.39 |      10 |      35 | ...,124-128,133-135,140-141,146,166 
  txBuilder.ts       |    14.6 |    11.42 |    8.33 |   15.29 | 74-204,211,219-222,231-235,241-245  
  unsignedTx.ts      |   15.47 |     12.5 |    7.14 |   16.04 | ...,190-201,208-212,216,222-223,228 
 ecash-lib/src/io    |   38.51 |       45 |   48.57 |   38.46 |                                     
  bytes.ts           |     7.4 |    57.14 |    12.5 |     7.4 | 13-64                               
  hex.ts             |   58.97 |    66.66 |      40 |   61.76 | 33-37,41-45,50,58,66-68             
  int.ts             |     100 |       60 |     100 |     100 | 1                                   
  str.ts             |     100 |       80 |     100 |     100 | 1                                   
  varsize.ts         |      12 |    19.04 |   33.33 |      12 | 14-24,35-47                         
  writer.ts          |     100 |       60 |     100 |     100 | 1                                   
  writerbytes.ts     |   43.33 |    34.78 |    62.5 |   43.33 | 33,42-67,79                         
  writerlength.ts    |    62.5 |       80 |   57.14 |    62.5 | 26-36                               
 ecash-lib/src/test  |    17.8 |        8 |    6.25 |   19.11 |                                     
  testRunner.ts      |    17.8 |        8 |    6.25 |   19.11 | 37-190                              
 ecash-lib/src/token |   27.37 |    20.77 |   13.33 |   27.52 |                                     
  alp.ts             |   18.29 |    21.05 |    8.33 |   18.29 | ...,110-123,127-131,135-137,141-145 
  common.ts          |     100 |       80 |     100 |     100 | 1                                   
  empp.ts            |   46.15 |    44.44 |      25 |      50 | 11-14,22-28                         
  slp.ts             |    28.2 |     9.09 |    7.69 |    28.2 | ...,166-167,174-178,185-197,201-211 
---------------------|---------|----------|---------|---------|-------------------------------------

##teamcity[blockOpened name='Code Coverage Summary']
##teamcity[buildStatisticValue key='CodeCoverageAbsBCovered' value='385']
##teamcity[buildStatisticValue key='CodeCoverageAbsBTotal' value='908']
##teamcity[buildStatisticValue key='CodeCoverageAbsRCovered' value='138']
##teamcity[buildStatisticValue key='CodeCoverageAbsRTotal' value='434']
##teamcity[buildStatisticValue key='CodeCoverageAbsMCovered' value='42']
##teamcity[buildStatisticValue key='CodeCoverageAbsMTotal' value='156']
##teamcity[buildStatisticValue key='CodeCoverageAbsLCovered' value='381']
##teamcity[buildStatisticValue key='CodeCoverageAbsLTotal' value='885']
##teamcity[blockClosed name='Code Coverage Summary']
mv: cannot stat 'test_results/ecash-lib-integration-tests-junit.xml': No such file or directory
Build ecash-lib-integration-tests failed with exit code 1
bytesofman added a comment.Fri, May 3, 18:14

seems like there should be a way to do this without introducing new libraries -- i.e. keeping the node / web init distinctions within ecash-lib.

There is a way to build/serve different code for node vs web users in a module -- for example see ecashaddrjs package.json file:

"main": "src/cashaddr.js",
"browser": "dist/cashaddrjs.min.js",

so web gets the webpack-built cashaddrjs.min.js but node gets the raw source.

modules/ecash-lib-wasm/build-wasm.sh
23 ↗(On Diff #47604)

will it not work if we just target node?

webpack is already doing some transpiling for node modules to work in web, for example use of Buffer

tobias_ruck updated this revision to Diff 47607.Fri, May 3, 20:56

move back into ecash-lib

Harbormaster completed remote builds in B28885: Diff 47607.Fri, May 3, 20:56
teamcity added a comment.Fri, May 3, 20:59
Build Bitcoin ABC Diffs / Diff Testing (cashtab-tests) failed.

Tail of the build log:

Installing mock-chronik-client dependencies...
/work/modules/mock-chronik-client /work/abc-ci-builds/cashtab-tests

added 236 packages, and audited 237 packages in 1s

35 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
/work/cashtab /work/modules/mock-chronik-client /work/abc-ci-builds/cashtab-tests

added 1676 packages, and audited 1680 packages in 28s

261 packages are looking for funding
  run `npm fund` for details

2 moderate severity vulnerabilities

To address all issues, run:
  npm audit fix

Run `npm audit` for details.

> cashtab@2.38.3 build
> node scripts/build.js

Creating an optimized production build...
Failed to compile.

Module not found: Error: Can't resolve 'ecash-lib/dist/initWeb' in '/work/cashtab/src'


Build cashtab-tests failed with exit code 1
tobias_ruck updated this revision to Diff 47608.Fri, May 3, 21:25

use "main" and "browser" to build two different variants

Harbormaster completed remote builds in B28886: Diff 47608.Fri, May 3, 21:25
teamcity added a comment.Fri, May 3, 21:28
Build Bitcoin ABC Diffs / Diff Testing (cashtab-tests) failed.

Tail of the build log:

Installing mock-chronik-client dependencies...
/work/modules/mock-chronik-client /work/abc-ci-builds/cashtab-tests

added 236 packages, and audited 237 packages in 1s

35 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
/work/cashtab /work/modules/mock-chronik-client /work/abc-ci-builds/cashtab-tests

added 1676 packages, and audited 1680 packages in 28s

261 packages are looking for funding
  run `npm fund` for details

2 moderate severity vulnerabilities

To address all issues, run:
  npm audit fix

Run `npm audit` for details.

> cashtab@2.38.3 build
> node scripts/build.js

Creating an optimized production build...
Failed to compile.

Module not found: Error: Can't resolve 'ecash-lib' in '/work/cashtab/src'


Build cashtab-tests failed with exit code 1
tobias_ruck added inline comments.Fri, May 3, 21:35
modules/ecash-lib-wasm/build-wasm.sh
23 ↗(On Diff #47604)

No because node it uses node functions to read files from the disk to load the wasm, which isn't just a simple transpilation thing

tobias_ruck mentioned this in D16107: [ecash-lib] "module": "CommonJS" + separate "main" and "browser".Sat, May 4, 14:45
tobias_ruck mentioned this in rABC8f74e2ff94d2: [ecash-lib] "module": "CommonJS" + separate "main" and "browser".Sat, May 4, 22:05
tobias_ruck abandoned this revision.Fri, May 10, 23:33

has since been fixed in D16107

Revision Contents
Changeset List

PathSize
cashtab/
config/
jest/
4 lines
2 lines
257 lines
1 line
src/
4 lines
transactions/
__tests__/
3 lines
21 lines
modules/
ecash-agora/
25 lines
src/
5 lines
tests/
5 lines
ecash-lib-wasm/
9 lines
ecash-lib/
2 lines
11 lines
4 lines
src/
7 lines
10 lines
2 lines
24 lines
2 lines
2 lines
18 lines
17 lines
test/
6 lines
3 lines
tests/
20 lines
24 lines
38 lines

Diff 47608

View Options

cashtab/config/jest/custom-environment.js

Loading...
View Options

cashtab/config/webpack.config.js

Loading...
View Options

cashtab/package-lock.json

Loading...
View Options

cashtab/package.json

Loading...
View Options

cashtab/src/index.js

Loading...
View Options

cashtab/src/transactions/__tests__/index.test.js

Loading...
View Options

cashtab/src/transactions/index.js

Loading...
View Options

modules/ecash-agora/package-lock.json

Loading...
View Options

modules/ecash-agora/src/adSlp.ts

Loading...
View Options

modules/ecash-agora/tests/oneshot.test.ts

Loading...
View Options

modules/ecash-lib-wasm/build-wasm.sh

Loading...
View Options

modules/ecash-lib/.gitignore

Loading...
View Options

modules/ecash-lib/package-lock.json

Loading...
View Options

modules/ecash-lib/package.json

Loading...
View Options

modules/ecash-lib/src/ecc.test.ts

Loading...
View Options

modules/ecash-lib/src/ecc.ts

Loading...
View Options

modules/ecash-lib/src/hash.test.ts

Loading...
View Options

modules/ecash-lib/src/hash.ts

Loading...
View Options

modules/ecash-lib/src/index.ts

Loading...
View Options

modules/ecash-lib/src/indexBrowser.ts

Loading...
View Options

modules/ecash-lib/src/indexNodeJs.ts

Loading...
View Options

modules/ecash-lib/src/init.ts

Loading...
View Options

modules/ecash-lib/src/initBrowser.ts

Loading...
View Options

modules/ecash-lib/src/initNodeJs.ts

Loading...
View Options

modules/ecash-lib/src/test/testRunner.ts

Loading...
View Options

modules/ecash-lib/src/unsignedTx.test.ts

Loading...
View Options

modules/ecash-lib/tests/alp.test.ts

Loading...
View Options

modules/ecash-lib/tests/slp.test.ts

Loading...
View Options

modules/ecash-lib/tests/txBuilder.test.ts

Loading...