Paths

Table of Contentst

Differential D16109

[ecash-lib] [CashTab] Split out wasm into separate modules
AbandonedPublicDraft
Actions

Authored by tobias_ruck on May 3 2024, 17:50.

Details

Reviewers

bytesofman

Group Reviewers

Restricted Project

Summary

Currently, integrating ecash-lib into CashTab is very difficult, due to the different environments between jest and webpack:

Jest can't handle ES modules, and either fails to import ecash-lib altogether (as it is a module), or, if we change it to be a CommonJS compatible, the generated JS from wasm-bindgen breaks it as import.meta.url can't be used, even if it's never executed.
Webpack can't find node:fs/promises, even though it's lazily loaded and never actually imported.

wasm-bindgen can build ffi modules that work for either NodeJS or web, but it can't build them for both. Since for CashTab, we need both in different occasions, one solution would be to split out the WASM part of ecash-lib, and allow users to specify which version should be used, by adding either ecash-lib-init-node for NodeJS or ecash-lib-init-web for web.

We use wasm-bindgen to build each version and then expose functions to mount either of them.

Test Plan

TBD

Diff Detail

Repository

rABC Bitcoin ABC

Branch

ecash-lib-fix-wasm

Lint

Lint Errors

Severity	Location	Code	Message
Error	cashtab/src/transactions/__tests__/index.test.js:84	ESLINT	jest/no-focused-tests

Unit

No Test Coverage

Build Status

Buildable 28886
Build 57310: Build Diff	ecash-lib-integration-tests · ecash-agora-integration-tests · ecash-lib-tests · cashtab-tests
Build 57309: arc lint + arc unit

Event Timeline

tobias_ruck created this revision.May 3 2024, 17:50

Herald added a reviewer: Restricted Project. · View Herald TranscriptMay 3 2024, 17:50

Harbormaster completed remote builds in B28882: Diff 47604.May 3 2024, 17:50

Build Bitcoin ABC Diffs / Diff Testing (ecash-lib-tests) failed.

Tail of the build log:

   Compiling wasm-bindgen v0.2.92
   Compiling abc-rust-lint v0.1.0 (/work/chronik/abc-rust-lint)
   Compiling generic-array v0.14.7
   Compiling quote v1.0.36
   Compiling syn v2.0.58
   Compiling secp256k1-sys-abc v0.4.1 (https://github.com/raipay/secp256k1-abc?rev=b23e742#b23e7421)
   Compiling crypto-common v0.1.6
   Compiling block-buffer v0.10.4
   Compiling digest v0.10.7
   Compiling ripemd v0.1.3
   Compiling sha2 v0.10.8
   Compiling wasm-bindgen-backend v0.2.92
   Compiling secp256k1-abc v0.20.3 (https://github.com/raipay/secp256k1-abc?rev=b23e742#b23e7421)
   Compiling thiserror-impl v1.0.58
   Compiling wasm-bindgen-macro-support v0.2.92
   Compiling wasm-bindgen-macro v0.2.92
   Compiling ecash-lib-wasm v0.1.0 (/work/modules/ecash-lib-wasm)
    Finished release-wasm [optimized] target(s) in 7.90s
Test depends on chronik-client. Building TypeScript...
/work/modules/chronik-client /work/modules/ecash-lib-wasm /work/abc-ci-builds/ecash-lib-tests

> chronik-client@0.26.2 prepublish
> npm run build


> chronik-client@0.26.2 build
> tsc


added 270 packages, and audited 271 packages in 6s

49 packages are looking for funding
  run `npm fund` for details

1 moderate severity vulnerability

To address all issues, run:
  npm audit fix

Run `npm audit` for details.
/work/modules/ecash-lib /work/modules/chronik-client /work/modules/ecash-lib-wasm /work/abc-ci-builds/ecash-lib-tests

added 363 packages, and audited 366 packages in 2s

60 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
CI configured to test build. Building...

> ecash-lib@0.1.0 build
> tsc && tsc -p ./tsconfig.build.json && cp -r ./src/ffi ./dist

src/ecc.test.ts(10,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
src/hash.test.ts(10,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
src/unsignedTx.test.ts(22,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
tests/alp.test.ts(28,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
tests/slp.test.ts(34,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
tests/txBuilder.test.ts(35,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
Build ecash-lib-tests failed with exit code 2

Build Bitcoin ABC Diffs / Diff Testing (cashtab-tests) failed.

Tail of the build log:

Installing mock-chronik-client dependencies...
/work/modules/mock-chronik-client /work/abc-ci-builds/cashtab-tests

added 236 packages, and audited 237 packages in 1s

35 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
/work/cashtab /work/modules/mock-chronik-client /work/abc-ci-builds/cashtab-tests

added 1678 packages, and audited 1682 packages in 28s

261 packages are looking for funding
  run `npm fund` for details

2 moderate severity vulnerabilities

To address all issues, run:
  npm audit fix

Run `npm audit` for details.

> cashtab@2.38.3 build
> node scripts/build.js

Creating an optimized production build...
Failed to compile.

Module not found: Error: Can't resolve 'ecash-lib' in '/work/cashtab/src'


Build cashtab-tests failed with exit code 1

Build Bitcoin ABC Diffs / Diff Testing (ecash-agora-integration-tests) failed.

Tail of the build log:

   Compiling thiserror v1.0.58
   Compiling abc-rust-lint v0.1.0 (/work/chronik/abc-rust-lint)
   Compiling generic-array v0.14.7
   Compiling quote v1.0.36
   Compiling syn v2.0.58
   Compiling secp256k1-sys-abc v0.4.1 (https://github.com/raipay/secp256k1-abc?rev=b23e742#b23e7421)
   Compiling block-buffer v0.10.4
   Compiling crypto-common v0.1.6
   Compiling digest v0.10.7
   Compiling sha2 v0.10.8
   Compiling ripemd v0.1.3
   Compiling wasm-bindgen-backend v0.2.92
   Compiling secp256k1-abc v0.20.3 (https://github.com/raipay/secp256k1-abc?rev=b23e742#b23e7421)
   Compiling thiserror-impl v1.0.58
   Compiling wasm-bindgen-macro-support v0.2.92
   Compiling wasm-bindgen-macro v0.2.92
   Compiling ecash-lib-wasm v0.1.0 (/work/modules/ecash-lib-wasm)
    Finished release-wasm [optimized] target(s) in 4.65s
Test depends on chronik-client. Building TypeScript...
/work/modules/chronik-client /work/modules/ecash-lib-wasm /work/abc-ci-builds/ecash-agora-integration-tests

> chronik-client@0.26.2 prepublish
> npm run build


> chronik-client@0.26.2 build
> tsc


added 270 packages, and audited 271 packages in 7s

49 packages are looking for funding
  run `npm fund` for details

1 moderate severity vulnerability

To address all issues, run:
  npm audit fix

Run `npm audit` for details.
Test depends on ecash-lib. Building TypeScript...
/work/modules/ecash-lib /work/modules/chronik-client /work/modules/ecash-lib-wasm /work/abc-ci-builds/ecash-agora-integration-tests

added 363 packages, and audited 366 packages in 2s

60 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

> ecash-lib@0.1.0 build
> tsc && tsc -p ./tsconfig.build.json && cp -r ./src/ffi ./dist

src/ecc.test.ts(10,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
src/hash.test.ts(10,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
src/unsignedTx.test.ts(22,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
tests/alp.test.ts(28,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
tests/slp.test.ts(34,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
tests/txBuilder.test.ts(35,26): error TS2307: Cannot find module 'ecash-lib-init-node' or its corresponding type declarations.
Build ecash-agora-integration-tests failed with exit code 2

Build Bitcoin ABC Diffs / Diff Testing (ecash-lib-integration-tests) failed.

Tail of the build log:

    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:315:7)
    at ModuleJob.run (node:internal/modules/esm/module_job:222:25)
    at async ModuleLoader.import (node:internal/modules/esm/loader:323:24)
    at async formattedImport (/work/modules/ecash-lib/node_modules/mocha/lib/nodejs/esm-utils.js:9:14)
    at async exports.requireOrImport (/work/modules/ecash-lib/node_modules/mocha/lib/nodejs/esm-utils.js:42:28)
    at async exports.loadFilesAsync (/work/modules/ecash-lib/node_modules/mocha/lib/nodejs/esm-utils.js:100:20)
    at async singleRun (/work/modules/ecash-lib/node_modules/mocha/lib/cli/run-helpers.js:125:3)
    at async exports.handler (/work/modules/ecash-lib/node_modules/mocha/lib/cli/run.js:370:5) {
  code: 'MODULE_NOT_FOUND',
  path: '/work/modules/ecash-lib/node_modules/ecash-lib-init-node/package.json',
  requestPath: 'ecash-lib-init-node'
}
---------------------|---------|----------|---------|---------|-------------------------------------
File                 | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s                   
---------------------|---------|----------|---------|---------|-------------------------------------
All files            |    42.4 |    31.79 |   26.92 |   43.05 |                                     
 ecash-lib           |       0 |        0 |       0 |       0 |                                     
  eslint.config.js   |       0 |        0 |       0 |       0 |                                     
 ecash-lib/src       |   52.01 |    36.12 |   26.66 |   52.84 |                                     
  ecc.ts             |   42.85 |       80 |      20 |   42.85 | 23-31,36                            
  hash.ts            |   55.55 |       80 |      20 |   55.55 | 14,17,20,24                         
  index.ts           |     100 |       80 |     100 |     100 | 1                                   
  init.ts            |   66.66 |       80 |      50 |      80 | 15                                  
  op.ts              |   49.15 |    55.55 |      80 |   49.15 | ...,82-86,89,92,104,107,109,117-122 
  opcode.ts          |     100 |       80 |     100 |     100 | 1                                   
  script.ts          |   42.22 |    22.72 |   23.07 |   39.53 | 33-35,55-138,147-158                
  sigHashType.ts     |      75 |    37.03 |   71.42 |      75 | 26-58                               
  tx.ts              |   33.33 |    17.39 |      10 |      35 | ...,124-128,133-135,140-141,146,166 
  txBuilder.ts       |    14.6 |    11.42 |    8.33 |   15.29 | 74-204,211,219-222,231-235,241-245  
  unsignedTx.ts      |   15.47 |     12.5 |    7.14 |   16.04 | ...,190-201,208-212,216,222-223,228 
 ecash-lib/src/io    |   38.51 |       45 |   48.57 |   38.46 |                                     
  bytes.ts           |     7.4 |    57.14 |    12.5 |     7.4 | 13-64                               
  hex.ts             |   58.97 |    66.66 |      40 |   61.76 | 33-37,41-45,50,58,66-68             
  int.ts             |     100 |       60 |     100 |     100 | 1                                   
  str.ts             |     100 |       80 |     100 |     100 | 1                                   
  varsize.ts         |      12 |    19.04 |   33.33 |      12 | 14-24,35-47                         
  writer.ts          |     100 |       60 |     100 |     100 | 1                                   
  writerbytes.ts     |   43.33 |    34.78 |    62.5 |   43.33 | 33,42-67,79                         
  writerlength.ts    |    62.5 |       80 |   57.14 |    62.5 | 26-36                               
 ecash-lib/src/test  |    17.8 |        8 |    6.25 |   19.11 |                                     
  testRunner.ts      |    17.8 |        8 |    6.25 |   19.11 | 37-190                              
 ecash-lib/src/token |   27.37 |    20.77 |   13.33 |   27.52 |                                     
  alp.ts             |   18.29 |    21.05 |    8.33 |   18.29 | ...,110-123,127-131,135-137,141-145 
  common.ts          |     100 |       80 |     100 |     100 | 1                                   
  empp.ts            |   46.15 |    44.44 |      25 |      50 | 11-14,22-28                         
  slp.ts             |    28.2 |     9.09 |    7.69 |    28.2 | ...,166-167,174-178,185-197,201-211 
---------------------|---------|----------|---------|---------|-------------------------------------

##teamcity[blockOpened name='Code Coverage Summary']
##teamcity[buildStatisticValue key='CodeCoverageAbsBCovered' value='385']
##teamcity[buildStatisticValue key='CodeCoverageAbsBTotal' value='908']
##teamcity[buildStatisticValue key='CodeCoverageAbsRCovered' value='138']
##teamcity[buildStatisticValue key='CodeCoverageAbsRTotal' value='434']
##teamcity[buildStatisticValue key='CodeCoverageAbsMCovered' value='42']
##teamcity[buildStatisticValue key='CodeCoverageAbsMTotal' value='156']
##teamcity[buildStatisticValue key='CodeCoverageAbsLCovered' value='381']
##teamcity[buildStatisticValue key='CodeCoverageAbsLTotal' value='885']
##teamcity[blockClosed name='Code Coverage Summary']
mv: cannot stat 'test_results/ecash-lib-integration-tests-junit.xml': No such file or directory
Build ecash-lib-integration-tests failed with exit code 1

seems like there should be a way to do this without introducing new libraries -- i.e. keeping the node / web init distinctions within ecash-lib.

There is a way to build/serve different code for node vs web users in a module -- for example see ecashaddrjs package.json file:

"main": "src/cashaddr.js",
"browser": "dist/cashaddrjs.min.js",

so web gets the webpack-built cashaddrjs.min.js but node gets the raw source.

modules/ecash-lib-wasm/build-wasm.sh
23 ↗	(On Diff #47604)	will it not work if we just target node? webpack is already doing some transpiling for node modules to work in web, for example use of `Buffer`

move back into ecash-lib

Harbormaster completed remote builds in B28885: Diff 47607.May 3 2024, 20:56

Build Bitcoin ABC Diffs / Diff Testing (cashtab-tests) failed.

Tail of the build log:

Installing mock-chronik-client dependencies...
/work/modules/mock-chronik-client /work/abc-ci-builds/cashtab-tests

added 236 packages, and audited 237 packages in 1s

35 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
/work/cashtab /work/modules/mock-chronik-client /work/abc-ci-builds/cashtab-tests

added 1676 packages, and audited 1680 packages in 28s

261 packages are looking for funding
  run `npm fund` for details

2 moderate severity vulnerabilities

To address all issues, run:
  npm audit fix

Run `npm audit` for details.

> cashtab@2.38.3 build
> node scripts/build.js

Creating an optimized production build...
Failed to compile.

Module not found: Error: Can't resolve 'ecash-lib/dist/initWeb' in '/work/cashtab/src'


Build cashtab-tests failed with exit code 1

use "main" and "browser" to build two different variants

Harbormaster completed remote builds in B28886: Diff 47608.May 3 2024, 21:25

Build Bitcoin ABC Diffs / Diff Testing (cashtab-tests) failed.

Tail of the build log:

Installing mock-chronik-client dependencies...
/work/modules/mock-chronik-client /work/abc-ci-builds/cashtab-tests

added 236 packages, and audited 237 packages in 1s

35 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
/work/cashtab /work/modules/mock-chronik-client /work/abc-ci-builds/cashtab-tests

added 1676 packages, and audited 1680 packages in 28s

261 packages are looking for funding
  run `npm fund` for details

2 moderate severity vulnerabilities

To address all issues, run:
  npm audit fix

Run `npm audit` for details.

> cashtab@2.38.3 build
> node scripts/build.js

Creating an optimized production build...
Failed to compile.

Module not found: Error: Can't resolve 'ecash-lib' in '/work/cashtab/src'


Build cashtab-tests failed with exit code 1

tobias_ruck added inline comments.May 3 2024, 21:35

modules/ecash-lib-wasm/build-wasm.sh
23 ↗	(On Diff #47604)	No because `node` it uses `node` functions to read files from the disk to load the wasm, which isn't just a simple transpilation thing