Web3 4x Upgrade

[nazarhussain]

[cds-amal]

Ooh nice! Thanks, @nazarhussain 🙇

[eggplantzzz]

Yeah thanks @nazarhussain! Just a note, I saw that you skipped the Vyper compiler tests. If you want to be able to run those successfully you will have to install the Vyper compiler locally. https://vyper.readthedocs.io/en/stable/installing-vyper.html I think I had to use the instructions using pip.

[nazarhussain]

@eggplantzzz There are few tests which I am still working on. So why I kept the PR in draft mode for now. As soon all tests passes I will mark it ready.

[cds-amal]

@nazarhussain the yarncheck failure indicates yarn.lock is out of sync with the various package.jsons. This can be resolved by doing yarn bootstrap and committing the updated yarn.lock

[nazarhussain]

@cds-amal After running the yarn bootstrap I am getting following error while building packages. It was building fine before. Any idea?

~/Hub/Chainsafe/Web3/projects/truffle/packages/codec on nh/web3-4x-upgrade *1 !2 .............................................................................................................................................................................................. took 14s at 22:28:38
> yarn build
yarn run v1.22.19
$ ttsc
/Users/nazar/Hub/Chainsafe/Web3/projects/truffle/node_modules/typescript/lib/typescript.js:117612
                throw e;
                ^

TypeError: Cannot read properties of undefined (reading 'impliedNodeFormat')
    at Object.getModeForUsageLocation (/Users/nazar/Hub/Chainsafe/Web3/projects/truffle/node_modules/typescript/lib/typescript.js:116234:18)
    at resolveExternalModule (/Users/nazar/Hub/Chainsafe/Web3/projects/truffle/node_modules/typescript/lib/typescript.js:51178:90)
    at resolveExternalModuleNameWorker (/Users/nazar/Hub/Chainsafe/Web3/projects/truffle/node_modules/typescript/lib/typescript.js:51154:19)
    at getExternalModuleFileFromDeclaration (/Users/nazar/Hub/Chainsafe/Web3/projects/truffle/node_modules/typescript/lib/typescript.js:87472:32)
    at Object.getExternalModuleFileFromDeclaration (/Users/nazar/Hub/Chainsafe/Web3/projects/truffle/node_modules/typescript/lib/typescript.js:87311:36)
    at tryGetModuleNameFromDeclaration (/Users/nazar/Hub/Chainsafe/Web3/projects/truffle/node_modules/typescript/lib/typescript.js:30114:59)
    at Object.getExternalModuleNameLiteral (/Users/nazar/Hub/Chainsafe/Web3/projects/truffle/node_modules/typescript/lib/typescript.js:30078:20)
    at createRequireCall (/Users/nazar/Hub/Chainsafe/Web3/projects/truffle/node_modules/typescript/lib/typescript.js:105088:33)
    at visitImportDeclaration (/Users/nazar/Hub/Chainsafe/Web3/projects/truffle/node_modules/typescript/lib/typescript.js:105039:80)
    at topLevelVisitor (/Users/nazar/Hub/Chainsafe/Web3/projects/truffle/node_modules/typescript/lib/typescript.js:104697:28)

[cds-amal]

@nazarhussain it looks like you may have built the yarn.lock from a different registry? Do you use verdaccio, or another npmjs proxy?

$ grep localhost yarn.lock | head -n 5
  resolved "http://localhost:4873/@ethereumjs%2fcommon/-/common-2.6.5.tgz#0a75a22a046272579d91919cb12d84f2756e8d30"
  resolved "http://localhost:4873/@ethereumjs%2ftx/-/tx-3.5.2.tgz#197b9b6299582ad84f9527ca961466fce2296c1c"
  resolved "http://localhost:4873/@ethersproject%2fabi/-/abi-5.6.4.tgz#f6e01b6ed391a505932698ecc0d9e7a99ee60362"
  resolved "http://localhost:4873/@noble%2fsecp256k1/-/secp256k1-1.6.3.tgz#7eed12d9f4404b416999d0c87686836c4c5c9b94"
  resolved "http://localhost:4873/@scure%2fbip39/-/bip39-1.1.0.tgz#92f11d095bae025f166bef3defcc5bf4945d419a"

$ git log -S"http://localhost:4873"
Thu Aug 4 23:01:18 2022 +0200 29671cc03 (HEAD -> nh/web3-4x-upgrade) :art: Fix the issues raised after merging base branch  [Nazar Hussain]
Thu Aug 4 15:34:12 2022 +0200 8c2d7c34e :arrow_up: Update dependenceies  [Nazar Hussain]%

[nazarhussain]

@cds-amal Yes I am using local NPM registry because these packages are not yet published. I will update yarn.lock soon after the release.

[eggplantzzz]

Hello! What is the status of this PR? Is there any chance you can rebase this? I would be happy to do the rebase if you'd like as it appears to be dependency-related things which hopefully will be easy to handle. Let me know!

[Muhammad-Altabba]

Hello @eggplantzzz Yes please, it would be great, if you could do the rebase.

[eggplantzzz]

Sorry for the delay @nazarhussain. Things have been shifting around here quite a bit and I've been preoccupied. I will do the rebase soon!

Update: it looks like I'm having trouble rebasing this PR as I think you are using a different npm registry for some unpublished packages, is that right? Also there were some dependencies that I think require Node >= 16 while current policy for Truffle is to support versions of Node as early as the latest version of 12. Is there any way we can get around this issue? It is failing after updating package.json's, regenerating the yarn.lock, and rebuilding. The library that the error is coming from is @noble/hashes.

Update2: we might actually be able to drop support for Node 14 in the near future, so that actually might not be a problem after we get to that point. We'll have to see how early we can do that.

[Muhammad-Altabba]

Sorry for the delay @nazarhussain. Things have been shifting around here quite a bit and I've been preoccupied. I will do the rebase soon!

Update: it looks like I'm having trouble rebasing this PR as I think you are using a different npm registry for some unpublished packages, is that right? Also there were some dependencies that I think require Node >= 16 while current policy for Truffle is to support versions of Node as early as the latest version of 12. Is there any way we can get around this issue? It is failing after updating package.json's, regenerating the yarn.lock, and rebuilding. The library that the error is coming from is @noble/hashes.

Update2: we might actually be able to drop support for Node 14 in the near future, so that actually might not be a problem after we get to that point. We'll have to see how early we can do that.

Thanks @eggplantzzz, Happy to hear that you are already planning to drop support for Node 14 in the near future. And it would be great if you will be able to rebase this MR. I did not try yet as I was working on some other related stuff. The current state of this MR is that you can install dependencies, and run builds and tests, without any errors. So, the following commands succeed on my machine:

yarn
yarn build
yarn test

However, there are still some skipped tests and TODOs.

Could you also please manage to run the CI for this MR? I suggest creating a feature branch at this repo and we merge this MR to that branch. If this would be helpful for both, running the CI and easing your collaboration on this.

Many thanks,

[nazarhussain]

Happy to hear that you are already planning to drop support for Node 14 in the near future.

That will be supper cool and will help to maintain Truffle.

@Muhammad-Altabba @eggplantzzz My suggestion would be to look keenly into skipped tests and if these are not critical, fix those in separate PRs, so we can get this big PR merged earlier.

[eggplantzzz]

Ok, let me switch to Node 16 and try the rebase again.

Also, do note @nazarhussain @Muhammad-Altabba that the command yarn bootstrap also needs to complete successfully. This is the command that builds all of Truffle's sub-packages.

[eggplantzzz]

So I rebased locally and am getting some build errors for @truffle/interface-adapter related to web3-providers-ws. Can you tell me anything about these? I'll paste a screenshot below -

[Muhammad-Altabba]

So I rebased locally and am getting some build errors for @truffle/interface-adapter related to web3-providers-ws. Can you tell me anything about these? I'll paste a screenshot below -

Thanks @eggplantzzz The issue is with "@types/ws" (https://github.com/web3/web3.js/issues/6303) Could you please try adding

"overrides": {
  "web3-providers-ws": {
    "@types/ws": "8.5.4"
  }
}

Thanks,

[eggplantzzz]

Thanks for the quick response! I added the overrides property and deleted the yarn.lock. When I try and build again I get more type errors.

Any other ideas?

[luu-alex]

@eggplantzzz Thanks for your efforts, the web3js team appreciates you :) I tried to understand the error but im still not sure completely what is wrong.

But this issue is coming from [email protected], which are the newest web3js release packages.

While I debug the issue with the build, I think the solution is using these versions without bumping to the latest release should work.

[eggplantzzz]

So I do have all packages bumped to ^v4.0.3, is that what you are recommending?

[luu-alex]

without the caret so it doesn't bump. v4.0.3. theres a PR i have made to fix this error, if you can wait until monday that release will be available with the fix.

[eggplantzzz]

Oh I see, you want me to fix the version at 4.0.3. I'll go ahead and try that.

Update: Now trying to install yields an error that a dependency requires a version of >=18.12.0. It seems to be coming from @yarnpkg/parsers. I don't think we can drop support for Node 16 yet. Do you know anything about this package?

[luu-alex]

@eggplantzzz we use lerna for our package manager and it seems to depend on yarnpkg/parsers@3 which is where the error is coming from for one of the packages. For the web3js repo, building with any node version of 16>= seems to work fine so im unsure why its asking for a version >=18 in the truffle suite