npx hardhat compile fails with self signed certificate in certificate chain

[dandagow3t]

I'm using a machine that sits inside the network of a big company. The company has implemented security measures which imply adding their own root CAs on top of my SSL/TLS cert. Because of this situation, for example git could not clone a repo. To allow git to clone, I've exported the certs added by my company to git's bundle-ca.crt file (according to this article). But when it comes to npx hardhat compile task, hardhat uses node-fetch, so node and somehow I was unable to let node know about my company's root CAs.

Here is how my github.com cert's certification path looks like.

And bellow is the execution output of npx hardhat compile --show-stack-traces.

HardhatError: HH502: Couldn't download compiler versions list. Please check your connection.
    at CompilerDownloader.downloadCompilersList (/usr/src/app/node_modules/hardhat/src/internal/solidity/compiler/downloader.ts:201:13)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at CompilerDownloader.getCompilersList (/usr/src/app/node_modules/hardhat/src/internal/solidity/compiler/downloader.ts:162:7)
    at CompilerDownloader._fetchVersionPath (/usr/src/app/node_modules/hardhat/src/internal/solidity/compiler/downloader.ts:256:16)
    at CompilerDownloader._versionExists (/usr/src/app/node_modules/hardhat/src/internal/solidity/compiler/downloader.ts:274:25)
    at CompilerDownloader.getCompilerBuild (/usr/src/app/node_modules/hardhat/src/internal/solidity/compiler/downloader.ts:183:9)
    at CompilerDownloader.isCompilerDownloaded (/usr/src/app/node_modules/hardhat/src/internal/solidity/compiler/downloader.ts:74:27)
    at SimpleTaskDefinition.action (/usr/src/app/node_modules/hardhat/src/builtin-tasks/compile.ts:492:36)
    at Environment._runTaskDefinition (/usr/src/app/node_modules/hardhat/src/internal/core/runtime-environment.ts:217:14)
    at Environment.run (/usr/src/app/node_modules/hardhat/src/internal/core/runtime-environment.ts:129:14)

    Caused by: FetchError: request to https://binaries.soliditylang.org/linux-amd64/list.json failed, reason: self signed certificate in certificate chain
        at ClientRequest.<anonymous> (/usr/src/app/node_modules/node-fetch/lib/index.js:1491:11)
        at ClientRequest.emit (node:events:513:28)
        at TLSSocket.socketErrorListener (node:_http_client:481:9)
        at TLSSocket.emit (node:events:513:28)
        at emitErrorNT (node:internal/streams/destroy:157:8)
        at emitErrorCloseNT (node:internal/streams/destroy:122:3)
        at processTicksAndRejections (node:internal/process/task_queues:83:21)

The workaround I'm using right now is to sett node's env variable NODE_TLS_REJECT_UNAUTHORIZED to zero, so disabling any SSL/TLS check, which is not secure and it's just temporarily.

[github-actions[bot]]

This issue is also being tracked on Linear.

We use Linear to manage our development process, but we keep the conversations on Github.

LINEAR-ID: e4c06675-4778-4343-a1af-a49ef9534d43

[alcuadrado]

Hey,

Can you try with hardhat@rc and report back? It uses a new compiler downloader.

[dandagow3t]

Hey, thanks for the fast reply! So, I've used hardhat version 2.11.0-rc.0. I've also added my company's root certificates using an environment variable that node knows which is NODE_EXTRA_CA_CERTS. Now, downloading the compiler works - it seems that the downloader correctly takes into account the newly added root CA certs. But now I have another problem with this version of hh. On Windows npx hardhat compile works fine but when I run it inside a docker container based on node:16-alpine3.16, npx hardhat compile --show-stack-traces sayis:

An unexpected error occurred:

Error: spawn /root/.cache/hardhat-nodejs/compilers/linux-amd64/solc-linux-amd64-v0.8.4+commit.c7e474f2 ENOENT
    at Process.ChildProcess._handle.onexit (node:internal/child_process:283:19)
    at onErrorNT (node:internal/child_process:478:16)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'spawn /root/.cache/hardhat-nodejs/compilers/linux-amd64/solc-linux-amd64-v0.8.4+commit.c7e474f2',
  path: '/root/.cache/hardhat-nodejs/compilers/linux-amd64/solc-linux-amd64-v0.8.4+commit.c7e474f2',
  spawnargs: [ '--standard-json' ],
  cmd: '/root/.cache/hardhat-nodejs/compilers/linux-amd64/solc-linux-amd64-v0.8.4+commit.c7e474f2 --standard-json'
}

Seems like it is looking for linux-amd64/solc-linux-amd64-v0.8.4+commit.c7e474f2 but it's not present. So, I think this bug is fixed in 2.11.0-rc.0, but it seems I've stumbled on something new.

[alcuadrado]

Can you please share the steps to reproduce it?

[dandagow3t]

Sure, so the error appears inside a Docker Container. Docker files looks like this:

FROM node:16-alpine3.16

COPY . /usr/src/app

WORKDIR /usr/src/app

RUN apk add git;

RUN apk --no-cache add curl;

RUN yarn install --non-interactive

COPY $PWD/docker/entrypoint.sh /usr/local/bin

ENTRYPOINT ["/bin/sh", "/usr/local/bin/entrypoint.sh"]

What's important is the FROM node:16-alpine3.16 of course and the RUN yarn install --non-interactive the rest are just support instructions. As for the package.json, here are my dependecies:

"dependencies": {
    "@nomiclabs/hardhat-ethers": "^2.0.5",
    "@nomiclabs/hardhat-waffle": "^2.0.2",
    "@openzeppelin/contracts": "^4.5.0",
    "axios": "^0.26.0",
    "chai": "^4.3.6",
    "ethereum-waffle": "^3.4.0",
    "ethers": "^5.5.4",
    "hardhat": "2.11.0-rc.0",
    "ipfs-http-client": "^56.0.1",
    "next": "^12.1.6",
    "react": "17.0.2",
    "react-dom": "17.0.2",
    "web3modal": "^1.9.5"
  },
  "devDependencies": {
    "autoprefixer": "^10.2.6",
    "eslint": "7.29.0",
    "eslint-config-next": "11.0.1",
    "postcss": "^8.3.5",
    "tailwindcss": "^2.2.4"
  }

(Please observe that I'm using "hardhat": "2.11.0-rc.0" as discussed to solve the download compiler issue.) So, I'm building the image docker build . -t ImageName, then run the container docker run -it -d -p 8545:8545 -p 3000:3000 --name myContainerName ImageName. Then inside the container, when I issue npx hardhat compile --show-stack-traces, I get the above error. Thanks

[fvictorio]

For the record: I don't think docker nor alpine are related, I just created a project inside a container from that image and it worked fine.

@dandamian83 can you access https://binaries.soliditylang.org/linux-amd64/list.json? For example, through the browser, or using curl or wget? (Ideally: try the three things)

[dandagow3t]

(All that I'm relating happens inside the docker container. On my localhost now everything works properly.)

Yes, I can access using curl and wget. I've tried with both hardhat 2.8.4 and 2.11.0-rc.0 to compile so npx hardhat compile. Now the compiler gets downloaded but no artifacts are generated.

/usr/src/app # npx hardhat compile
Downloading compiler 0.8.4
npm notice
npm notice New minor version of npm available! 8.15.0 -> 8.19.1
npm notice Changelog: https://github.com/npm/cli/releases/tag/v8.19.1
npm notice Run npm install -g [email protected] to update!
npm notice

When I issue a second npx hardhat compile --show-stack-traces, I get:

/usr/src/app # npx hardhat compile
An unexpected error occurred:

Error: spawn /root/.cache/hardhat-nodejs/compilers/linux-amd64/solc-linux-amd64-v0.8.4+commit.c7e474f2 ENOENT
    at Process.ChildProcess._handle.onexit (node:internal/child_process:283:19)
    at onErrorNT (node:internal/child_process:478:16)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'spawn /root/.cache/hardhat-nodejs/compilers/linux-amd64/solc-linux-amd64-v0.8.4+commit.c7e474f2',
  path: '/root/.cache/hardhat-nodejs/compilers/linux-amd64/solc-linux-amd64-v0.8.4+commit.c7e474f2',
  spawnargs: [ '--standard-json' ],
  cmd: '/root/.cache/hardhat-nodejs/compilers/linux-amd64/solc-linux-amd64-v0.8.4+commit.c7e474f2 --standard-json'
}

When I check inside the /root/.cache/hardhat-nodejs/compilers/linux-amd64/ directory, I can see that solc-linux-amd64-v0.8.4+commit.c7e474f2 is present.

/usr/src/app # cd /root/.cache/hardhat-nodejs/compilers/linux-amd64/
~/.cache/hardhat-nodejs/compilers/linux-amd64 # ls -laF
total 11612
drwxr-xr-x    2 root     root          4096 Sep  7 06:49 ./
drwxr-xr-x    3 root     root          4096 Sep  7 06:49 ../
-rw-r--r--    1 root     root         42148 Sep  7 06:49 list.json
-rwxr-xr-x    1 root     root      11833912 Sep  7 06:49 solc-linux-amd64-v0.8.4+commit.c7e474f2*

But when I want to execute the file by myself, I get the same not found error. It's like bin/sh can't see and execute the file.

~/.cache/hardhat-nodejs/compilers/linux-amd64 # cd /
/ # ./root/.cache/hardhat-nodejs/compilers/linux-amd64/solc-linux-amd64-v0.8.4+commit.c7e474f2
/bin/sh: ./root/.cache/hardhat-nodejs/compilers/linux-amd64/solc-linux-amd64-v0.8.4+commit.c7e474f2: not found
/ #

It seems that my /bin/sh points to /bin/busybox. 0 lrwxrwxrwx 1 root root 12 Aug 9 08:47 sh -> /bin/busybox*

[fvictorio]

@dandamian83 any chance of creating a complete reproduction example? That would mean a repo with the Dockerfile (which I guess is just the one that you already shared) and the code you are copying into the image.

[dandagow3t]

Hi,

Unfortunately, not yet... Sorry. Dan Damian

Bucharest, Romania

On Sat, Sep 17, 2022 at 4:33 PM jankulma @.***> wrote:

@dandamian83 https://github.com/dandamian83 did you fix it? I'm running into the same/similar issue

— Reply to this email directly, view it on GitHub https://github.com/NomicFoundation/hardhat/issues/3118#issuecomment-1250072674, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC4KQKPPZ2N6FNDEPN4YK2TV6XCDBANCNFSM6AAAAAAQDCFQ5M . You are receiving this because you were mentioned.Message ID: @.***>

[hickscorp]

We seem to be having a very similar (if not identical) problem. We can build our projects just fine locally on dev computers. But when reaching CI / CD, it seems that the container is having issues executing solc...

yarn run v1.22.19
$ yarn build:contracts && yarn build:abi && yarn build:types
$ TS_NODE_TRANSPILE_ONLY=1 yarn hardhat compile
$ /home/app/node_modules/tt-white-contracts/node_modules/.bin/hardhat compile
Cannot read keys file at conf/keys.staging.json .
Cannot read keys file at conf/keys.production.json .
Cannot read keys file at conf/keys.mumbai.json .
An unexpected error occurred:

Error: spawn /root/.cache/hardhat-nodejs/compilers/linux-amd64/solc-linux-amd64-v0.8.10+commit.fc410830 ENOENT
    at Process.ChildProcess._handle.onexit (node:internal/child_process:283:19)
    at onErrorNT (node:internal/child_process:478:16)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'spawn /root/.cache/hardhat-nodejs/compilers/linux-amd64/solc-linux-amd64-v0.8.10+commit.fc410830',
  path: '/root/.cache/hardhat-nodejs/compilers/linux-amd64/solc-linux-amd64-v0.8.10+commit.fc410830',
  spawnargs: [ '--standard-json' ],
  cmd: '/root/.cache/hardhat-nodejs/compilers/linux-amd64/solc-linux-amd64-v0.8.10+commit.fc410830 --standard-json'
}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.

However ls /root/.cache/hardhat-nodejs/compilers/linux-amd64/ gives us confirmation that /root/.cache/hardhat-nodejs/compilers/linux-amd64/solc-linux-amd64-v0.8.10+commit.fc410830 exists...

We will investigate a bit more, by dropping a shell in the container and trying to see if maybe the compiler is in fact an invalid symlink.

This issue has arised recently - we updated a few dependencies (Including hardhat, hardhat-deploy and some others).

[fvictorio]

@hickscorp was your CI caching the downloaded compilers by any chance?

[jankulma]

I had a similar issue, and in my case it was cache. I was using docker image (built in CI), and I just had to rebuild it with no cache

(Also, I did not change anything related to certificates)

[fvictorio]

Closing this for lack of reproduction steps.

[dandagow3t]

hi, bringing some updates on this issue. Context: inside the docker container (same as described above). npx hardhat compile is not working:

npx hardhat compile --show-stack-traces
An unexpected error occurred:

Error: spawn /root/.cache/hardhat-nodejs/compilers/linux-amd64/solc-linux-amd64-v0.8.4+commit.c7e474f2 ENOENT
    at Process.ChildProcess._handle.onexit (node:internal/child_process:285:19)
    at onErrorNT (node:internal/child_process:485:16)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'spawn /root/.cache/hardhat-nodejs/compilers/linux-amd64/solc-linux-amd64-v0.8.4+commit.c7e474f2',
  path: '/root/.cache/hardhat-nodejs/compilers/linux-amd64/solc-linux-amd64-v0.8.4+commit.c7e474f2',
  spawnargs: [ '--standard-json' ],
  cmd: '/root/.cache/hardhat-nodejs/compilers/linux-amd64/solc-linux-amd64-v0.8.4+commit.c7e474f2 --standard-json'
}

Went to /root/.cache/hardhat-nodejs/compilers/linux-amd64/, the compiler was there. So I've tried to run it manually again.

bash-5.1# ./solc-linux-amd64-v0.8.4+commit.c7e474f2
bash: ./solc-linux-amd64-v0.8.4+commit.c7e474f2: No such file or directory

Googling for this I've found that - Typically, the "unable to execute... No such file or directory" means that either the executable binary itself or one of the libraries it needs does not exist. Libraries can also need other libraries themselves.; here is the article. For now I think I'll precompile the smart contracts and just copy them on the docker image...

[fvictorio]

@dandamian83 are you using the latest version of Hardhat? We fixed a similar issue some versions ago.

[dandagow3t]

I was using 2.8.4...switched to latest version which is 2.12.4 and it worked. Thanks!

[ssolson]

I came across this issue and wanted to explicitly lay out what solved it for me.

• My proxy was set in the hardhat config file.
• I was getting an HH502 for certificates (CA) error.
• I solved the issue by setting NODE_EXTRA_CA_CERTS.

I found my certs using python:

python3 -c "import requests; print(requests.certs.where())"

From the file it returned I copied the cert I wanted into a new file I called certs.pem. The copied part was something like this:

----Begin Cert--- kajhklsadjfhdfgh... ---END CERT----

I then set this as the certificate set NODE_EXTRA_CA_CERTS=/path/to/cert.pem

Hope this helps someone.