truffle Bad Requests behind a proxy since version 5.3.4

[x] I've asked for help in the Truffle Gitter before filing this issue.

Issue

Using a truffle operation on a Windows System with truffle version 5.3.4, that sends a HTTP request, you get a Bad Request Error. My Windows System is behind a proxy.

Maybe the switch to axios in 5.3.4 is causing this issue.

Steps to Reproduce

Upgrade your truffle version to 5.3.4 and try to use an operation that sends a HTTP request, e.g. truffle unbox metacoin or in an existing project truffle compile --list.

Expected Behavior

Truffle does not send bad requests.

Actual Results

One of the operations that sends a bad request ist truffle compile --version.

Output with version 5.3.4 (not working):

C:\Temp>truffle compile --list
× Fetching solc version list from solc-bin. Attempt #1
× Fetching solc version list from solc-bin. Attempt #2
× Fetching solc version list from solc-bin. Attempt #3
Error: Failed to complete request to: version URLs. Are you connected to the internet?

Error: Request failed with status code 400
    at VersionRange.errors (C:\Users\xn06112\AppData\Roaming\npm\node_modules\truffle\build\webpack:\packages\compile-solidity\compilerSupplier\loadingStrategies\LoadingStrategy.js:66:1)
    at C:\Users\xn06112\AppData\Roaming\npm\node_modules\truffle\build\webpack:\packages\compile-solidity\compilerSupplier\loadingStrategies\VersionRange.js:158:1
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.listVersions (C:\Users\xn06112\AppData\Roaming\npm\node_modules\truffle\build\webpack:\packages\core\lib\commands\compile.js:122:1)
    at Object.run (C:\Users\xn06112\AppData\Roaming\npm\node_modules\truffle\build\webpack:\packages\core\lib\commands\compile.js:71:1)
    at Command.run (C:\Users\xn06112\AppData\Roaming\npm\node_modules\truffle\build\webpack:\packages\core\lib\command.js:140:1)
Truffle v5.3.4 (core: 5.3.4)
Node v16.1.0

Output with version 5.3.3 (working):

C:\Temp>truffle compile --list
√ Fetching solc version list from solc-bin. Attempt #1
[
 "0.8.4",
 "0.8.3",
 "0.8.2",
 "0.8.1",
 "0.8.0",
 "0.7.6",
 "0.7.5",
 "0.7.4",
 "0.7.3",
 "0.7.2",
 ".. and 77 more. Use `--all` to see full list."
]

Environment

Operating System: Windows 10
Ethereum client:
Truffle version (truffle version): 5.3.4
node version (node --version): 16.1.0
npm version (npm --version): 7.11.2

May 05 '21 10:05 ityreh

Thanks for reporting this. We'll get this on our backlog to see what's going wrong here. You're probably right about the change to the axios library. But we'll do further digging to see if we can't fix the issue.

Sorry about this!

May 05 '21 18:05 gnidan

I found an option for axios, maxRedirects which is 5 by default. I'll put in a PR to set it to 50 and perhaps this will solve the issue.

May 10 '21 20:05 eggplantzzz

@ityreh We just released a new version of Truffle (5.3.6). It might fix your issue, could you please upgrade and let us know whether it does? Thank you!

May 11 '21 19:05 eggplantzzz

@eggplantzzz thank you, but I get the same error with version 5.3.6 and after downgrading to 5.3.3 it works again.

May 12 '21 10:05 ityreh

Hmm, I thought that increasing the amount of allowable redirects might solve it...I wonder what could be causing this.

May 12 '21 15:05 eggplantzzz

Can you tell us anything more about your setup? I'm not sure how involved it would be for us to reproduce your error. I guess we'll have to look into the axios options again and maybe compare them to what request is doing.

May 12 '21 19:05 eggplantzzz

Hey guys, I'm a colleague of Yannick and I have the same problem. Our environment is a heavily restricted enterprise environment, but the http proxy is nothing special. All tools that use the "http_proxy" environment variable work with it, e.g. curl. Other tools that provide proxy settings with an "http proxy" option also work with it, e.g. npm.

We already know that truffle stopped working when the http client was migrated to axios. And I've found that axios has two modes regarding proxy settings:

browser mode (uses browser proxy settings)
nodejs mode: requires a proxy object parameter

Please see this axios issue for more information.

Jun 08 '21 17:06 thorstenhirsch

Hey thanks for the information @thorstenhirsch! I'm going to look into it directly.

In this case it looks like the proxy object is used for you to customize where the request will be sent with the idea that it will be handled and sent by whatever is listening there. Does this mean in these cases that the user will be required to enter in their own custom info to use here? This doesn't seem great.

Jun 09 '21 14:06 eggplantzzz

Maybe we need something like this...

if (env['http_proxy']) {
  proxy = build_proxy_object(env['http_proxy']);
} else {
  proxy = {}
}
axios.get(..., proxy);

But first I'd suggest to take a look into the code of the old http client (which was used before the migration to axios), because I think that the old one solved the task perfectly. There might be some quirks related to the http_proxy variable, because there might also be https_proxy. And I don't know which one to prefer, because in our environment there's no difference between them. Others might do.

It's a bit unfortunate that axios lacks this feature and we need to implement it ourselves. Maybe axios wasn't supposed to be run in other runtimes than the browser...?

Jun 09 '21 15:06 thorstenhirsch

Yeah this should be something that we can solve. Do you really think it has something to do with setting up the proxy object? The old lib did it by itself without any extra setup and so I would assume it should be something we can set and forget. I'll go back to the old lib and see what it does differently under the hood.

Jun 09 '21 15:06 eggplantzzz

So I did some digging with a colleague and understand a little bit better how axios and request handle some of the proxy details. request seems to have a much more robust handling of the environment variables. It looks like request handles "http_proxy" and "https_proxy" but it doesn't look for the other variants such as "HTTPS_PROXY".

I am curious to see how your environment is set up. Would you be able to open a terminal and see which ones are populated?

That would be something like env | grep -i "http" on a mac or linux to get all the environment variables that match "http". Does that work on a Windows machine? You could also just open node and manually inspect those variables. Let me know which ones have values if you get a chance. That would help in narrowing down the problem and figuring out the solution.

The other possibility is that axios won't handle username/password stuff but it doesn't look like request handles that out of the box without some extra information passed to it. Truffle had a bare bones implementation that didn't include passing request any auth info.

Jun 09 '21 21:06 eggplantzzz

I'm on Windows (unfortunately) and I've set http_proxy=http://username:[email protected]:8080. No uppercase variable and no https_proxy.

Jun 09 '21 21:06 thorstenhirsch

What happens if you populate the https_proxy variable with proxy info? I wonder if axios will pick that up as the target uri specifies https protocol.

Jun 09 '21 22:06 eggplantzzz

Setting https_proxy makes a little difference... now truffle unbox fails at once, no waiting, no timeout:

>truffle unbox pet-shop

Starting unbox...
=================

√ Preparing to download box
× Downloading
Unbox failed!
× Downloading
Unbox failed!
Error: Error connecting to github.com. Please check your internet connection and try again.
    at C:\Program Files\nodejs\node_modules\truffle\build\webpack:\packages\box\dist\lib\utils\unbox.js:45:1
    at Generator.throw (<anonymous>)
    at rejected (C:\Program Files\nodejs\node_modules\truffle\build\webpack:\packages\box\dist\lib\utils\unbox.js:6:41)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
Truffle v5.3.9 (core: 5.3.9)
Node v14.16.1

Without https_proxy the command at least is running for a few seconds before it fails (at the first "Downloading"). This behavior is reproducible. Unfortunately there's no difference in the output. The error and line numbers in the stack trace are exactly the same.

Jun 10 '21 09:06 thorstenhirsch

Hmm, ok. How easy is it to setup a proxy situation like you have there? I would love to be able to duplicate a setup like that locally here to test and hopefully solve this issue.

Jun 10 '21 14:06 eggplantzzz

Unfortunately I don't know what software our http proxy server is using, but since it works with default http proxy settings on the client I guess it's something like squid. I guess you don't even need to enable user authentication to reproduce the problem.

Jun 10 '21 16:06 thorstenhirsch

So we were able to cook up a minimal example proof of concept here locally using http toolkit. In the environment we needed to populate http_proxy and some certificate-related variables, but ultimately we were able to use the proxy to unbox successfully. So there must be something specific about your setup that axios doesn't jive with.

There was also no username/password-type auth involved here. The original error by the OP was 400 error which means that there is something bad about the request - this 400 might be coming from the proxy server itself? I'll keep thinking about this problem and see if I can think of any way to proceed on troubleshooting this. Let me know if you have any ideas or can get any more useful information about this issue. Thanks!

Jun 10 '21 17:06 eggplantzzz

I think the bug behind this pull request is causing the 400 error. We've exactly the same setup: our https_proxy is reached via http. So if axios creates an https request it's the wrong protocol for an http listener, thus 400 is expected.

Unfortunately the PR hasn't been merged into axios, yet.

Jun 14 '21 18:06 thorstenhirsch

Alright, I did some further investigation and it seems like it's already fixed in the latest version of axios thanks to this commit. So we only need to update the dependencies from axios v0.20.0 to v0.21.1.

edit: I see that packages/*/package.json already depend on axios v0.21.1 ("axios": "^0.21.1"), but npm i -g truffle results in an installation with node_modules/axios that contains a package.json with v0.20.0.

Jun 14 '21 20:06 thorstenhirsch

Oh sweet! That PR should fix this issue you think? Did you ever narrow down exactly what was causing it?

I'll go ahead and put in a PR to update this. Thanks for helping me troubleshoot this issue @thorstenhirsch!

Jun 15 '21 17:06 eggplantzzz

@thorstenhirsch I just checked and Truffle is actually using version 0.21.1 of axios. There is a dependency that uses version 0.20.0 but the unboxing portion of Truffle uses 0.21.1. I verified this by logging out the package.json where it is required in packages/box.

What happens is that Truffle gets bundled by webpack and that bundled file ends up getting run when you do truffle <command>. I'll have to take a look at the PR you referenced a bit ago to see if that sheds light.

Jun 15 '21 18:06 eggplantzzz

I'm still doing some research on this. I wonder if it has something to do with the fact that the target specifies https protocol and your proxy uses http. Hmmm...

Jun 15 '21 20:06 eggplantzzz

Having this issue in 5.3.12

Jul 01 '21 10:07 Jason5Lee

It is still a problem in Truffle v5.4.5

Aug 10 '21 14:08 tonguyengit

Hello, I still have this issue with linux in Truffle v5.4.11

Sep 22 '21 15:09 meremST

Hello, i am also running into this problem on Windows with Truffle v5.4.11

Sep 27 '21 05:09 UISebastian

Same issue for me with the latest version. I use the docker option to bypass this problem:

compilers: {
  solc: {
    version: "0.8.9",
    docker: true,  
  }
}

Nov 06 '21 10:11 nyg

@nyg, @UISebastian can you run the unbox command with DEBUG enabled and share the output? DEBUG=* truffle unbox <the box you want>

Nov 06 '21 14:11 cds-amal

@cds-amal Thanks for looking into it! :) Here you go:

$ mkdir test && cd test
$ npm init -y && npm i truffle
$ DEBUG=* npx truffle unbox metacoin metacoin
2021-11-09T07:49:41.275Z core:config-migration Truffle files need to be migrated
2021-11-09T07:49:41.296Z core:command:run Truffle data migration failed: Error: Cannot find module '/home/xyz/.config/truffle/config.json' at webpackEmptyContext (/home/xyz/Documents/x/test/node_modules/truffle/build/webpack:/packages/core/lib|sync:2:1) at Object.migrateGlobalConfig (/home/xyz/Documents/x/test/node_modules/truffle/build/webpack:/packages/core/lib/config-migration.js:47:25) at Object.migrateTruffleDataIfNecessary (/home/xyz/Documents/x/test/node_modules/truffle/build/webpack:/packages/core/lib/config-migration.js:34:1) at Command.run (/home/xyz/Documents/x/test/node_modules/truffle/build/webpack:/packages/core/lib/command.js:85:1) at Object.586806 (/home/xyz/Documents/x/test/node_modules/truffle/build/webpack:/packages/core/cli.js:57:1) at __webpack_require__ (/home/xyz/Documents/x/test/node_modules/truffle/build/webpack:/webpack/bootstrap:18:1) at __webpack_require__.x (/home/xyz/Documents/x/test/node_modules/truffle/build/webpack:/webpack/bootstrap:36:1) at Function.__webpack_require__.x (/home/xyz/Documents/x/test/node_modules/truffle/build/webpack:/webpack/runtime/startup chunk dependencies:37:1) at /home/xyz/Documents/x/test/node_modules/truffle/build/webpack:/webpack/startup:3:1 at Object.<anonymous> (/home/xyz/Documents/x/test/node_modules/truffle/build/cli.bundled.js:633:12) at Module._compile (node:internal/modules/cjs/loader:1095:14) at Object.Module._extensions..js (node:internal/modules/cjs/loader:1147:10) at Module.load (node:internal/modules/cjs/loader:975:32) at Function.Module._load (node:internal/modules/cjs/loader:822:12) at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) at node:internal/main/run_main_module:17:47 { code: 'MODULE_NOT_FOUND' }
2021-11-09T07:49:41.341Z unbox { in: 'metacoin', out: 'https://github.com:truffle-box/metacoin-box' }

Starting unbox...
=================

- Preparing to download box
✔ Preparing to download box
- Downloading
2021-11-09T07:49:41.347Z follow-redirects options {
  maxRedirects: 50,
  maxBodyLength: 10485760,
  protocol: 'http:',
  path: 'https://raw.githubusercontent.com/truffle-box/metacoin-box/master/truffle-box.json',
  method: 'HEAD',
  headers: {
    Accept: 'application/json, text/plain, */*',
    'User-Agent': 'axios/0.21.1',
    host: 'raw.githubusercontent.com'
  },
  agent: undefined,
  agents: { http: undefined, https: undefined },
  auth: undefined,
  hostname: '<ip of proxy>',
  port: '<port of proxy>',
  host: '<ip of proxy>',
  beforeRedirect: [Function: beforeRedirect],
  nativeProtocols: {
    'http:': {
      _connectionListener: [Function: connectionListener],
      METHODS: [Array],
      STATUS_CODES: [Object],
      Agent: [Function],
      ClientRequest: [Function: ClientRequest],
      IncomingMessage: [Function: IncomingMessage],
      OutgoingMessage: [Function: OutgoingMessage],
      Server: [Function: Server],
      ServerResponse: [Function: ServerResponse],
      createServer: [Function: createServer],
      validateHeaderName: [Function: __node_internal_],
      validateHeaderValue: [Function: __node_internal_],
      get: [Function: get],
      request: [Function: request],
      maxHeaderSize: [Getter],
      globalAgent: [Getter/Setter]
    },
    'https:': {
      Agent: [Function: Agent],
      globalAgent: [Agent],
      Server: [Function: Server],
      createServer: [Function: createServer],
      get: [Function: get],
      request: [Function: request]
    }
  }
}
✖ Downloading
Unbox failed!
✖ Downloading
Unbox failed!
Error: Error connecting to https://raw.githubusercontent.com/truffle-box/metacoin-box/master/truffle-box.json. Please check your internet connection and try again.

socket hang up
    at connResetException (node:internal/errors:691:14)
    at Socket.socketOnEnd (node:_http_client:471:23)
    at Socket.emit (node:events:402:35)
    at endReadableNT (node:internal/streams/readable:1340:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)
Truffle v5.4.18 (core: 5.4.18)
Node v17.0.1

I see that the Axios version is still 0.21.1. Looking at the previous comments, the issue may have been fixed in more recent versions (it might just be a question of updating the dependency).

Nov 09 '21 08:11 nyg

the same:

jiecheng@Jiecheng-Linux-CUDA:~/workspace/learn/MetaCoin$ DEBUG=* truffle unbox metacoin
  unbox { in: 'metacoin', out: 'https://github.com:truffle-box/metacoin-box' } +0ms

Starting unbox...
=================

✔ Preparing to download box
⠋ Downloading  follow-redirects options {
  maxRedirects: 50,
  maxBodyLength: 10485760,
  protocol: 'http:',
  path: 'https://raw.githubusercontent.com/truffle-box/metacoin-box/master/truffle-box.json',
  method: 'HEAD',
  headers: {
    Accept: 'application/json, text/plain, */*',
    'User-Agent': 'axios/0.21.1',
    host: 'raw.githubusercontent.com'
  },
  agent: undefined,
  agents: { http: undefined, https: undefined },
  auth: undefined,
  hostname: '127.0.0.1',
  port: '12333',
  host: '127.0.0.1',
  beforeRedirect: [Function: beforeRedirect],
  nativeProtocols: {
    'http:': {
      _connectionListener: [Function: connectionListener],
      METHODS: [Array],
      STATUS_CODES: [Object],
      Agent: [Function],
      ClientRequest: [Function: ClientRequest],
      IncomingMessage: [Function: IncomingMessage],
      OutgoingMessage: [Function: OutgoingMessage],
      Server: [Function: Server],
      ServerResponse: [Function: ServerResponse],
      createServer: [Function: createServer],
      validateHeaderName: [Function: __node_internal_],
      validateHeaderValue: [Function: __node_internal_],
      get: [Function: get],
      request: [Function: request],
      maxHeaderSize: [Getter],
      globalAgent: [Getter/Setter]
    },
    'https:': {
      Agent: [Function: Agent],
      globalAgent: [Agent],
      Server: [Function: Server],
      createServer: [Function: createServer],
      get: [Function: get],
      request: [Function: request]
    }
  }
} +0ms
✖ Downloading
Unbox failed!
✖ Downloading
Unbox failed!
Error: Error connecting to https://raw.githubusercontent.com/truffle-box/metacoin-box/master/truffle-box.json. Please check your internet connection and try again.

Request failed with status code 500
    at createError (/home/jiecheng/.nvm/versions/node/v16.11.1/lib/node_modules/truffle/build/webpack:/node_modules/axios/lib/core/createError.js:16:1)
    at settle (/home/jiecheng/.nvm/versions/node/v16.11.1/lib/node_modules/truffle/build/webpack:/node_modules/axios/lib/core/settle.js:17:1)
    at IncomingMessage.handleStreamEnd (/home/jiecheng/.nvm/versions/node/v16.11.1/lib/node_modules/truffle/build/webpack:/node_modules/axios/lib/adapters/http.js:260:1)
    at IncomingMessage.emit (node:events:402:35)
    at endReadableNT (node:internal/streams/readable:1343:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)
Truffle v5.4.21 (core: 5.4.21)
Node v16.11.1
jiecheng@Jiecheng-Linux-CUDA:~/workspace/learn/MetaCoin$

Nov 23 '21 23:11 JiechengZhao

truffle truffle copied to clipboard

Bad Requests behind a proxy since version 5.3.4

Issue

Steps to Reproduce

Expected Behavior

Actual Results

Environment

truffle
truffle copied to clipboard