Cannot use proxy option to send https to the proxy

[jimamster]

When I run needle with the proxy option, I can only get needle to work when setting the proxy to http and not https.

Sample options are shown below.
rpurl = "https://endpointurl.com";

options = { auth: 'auto', proxy: 'http://sampleproxy.com:8080', multipart: true };

In this example, needle sends http protocol to the proxy even though the endpoint url is requiring https. So the proxy gets http and is required to translate to https. This is causing us some problems in negotiating cipher suites. We have a work around right now but would prefer to be able to run end to end https from needle to the proxy to the endpoint url.

When I try "proxy: 'https://sampleproxy.com:8080'", I receive the following error.

Error: [Error: 140735290954512:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:766:

Note that when running curl behind a proxy, curl initiates the https session with the proxy which simply passes the session out to the endpoint url. Curl appears to use the /etc/environment file proxy settings.

[jimamster]

A little more info. I am running needle on: mac osx 10.9.5

[tomas]

Good point. I'll take a look and let you know what I find.

[jimamster]

Thanks.

[tomas]

Fixed on 1e18950a91bc92d494d4f6749cb6bd1b0dbd52bf. Should be up on npm soon!

[jimamster]

That's great. Let me know when it reaches npm. Thanks.

Sent from my iPhone

On Dec 18, 2014, at 10:03 AM, Tomás Pollak [email protected] wrote:

Fixed on 1e18950. Should be up on npm soon!

— Reply to this email directly or view it on GitHub.

[zetlen]

Hey, I can't wait to switch away from mikeal/request for a lightweight SDK, but this has to get released first. +1!

[tomas]

This is already up! Sorry for not letting you know on time.

[zetlen]

Thanks for letting me know, but I'm still having some trouble! I'm able to tunnel requests through an HTTPS proxy successfully in Windows (with Fiddler), but not on OSX (with Charles). I have exported the root certificate, all other HTTPS requests work, and I can use Charles successfully with the mikeal/request module. But I switched away from mikeal/request because it's so heavy, and I really want to keep using your light, fluffy module. I need to be able to monitor requests in OSX! Do you have access to an OSX machine? Can you reproduce my problem? Thanks buddy.

[tomas]

I do have access to an OSX machine. How can I reproduce the problem?

[zetlen]

Thanks for waiting. I created a quick Google Apps Script for demonstration purposes, because I needed something that delivered innocuous JSON from HTTPS. Here is your repro:

1. Run Charles.app on OSX. Make sure that SSL is configured for all hosts, and that the certificates are configured correctly.

2. Put the following code in a file called repro.js.

   var needle = require('needle');
   
   needle.post('https://script.google.com/macros/s/AKfycbypnnX10X3ER4eMNOYUmKGF7noa5HMk0irFhKs0Ig7mpMnnG_OA/exec', {
     year: process.argv[process.argv.length-1]
   }, {
     proxy: 'http://127.0.0.1:8888',
     rejectUnauthorized: false,
     follow_max: 5,
     json: true
   }, function(err, res) {
     if (err) throw err;
     console.log(('got post response', res.body));
   });
   

3. Run the script and argue a four-digit random year. The service should tell you if it is a leap year. bash $ node repro.js 2016

4. Observe Charles. There is one redirect (the Google Apps Engine requires this) but then the service returns 200, with a small JSON response.

Expected: Needle should run the callback and the console.log should execute, or at least an exception should throw. Actual: The callback never runs.

I've been tracing this and it appears to be happening inside the JSON parser TransformStream in parsers.js. Its _transform function gets called, so the HttpMessage stream is fine, but its _flush function does not. It seems like Charles in SSL mode might fail to send whatever terminating message triggers Node's HTTPS client to end its response stream. Could the Proxy-Connection header be at fault here?

[tomas]

Holy mac! Once I get my hard hat I'll take a look into this, hehe.

[zetlen]

Hey, any luck on this? I could open a pull request to cover the timeout failure case, but the fix I have in mind is not so good.

[ostriandoni]

i still get this error:

2018-05-21T02:27:42.447149+00:00 app[web.1]: { Error: write EPROTO 139950560855872:error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error:../deps/openssl/openssl/ssl/s23_clnt.c:769:
2018-05-21T02:27:42.447152+00:00 app[web.1]: 
2018-05-21T02:27:42.447155+00:00 app[web.1]:     at exports._errnoException (util.js:1050:11)
2018-05-21T02:27:42.447156+00:00 app[web.1]:     at WriteWrap.afterWrite [as oncomplete] (net.js:814:14) code: 'EPROTO', errno: 'EPROTO', syscall: 'write' }

[tomas]

Yikes! Bringing this issue back from the dead...

[tomas]

@zetlen Sorry for the insanely late response, but since this issue was marked as closed I hadn't noticed it. Would you mind opening a PR with the fix you mentioned earlier? :)