scalaj-http icon indicating copy to clipboard operation
scalaj-http copied to clipboard

Published 20 hours ago • verified publisher icon scalaj

Proxy with Authentication

Open crockpotveggies opened this issue 9 years ago • 33 comments

It appears that scalaj may not support Proxies with authentication. After attempting to set a Proxy-Authorization header and passing an appropriate proxy config to the request object, I get the following error:

ava.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 proxy authorization required"
  at sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2085)
  at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183)
  at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
  at scalaj.http.BaseHttp$$anonfun$apply$8.apply(Http.scala:638)
  at scalaj.http.BaseHttp$$anonfun$apply$8.apply(Http.scala:638)
  at scalaj.http.HttpRequest.exec(Http.scala:293)
  at scalaj.http.HttpRequest.execute(Http.scala:268)
  at scalaj.http.HttpRequest.asString(Http.scala:483)

crockpotveggies avatar Jan 20 '16 17:01 crockpotveggies

After coordinating with a couple people on this one, it appears that there's an underlying issue that I'm hoping may be addressable within the current version of scalaj.

Because the endpoint beyond the proxy is HTTPS, my Proxy-Authorization header is being stripped on the initial CONNECT with the proxy server.

Is there a way to modify the connection to ensure that the header stays put?

crockpotveggies avatar Jan 20 '16 17:01 crockpotveggies

hi, i'm not sure, could you send a code example for the request your making? also, a public authenticated proxy or explanation of how to create one would be helpful for me to reproduce

hoffrocket avatar Jan 20 '16 18:01 hoffrocket

You can set up a free trial at http://proxymesh.com/ which can give you some credentials and a proxy to connect. Steps to reproduce the problem (and code) are as follows:

  • Make an HTTPS connection to a destination beyond the proxy server.
  • Add a proxy host and port to scalaj
  • Add a Proxy-Authorization header with base64-encoded credentials in the form of user:pass
  • Initiate the connection, which will receive a 407 response code
val request = 
  Http("https://github.com/")
    .method("GET")
    .proxy("somehost.proxy.com", 32017)
    .header("Proxy-Authorization","Basic 1b2a3s4e56647e8n9c0o1d2e3c4r5e6d7s")

println(request.asString.body)

crockpotveggies avatar Jan 20 '16 19:01 crockpotveggies

I think you might need to set a global Authenticator to handle this. See this answer: http://stackoverflow.com/a/16340273

It's a bit gross to have to set a global like this, but let me know if this works. Maybe it can be baked into this library:

import java.net.{Authenticator, PasswordAuthentication}
def setProxyCreds(proxyHost: String, proxyPort: Int, proxyUser: String, proxyPassword: String): Unit = {
  Authenticator.setDefault(new Authenticator() {
    override def getPasswordAuthentication(): PasswordAuthentication = {
      if (getRequestorType() == Authenticator.RequestorType.PROXY) {
        if (getRequestingHost().equalsIgnoreCase(proxyHost) && getRequestingPort() == proxyPort) {
          new PasswordAuthentication(proxyUser, proxyPassword.toCharArray())
        }
      }
      null
    }
  })
}

hoffrocket avatar Jan 20 '16 19:01 hoffrocket

I tried this very same solution and wasn't able to get it to work, unfortunately. I also tried setting the global System.setProperty and experienced the same failure.

Is it possible that scalaj doesn't use the default authenticator when initiating the connect? Were you able to successfully apply your solution?

crockpotveggies avatar Jan 20 '16 20:01 crockpotveggies

I haven't tested myself, but scalaj uses the HTTPURLConnection under the covers, so I was thinking that would work. If you add some printlns to the Authenticator, is it at least being called?

On Wed, Jan 20, 2016 at 3:06 PM, crockpotveggies [email protected] wrote:

I tried this very same solution and wasn't able to get it to work, unfortunately. I also tried setting the global System.setProperty and experienced the same failure.

Is it possible that scalaj doesn't use the default authenticator when initiating the connect? Were you able to successfully apply your solution?

— Reply to this email directly or view it on GitHub https://github.com/scalaj/scalaj-http/issues/87#issuecomment-173344740.

hoffrocket avatar Jan 20 '16 20:01 hoffrocket

Ran a few tests and can confirm that the Authenticator isn't being called. Does scalaj use its own classloader or anything else unrelated?

crockpotveggies avatar Jan 20 '16 21:01 crockpotveggies

No, it's not really doing any crazy tricks. The source is pretty straightforward: https://github.com/scalaj/scalaj-http/blob/master/src/main/scala/scalaj/http/Http.scala Unfortunately, I won't have a chance to look into this in more detail until next week. Please let me know if you discover anything.

On Wed, Jan 20, 2016 at 4:58 PM, crockpotveggies [email protected] wrote:

Ran a few tests and can confirm that the Authenticator isn't being called. Does scalaj use its own classloader or anything else unrelated?

— Reply to this email directly or view it on GitHub https://github.com/scalaj/scalaj-http/issues/87#issuecomment-173374644.

hoffrocket avatar Jan 21 '16 15:01 hoffrocket

The nice folks at WonderProxy took a stab at the problem, and apparently had some results. The difference between their solution and my solution, is that they changed the default authenticator after instantiating an Http object.

import scalaj.http._
import java.net.{Authenticator,PasswordAuthentication}

val request = Http("https://github.com/").proxy("buffalo.wonderproxy.com", 10000)
Authenticator.setDefault(new Authenticator() {
  override def getPasswordAuthentication(): PasswordAuthentication = {
    new PasswordAuthentication( "user", "password".toCharArray())
  }
})
println(request.asString.body)

My solution was doing the opposite. Any ideas why this seems to matter?

crockpotveggies avatar Jan 21 '16 16:01 crockpotveggies

No, I wouldn't think it would matter. Building the HttpRequest doesn't actually do anything, it just stores some state in a POJO. If you reverse things now do you still see the problem? Maybe something else was going on?

On Thu, Jan 21, 2016 at 11:15 AM, crockpotveggies [email protected] wrote:

The nice folks at WonderProxy took a stab at the problem, and apparently had some results. The difference between their solution and my solution, is that they changed the default authenticator after instantiating an Http object.

import scalaj.http._ import java.net.{Authenticator,PasswordAuthentication}

val request = Http("https://github.com/").proxy("buffalo.wonderproxy.com", 10000) Authenticator.setDefault(new Authenticator() { override def getPasswordAuthentication(): PasswordAuthentication = { new PasswordAuthentication( "user", "password".toCharArray()) } }) println(request.asString.body)

My solution was doing the opposite. Any ideas why this seems to matter?

— Reply to this email directly or view it on GitHub https://github.com/scalaj/scalaj-http/issues/87#issuecomment-173621120.

hoffrocket avatar Jan 21 '16 18:01 hoffrocket

Going to test this later today when I have the time, but I'm starting to wonder if this could have been at all affected by dependency injection. I'm going to have to think about our application architecture through the whole stack.

crockpotveggies avatar Jan 21 '16 18:01 crockpotveggies

Closing. plz re-open if there is still an issue.

hoffrocket avatar Mar 02 '16 03:03 hoffrocket

I still have this problem. I am using the code that suggested @crockpotveggies but i still recive this error: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required"

This is the code i used:

import scalaj.http._
import java.net.{Authenticator,PasswordAuthentication}

val request = Http("https://github.com/").proxy("buffalo.wonderproxy.com", 10000)
Authenticator.setDefault(new Authenticator() {
  override def getPasswordAuthentication(): PasswordAuthentication = {
    new PasswordAuthentication( "user", "password".toCharArray())
  }
})
println(request.asString.body)

raxkin avatar Mar 02 '17 14:03 raxkin

Does the equivalent curl command work for you? I don't have access to an authenticated proxy to test myself curl -v -U user:password -x buffalo.wonderproxy.com:10000 --url http://httpbin.org/

hoffrocket avatar Mar 02 '17 22:03 hoffrocket

Also, going back the originally reported problem, I don't see the Proxy-Authorization header being stripped off the request on my systems OSX Java 1.8.0_92 and on Linux Java 1.7.0_79

I even setup a lightweight proxy server using nginx to test out. And then executed this request to verify all the headers being passed along:

scala> val request = Http("http://httpbin.org/get").proxy("localhost", 8888).header("Proxy-Authorization", "Basic dGVzdDp0ZXN0").auth("test","test").asString
request: scalaj.http.HttpResponse[String] = 
HttpResponse({
  "args": {}, 
  "headers": {
    "Accept": "text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2", 
    "Accept-Encoding": "gzip,deflate", 
    "Authorization": "Basic dGVzdDp0ZXN0", 
    "Host": "httpbin.org", 
    "Proxy-Authorization": "Basic dGVzdDp0ZXN0", 
    "Proxy-Connection": "keep-alive", 
    "User-Agent": "scalaj-http/1.0"
  }, 
  "origin": "127.0.0.1, 67.151.197.146", 
  "url": "http://httpbin.org/get"
}
...

nginx.conf:

worker_processes  1;
daemon off;
error_log /dev/stdout info;
events {
    worker_connections  1024;
}

http {
  access_log /dev/stdout;

  server {
    listen 8888;
    server_name localhost;


    location / {
      resolver 8.8.8.8;
      proxy_pass http://$http_host$uri$is_args$args;
      proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
      proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
      proxy_set_header  X-Forwarded-Proto $scheme;
      proxy_read_timeout                  900;
    }
  }
}

hoffrocket avatar Mar 03 '17 00:03 hoffrocket

Yes, the curl command working for me. I also tried sending the Proxy-Authorization header but it don't work for me.

raxkin avatar Mar 03 '17 09:03 raxkin

What OS and Java version are you using?

hoffrocket avatar Mar 03 '17 15:03 hoffrocket

OS: Linux dell 4.9.0-1-amd64 #1 SMP Debian 4.9.2-2 (2017-01-12) x86_64 GNU/Linux JAVA: 1.8.0_121

raxkin avatar Mar 06 '17 08:03 raxkin

Hi, can you try cloning the source and running ./sbt test ? I added some unit tests to verify that the Proxy-Authorization was being passed through correctly.

hoffrocket avatar Mar 06 '17 13:03 hoffrocket

Failed 2 tests. This is the output of the 2 fails:

[error] Test scalaj.http.HttpTest.proxyNoAuthTest failed: java.net.SocketTimeoutException: Read timed out, took 5.043 sec
[error]     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[error]     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[error]     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[error]     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[error]     at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1926)
[error]     at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1921)
[error]     at java.security.AccessController.doPrivileged(Native Method)
[error]     at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1920)
[error]     at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1490)
[error]     at sun.net.www.protocol.http.HttpURLConnection.access$200(HttpURLConnection.java:91)
[error]     at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1466)
[error]     at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1464)
[error]     at java.security.AccessController.doPrivileged(Native Method)
[error]     at java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:782)
[error]     at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1463)
[error]     at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
[error]     at scalaj.http.HttpRequest.scalaj$http$HttpRequest$$doConnection(Http.scala:367)
[error]     at scalaj.http.HttpRequest.exec(Http.scala:343)
[error]     at scalaj.http.HttpRequest.asString(Http.scala:490)
[error]     at scalaj.http.HttpTest$$anonfun$proxyNoAuthTest$1$$anonfun$apply$4.apply(HttpTest.scala:331)
[error]     at scalaj.http.HttpTest$$anonfun$proxyNoAuthTest$1$$anonfun$apply$4.apply(HttpTest.scala:330)
[error]     at scalaj.http.HttpTest.makeRequest(HttpTest.scala:43)
[error]     at scalaj.http.HttpTest$$anonfun$proxyNoAuthTest$1.apply(HttpTest.scala:330)
[error]     at scalaj.http.HttpTest$$anonfun$proxyNoAuthTest$1.apply(HttpTest.scala:326)
[error]     at scalaj.http.HttpTest.makeProxiedRequest(HttpTest.scala:104)
[error]     at scalaj.http.HttpTest.proxyNoAuthTest(HttpTest.scala:326)
[error]     ...
[error] Caused by: java.net.SocketTimeoutException: Read timed out
[error]     at java.net.SocketInputStream.socketRead0(Native Method)
[error]     at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
[error]     at java.net.SocketInputStream.read(SocketInputStream.java:171)
[error]     at java.net.SocketInputStream.read(SocketInputStream.java:141)
[error]     at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
[error]     at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
[error]     at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
[error]     at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
[error]     at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
[error]     at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)
[error]     at sun.net.www.protocol.http.HttpURLConnection.access$200(HttpURLConnection.java:91)
[error]     at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1466)
[error]     at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1464)
[error]     at java.security.AccessController.doPrivileged(Native Method)
[error]     at java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:782)
[error]     at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1463)
[error]     at scalaj.http.HttpRequest.scalaj$http$HttpRequest$$doConnection(Http.scala:365)
[error]     ... 64 more
[error] Test scalaj.http.HttpTest.proxyCorrectAuthTest failed: java.net.SocketTimeoutException: Read timed out, took 5.025 sec
[error]     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[error]     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[error]     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[error]     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[error]     at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1926)
[error]     at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1921)
[error]     at java.security.AccessController.doPrivileged(Native Method)
[error]     at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1920)
[error]     at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1490)
[error]     at sun.net.www.protocol.http.HttpURLConnection.access$200(HttpURLConnection.java:91)
[error]     at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1466)
[error]     at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1464)
[error]     at java.security.AccessController.doPrivileged(Native Method)
[error]     at java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:782)
[error]     at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1463)
[error]     at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
[error]     at scalaj.http.HttpRequest.scalaj$http$HttpRequest$$doConnection(Http.scala:367)
[error]     at scalaj.http.HttpRequest.exec(Http.scala:343)
[error]     at scalaj.http.HttpRequest.asString(Http.scala:490)
[error]     at scalaj.http.HttpTest$$anonfun$proxyCorrectAuthTest$1$$anonfun$apply$8.apply(HttpTest.scala:358)
[error]     at scalaj.http.HttpTest$$anonfun$proxyCorrectAuthTest$1$$anonfun$apply$8.apply(HttpTest.scala:357)
[error]     at scalaj.http.HttpTest.makeRequest(HttpTest.scala:43)
[error]     at scalaj.http.HttpTest$$anonfun$proxyCorrectAuthTest$1.apply(HttpTest.scala:357)
[error]     at scalaj.http.HttpTest$$anonfun$proxyCorrectAuthTest$1.apply(HttpTest.scala:353)
[error]     at scalaj.http.HttpTest.makeProxiedRequest(HttpTest.scala:104)
[error]     at scalaj.http.HttpTest.proxyCorrectAuthTest(HttpTest.scala:353)
[error]     ...
[error] Caused by: java.net.SocketTimeoutException: Read timed out
[error]     at java.net.SocketInputStream.socketRead0(Native Method)
[error]     at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
[error]     at java.net.SocketInputStream.read(SocketInputStream.java:171)
[error]     at java.net.SocketInputStream.read(SocketInputStream.java:141)
[error]     at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
[error]     at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
[error]     at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
[error]     at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
[error]     at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
[error]     at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)
[error]     at sun.net.www.protocol.http.HttpURLConnection.access$200(HttpURLConnection.java:91)
[error]     at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1466)
[error]     at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1464)
[error]     at java.security.AccessController.doPrivileged(Native Method)
[error]     at java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:782)
[error]     at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1463)
[error]     at scalaj.http.HttpRequest.scalaj$http$HttpRequest$$doConnection(Http.scala:365)
[error]     ... 64 more
[error] Failed: Total 51, Failed 2, Errors 0, Passed 49
[error] Failed tests:
[error] 	scalaj.http.HttpTest
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful

raxkin avatar Mar 06 '17 15:03 raxkin

Sorry, a recent test code change I made was causing this failure on certain systems. I reverted that change. Could you pull the latest and try again?

hoffrocket avatar Mar 07 '17 14:03 hoffrocket

All the tests success now.

raxkin avatar Mar 07 '17 15:03 raxkin

Great, then the proxying should work from your system. Can you try this from the console:

./sbt console
val request = Http("http://httpbin.org/get").proxy("proxyhost", proxyport).proxyAuth("username", "password").asString

I added a helper method to set the proxy-authorization header.

hoffrocket avatar Mar 07 '17 15:03 hoffrocket

Ok, thanks, i will test it as soon as i can today and give you a feedback.

raxkin avatar Mar 08 '17 11:03 raxkin

I made some tests and this are the results: It give me the error HTTP/1.1 407 Proxy Authentication Required when i try to access to https urls. This returns me the correct result ip: val request = scalaj.http.Http("http://api.ipify.org").proxy("proxyhost, proxyport).proxyAuth("username", "password").asString

This give me the Authentication error: val request = scalaj.http.Http("https://api.ipify.org").proxy("proxyhost, proxyport).proxyAuth("username", "password").asString

Then i assume that https is not supported? I can navigate to this url from the browser and with curl without problems with http and https.

raxkin avatar Mar 08 '17 12:03 raxkin

Hi, In my sbt file I have: libraryDependencies += "org.scalaj" %% "scalaj-http" % "2.3.0" But if I try to compile: val request = Http("http://httpbin.org/get").proxy("proxyhost", 80).proxyAuth("username", "password").asString I get the following error: value proxyAuth is not a member of scalaj.http.HttpRequest

karimagnusson avatar Jun 24 '17 14:06 karimagnusson

yes, same thing, never made it to released version?

clement-bramy avatar Mar 05 '18 21:03 clement-bramy

The proxy thing doesn't work? Does someone have a solution.

abhinavgazta avatar Mar 27 '19 12:03 abhinavgazta

In build.sbt -:
"org.scalaj" %% "scalaj-http" % "2.3.0",

Tried this -: val resp = Http(s"$url").proxy(proxy.host, proxy.port, Proxy.Type.HTTP) .auth(proxy.userName.get, proxy.secret.get) .method("GET").asBytes Always get this, despite trying eveything mentioned in all the above threads. Does someone know what is the solution for this or do we dump scalaj and pick up akka. ?

Unable to tunnel through proxy. Proxy returns "HTTP/1.0 407 Unauthorized" java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.0 407 Unauthorized"

abhinavgazta avatar Mar 27 '19 12:03 abhinavgazta

Hi @abhinavgazta -- can you try upgrading to version "2.4.1" and pass your proxy credentials using a new proxyAuth method instead of auth

hoffrocket avatar Mar 27 '19 13:03 hoffrocket