Expand proxy support: Authenticated Proxy and Websockets (`HTTP CONNECT` tunnel)

[mattheworiordan]

A customer has requested the ability to set the proxy manually for HTTP and WebSocket connections.

We need to add this functionality to the library.

┆Issue is synchronized with this Jira Task by Unito

[paddybyers]

A few requirements questions:

• for HTTP proxy, I assume we want support for an explicit proxy, which supports HTTP, or HTTPS as a MITM proxy;
• do we need to support configuration of proxy host and port explicitly in the client API which targets only the Ably host, or should we rely on the JVM-wide system properties? If the latter, do we populate those properties based on given client options, or is that configured outside our control?
• do we need to support proxy authentication? If so, the same questions apply as above: will we use the JVM-wide properties, or enable it only for proxied connections to the Ably host?
• is there any requirement to support tunnelling?

For websockets, would the aim be to support explicit proxying in the same way?

If implementing comet as a fallback to websockets, assuming that the proxy cannot support websockets, will we be able to support streamed comet or must it be polled? If we support both, do we have to detect which will work, or can we rely on a client configuration?

[mattheworiordan]

@paddybyers I will ask the customer now to see if he can comment to help direct us.

[Frintrop]

• yes. explicit proxy ( java.net.Proxy object?) -not system wide proxy. some companies have more than one proxy. setter i.e. as java.net.Proxy object ) to connect method. proxy authentication would be nice, too. -for websockets, too. 50 percent of proxies don´t support websocket, 50 percent works. seamly fallback to comet first and than polled would be fine. rely on a client configuration may be possible as we have a b2b applicaiton and not c2b..

Might this helpful?

[Frintrop]

Think wasync (configured with async http client ) from atmosphere framkework supports this setup ...

[paddybyers]

explicit proxy ( java.net.Proxy object?) -not system wide proxy

Yes, agreed, this should be used, configured by proxy options given when initialising the Ably library.

proxy authentication would be nice, too

This seems to be slightly more problematic using HttpURLConnection because the way to do this seems to be with Authenticator.setDefault(), which sets a global handler, not one for a specific context. We need to look into this further.

for websockets, too. 50 percent of proxies don´t support websocket, 50 percent works

OK, we will look into this.

fallback to comet first and than polled would be fine. rely on a client configuration may be possible as we have a b2b applicaiton and not c2b..

OK. We would prefer to make it fallback seamlessly, but an interim solution may be achievable more quickly by having an explicit transports = new String[]{ "comet" } so we avoid the complications of implementing the fallback sequence in the first instance.

[mattheworiordan]

@gokhanbarisaker I have added Proxy support to Ruby, but unfortunately it's blocked by a WebSocket issue so I've only been able to test the HTTP parts. See https://github.com/ably/ably-ruby/pull/84

We would like to do the same for the Java library, see @paddybyers branch at https://github.com/ably/ably-java/tree/proxy-support. Can you add suitable test coverage to his work, and:

• add support for java.net.Proxy, initialised using host/port configured with new members in ClientOptions. The exact options and naming we'll review in your PR so don't worry too much about what you choose.
• Proxy settings will apply to all HTTP requests
• Please ensure you have tests against sandbox-proxy.ably.io
• Once done, we need to look further into how to enable proxy authentication without it being a global setting

Later we'll then need to address how this applies to WebSockets, and https://github.com/ably/ably-java/issues/121 will address fallback Comet support.

[paddybyers]

We would like to do the same for the Java library

See the latest commits to that branch.

add support for java.net.Proxy, initialised using host/port configured with new members in ClientOptions. The exact options and naming we'll review in your PR so don't worry too much about what you choose.

This is done.

Proxy settings will apply to all HTTP requests

I've added ProxyOptions.nonProxyHosts as an array of regexp strings for hostnames that are not to be proxied.

Please ensure you have tests against sandbox-proxy.ably.io

I think this is the first thing to look at: get some tests running against that (without authentication initially); we should decide what constitutes appropriate coverage. There should be a http and async http request which successfully uses a configured proxy, and another pair of requests that bypass the proxy based on a nonProxyHosts configuration. We need to work out how we would know that the proxy was or was not used.

we need to look further into how to enable proxy authentication without it being a global setting

Having thought about this a bit more this should be simple enough, but we'll wait until the proxy is enabled for authentication to try things out.

[mattheworiordan]

We need to work out how we would know that the proxy was or was not used.

Easy enough, see https://github.com/ably/ably-ruby/pull/84/commits/cf437136698f68121ad1482446e6395cc6879433#diff-d4400109d5e5b080196aad253c2e7f4fR39 and https://github.com/ably/ably-ruby/pull/84/commits/cf437136698f68121ad1482446e6395cc6879433#diff-d4400109d5e5b080196aad253c2e7f4fR72. It checks that the proxy fails when the details are incorrect or TLS was supposed to enabled etc.

[paddybyers]

@gokhanbarisaker: please see https://github.com/ably/ably-java/commit/67b59617831a7034d88e95a6ee1cef128c4e77ab

I've added auth support and a few initial tests to run against sandbox-proxy.ably.io.

[paddybyers]

Reopening after Github auto-closed.

The current shortfalls are:

• no support for auth when using TLS. This is because although we set the Proxy-Authorization header on a connection following a 407, it is not added to the CONNECT request. We can set a global Authenticator which might fix the issue, but that would be a temporary fix because it's not really acceptable for us to set a global Authenticator. I can't see a way of us setting a per-connection Authenticator using the HttpURLConnection API so ultimately we would need to go another route, either to use an alternative library, eg okhttp
• no support for websockets. This is because the websocket library we currently use does not support CONNECT and tunnelling. (It does support a SOCKS proxy). To address this we would either need to add support to that library, or switch to an alternative websocket library. okhttp also supports ws.
• digest auth not working. It is implemented but so far not passing tests.

I think we've nearly gone as far as we can with this without looking again at the http API we use. I do think as a short term improvement we should set a global authenticator if proxy auth is configured, and probably add some more tests.

[shailendher]

Hi,

I understand it's been a while since this issue had any activity. However, here is my experience regarding usage of proxy for websocket connections: I found the following code snippet to work behind proxies: https://github.com/TooTallNate/Java-WebSocket/pull/205#issuecomment-145865275

I made a barebones test by including those changes in io.ably.lib.transport.WebSocketTransport.connect method and it seemed to work.

Perhaps you could reconsider implementing this feature?

[mattheworiordan]

@shailu mind giving us a little background as to what you're doing and why proxy support is needed?

[MathikumarC]

Team,

Is proxy support added for java lib?

Thanks

[paddybyers]

Is proxy support added for java lib?

Proxy support is very limited at the moment: proxies are supported for REST operations only, and only using basic proxy authentication (not digest).

Of course we would like to support proxies fully but to go beyond what's currently supported didn't appear to be straightforward at the time, but if there is sufficient interest we could look again at it.

[paddybyers]

to go beyond what's currently supported didn't appear to be straightforward at the time

From memory the issues were:

• the HttpURLConnection primitive we use for HTTP doesn't support the CONNECT tunnelling protocol without using a JVM-wide configuration (ie it's not supported on a per-connection basis);

• most proxies don't support websockets so without CONNECT tunnelling it's not possible to support them, and the library doesn't support the comet fallback transport;

• there was some other reason that we couldn't easily support digest auth for proxies so the library currently only supports basic auth.

[paddybyers]

The latest status is that the proxy server for the unit tests has been re-enabled, and the tests have been re-enabled (https://github.com/ably/ably-java/commit/24a99bb3ccddf934cf1ded24231712b55a8f84fd).

The current functionality is:

• HTTP (ie non-TLS) via a proxy, both unauthenticated and using basic and digest auth (non-TLS) are now working;
• HTTPS via a proxy, unauthenticated, is working;
• HTTPS via an authenticated proxy is not supported;
• websockets (ie all realtime connections) via proxy is not supported.

The unsupported features can be supported by implementing a tunnel via HTTP CONNECT; this can happen in two ways:

• migrate away from HttpURLConnection as the HTTP primitive, and use another third-party HTTP client such as okhttp;

• implement a tunnel directly (as prototyped here: https://github.com/ably/ably-java/commit/7686b31ccc35e21ab12c66a45ee5a19a1fd11018).

[tomczoink]

Note: When using the REST API to connect, the protocol is seen as socket when using ProxySelector rather than HTTP.

[ttypic]

Authenticated proxy support and proxy for WebSockets features have been postponed. We will take them into account during the next major release of ably-java, but they are not considered high-priority for us.

[ttypic]

We have decided to start working on improving the proxy implementation