Consolidated proxy support

[biddster]

This is a follow on from http://code.google.com/p/jclouds/issues/detail?id=122.

We require full proxy support across all socket connections so that we can use jClouds from behind corporate proxies.

This includes, but is not limited to:

• all jsch usage (we have to tunnel ssh through our corporate proxy using nc and corkscrew).
• httpcommandexecutors
• socket open implementations.

Adrian has already started work here...

https://github.com/jclouds/jclouds/commits/consolidated-proxy

[biddster]

I've given this a poke - haven't had time to debug but here's the first stack trace that's popped up

java.lang.RuntimeException: java.util.concurrent.TimeoutException: Timeout waiting for task. at com.google.common.base.Throwables.propagate(Throwables.java:160) at org.jclouds.location.suppliers.derived.RegionIdsFromRegionIdToURIKeySet.get(RegionIdsFromRegionIdToURIKeySet.java:53) at org.jclouds.location.suppliers.derived.RegionIdsFromRegionIdToURIKeySet.get(RegionIdsFromRegionIdToURIKeySet.java:38) at org.jclouds.util.Suppliers2$SupplierComposition.get(Suppliers2.java:176) at org.jclouds.rest.suppliers.MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier$SetAndThrowAuthorizationExceptionSupplierBackedLoader.load(MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.java:81) at org.jclouds.rest.suppliers.MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier$SetAndThrowAuthorizationExceptionSupplierBackedLoader.load(MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.java:65) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252) at com.google.common.cache.LocalCache.get(LocalCache.java:3990) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) at org.jclouds.rest.suppliers.MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.get(MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.java:130) at org.jclouds.aws.ec2.compute.strategy.AWSEC2ListNodesStrategy.pollRunningInstances(AWSEC2ListNodesStrategy.java:71) at org.jclouds.ec2.compute.strategy.EC2ListNodesStrategy.listDetailsOnNodesMatching(EC2ListNodesStrategy.java:92) at org.jclouds.ec2.compute.strategy.EC2ListNodesStrategy.listDetailsOnNodesMatching(EC2ListNodesStrategy.java:59) at org.jclouds.compute.internal.BaseComputeService.listNodesDetailsMatching(BaseComputeService.java:361) at com.csc.fs.emea.nimbus.provisioning.AwsEC2IaasProvider.getNodesInGroup(AwsEC2IaasProvider.java:477) at com.csc.fs.emea.nimbus.provisioning.Environment.provisionEnvironment(Environment.java:328) at com.csc.fs.emea.nimbus.provisioning.Environment.main(Environment.java:147) Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task. at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:264) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96) at org.jclouds.concurrent.ExceptionParsingListenableFuture.get(ExceptionParsingListenableFuture.java:76) at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:170) at $Proxy74.describeRegions(Unknown Source) at org.jclouds.ec2.suppliers.DescribeRegionsForRegionURIs.get(DescribeRegionsForRegionURIs.java:49) at org.jclouds.ec2.suppliers.DescribeRegionsForRegionURIs.get(DescribeRegionsForRegionURIs.java:36) at org.jclouds.rest.suppliers.MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier$SetAndThrowAuthorizationExceptionSupplierBackedLoader.load(MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.java:81) at org.jclouds.rest.suppliers.MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier$SetAndThrowAuthorizationExceptionSupplierBackedLoader.load(MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.java:65) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252) at com.google.common.cache.LocalCache.get(LocalCache.java:3990) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) at org.jclouds.rest.suppliers.MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.get(MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.java:130) at org.jclouds.location.suppliers.derived.RegionIdsFromRegionIdToURIKeySet.get(RegionIdsFromRegionIdToURIKeySet.java:51) ... 19 more

[jclouds]

can you paste the parameters you are using?

[biddster]

As in JRE args? I'm setting http.proxyHost, http.proxyPort and the corresponding https properties.

[jclouds]

I meant jclouds context properties, as this controls the code that selects proxy details.

props.setProperty(Constants.PROPERTY_PROXY_HOST, yourProxy); .... contextBuider.overrides(props);

On Thu, Dec 6, 2012 at 1:32 AM, biddster [email protected] wrote:

proxyPort

[biddster]

Is there an emoticon for sheepish? Repeating test. SiG.

[biddster]

OK - now we're getting somewhere.

First point of issue is waiting for port 22 after an initial provision. InetSocketAddressConnect has the correct proxy for URI Function injected and uses that in the Socket constructor. However, the proxy object created specifies an HTTP proxy and the Socket constructor only likes proxies of type SOCKS or NO_PROXY. Consequently I get a runtime exception thrown.

Would it be easier to do the whole thing over SOCKS?

[jclouds]

Yeah I agree. @demobox thoughts?

[demobox]

@demobox thoughts?

Not done all that much with Java proxying lately so this is not an "expert opinion". But based on the comments it certainly seems like a good plan. I guess the main question would be: do we feel SOCKS proxies are likely enough in enterprise environments for this to be sufficient?

[codefromthecrypt]

well http proxy can only work on protocols like http and ftp. socks can do TCP, which is what we need for ssh. I'll add a proxy type param

[biddster]

The enterprise question is a good one. I guess I'm a statistically meaningful sample of 1!

In all the places I've ever worked or visited, all the proxies have been transparent. It's only my current employer which doesn't have one.

@adriancole I'll pick this up on Monday when I'm back in the office and try it all over socks.