Error resolving proxy FQDN

[CarlInglisBJSS]

Jenkins version: 2.289.1 Plugin-version: 1.42

Problem: Trying to use s3DoesObjectExist through a proxy is not working due to an error resolving the proxy by hostname

Background: I have developed a number of pipeline processes which have worked fine with our transparent proxy (i.e. not needing to set the http_proxy etc. group of environment variables). For reasons involving another project our jenkins workers are now going to be behind a client proxy (i.e. setting those environment variables).

I had hoped that I would be able to just set the environment variables and all would be well, however that's turned out not to be the case.

Bucket name and other identifying marks are redacted for client confidentiality reasons.

Result of set | grep -i http (snipped to just relevant entries)

15:18:39  HTTP_PROXY=http://squid.[rest-of-fqdn]:3128/
15:18:39  https_proxy=http://squid.[rest-of-fqdn]:3128/

Using aws s3 via sh to show connectivity

15:18:39  + aws s3 cp s3://[bucketname]/latest.txt .
15:18:40  Completed 29 Bytes/29 Bytes (307 Bytes/s) with 1 file(s) remaining
download: s3://[bucketname]/latest.txt to ./latest.txt     

Actual error including stacktrace

15:18:40  [Pipeline] s3DoesObjectExist
15:18:40  Searching s3://[bucketname] for object:'latest.txt'
15:18:40  Retrieving credentials from node.
15:19:18  [Pipeline] }
15:19:18  [Pipeline] // script
15:19:18  [Pipeline] }
15:19:18  [Pipeline] // stage
15:19:18  [Pipeline] }
15:19:18  [Pipeline] // node
15:19:18  [Pipeline] End of Pipeline
15:19:18  java.net.UnknownHostException: squid.[rest-of-fqdn].: Name or service not known
15:19:18  	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
15:19:18  	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
15:19:18  	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
15:19:18  	at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
15:19:18  	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
15:19:18  	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
15:19:18  	at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)
15:19:18  	at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38)
15:19:18  	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
15:19:18  	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
15:19:18  	at sun.reflect.GeneratedMethodAccessor1928.invoke(Unknown Source)
15:19:18  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
15:19:18  	at java.lang.reflect.Method.invoke(Method.java:498)
15:19:18  	at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
15:19:18  	at com.amazonaws.http.conn.$Proxy134.connect(Unknown Source)
15:19:18  	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:401)
15:19:18  	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
15:19:18  	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
15:19:18  	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
15:19:18  	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
15:19:18  	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
15:19:18  	at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
15:19:18  	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1331)
15:19:18  	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
15:19:18  Caused: com.amazonaws.SdkClientException: Unable to execute HTTP request: squid.[rest-of-fqdn]: Name or service not known
15:19:18  	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207)
15:19:18  	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153)
15:19:18  	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
15:19:18  	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
15:19:18  	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
15:19:18  	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
15:19:18  	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
15:19:18  	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
15:19:18  	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
15:19:18  	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5445)
15:19:18  	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5392)
15:19:18  	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1367)
15:19:18  	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1341)
15:19:18  	at com.amazonaws.services.s3.AmazonS3Client.doesObjectExist(AmazonS3Client.java:1422)
15:19:18  	at de.taimos.pipeline.aws.S3DoesObjectExistStep$Execution.run(S3DoesObjectExistStep.java:115)
15:19:18  	at de.taimos.pipeline.aws.S3DoesObjectExistStep$Execution.run(S3DoesObjectExistStep.java:93)
15:19:18  	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
15:19:18  	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
15:19:18  	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
15:19:18  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
15:19:18  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
15:19:18  	at java.lang.Thread.run(Thread.java:748)

What I've tried:

• just squid instead of the FQDN - same issue but does resolve in an sh step
• with and without trailing .s - same issue
• curl - worked

I'd appreciate any thoughts on what else I can dig out and/or try as I really don't want to have to rewrite everything I've done with this plug in to use sh calls.

[CarlInglisBJSS]

On suggestion which has arisen; is the DNS lookup for the host and proxy being done on the worker node or the leader?

If the latter, then that's what's causing the problem as the worker and the leader are in different VPCs with different DNS Zones.