Issues connecting maven repos with fabric8 on OpenShift using Corporate Proxy

[vpiduri]

we are running fabric8 on OpenShift with access to internet URLs using Corporate Proxy. As per the suggestion given by one of the fabric8 developers, we did updated jenkins-maven-settings secret to add corporate proxy settings. Then restarted the RC/PODs, still it does not work. So i went ahead and made a change to maven settings xml to make it invalid xml to see if that breaks the mvn stuff, apparently fabric8 does not use that XML at all because still fabric8-forge tries to connect to maven repo or I need to restart something else after I update the jenkins-maven-settings secret. Also I have added HTTP_PROXY, HTTPS_PROXY and NO_PROXY to fabric8-forge deployment configuration file. Still no luck

if anyone come across this kind of issue or you have inputs that would be great. Really appreciate your help org.jboss.forge.furnace.manager.maven.MavenOperationException: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact io.fabric8.archetypes:spring-boot-webmvc-archetype:jar:2.2.164 from/to central (http://repo1.maven.org/maven2): Error transferring file: Connection refused from http://repo1.maven.org/maven2/io/fabric8/archetypes/spring-boot-webmvc-archetype/2.2.164/spring-boot-webmvc-archetype-2.2.164.jar

[davsclaus]

try to reach out to ggastalid on #forge irc chat room on freenode, he is a core forge developer and may know how these forge commands works with maven / proxy settings.

[rawlingsj]

I've just had a quick look and the maven settings.xml that's added to fabric8 forge is under /root/.m2 but forge is running as the jboss user. Can you try to copy https://raw.githubusercontent.com/fabric8io/fabric8-forge/master/fabric8-forge/src/main/fabric8/m2/settings.xml into /opt/jboss/.m2?

You can do that using

oc exec -ti fabric8-forge-xxxx bash

[jimmidyson]

forge is running as the jboss user

Doesn't this run as an arbitrary uid on OpenShift? If so we might need to run with nss_wrapper as we've had to do with other images (gogs, jenkins) to map arbitrary uid to jboss user in container & we can set up settings.xml consistently in that case. I haven't checked what it's doing so sorry if this isn't relevant.

[rawlingsj]

Yeah we may well need to do that, also we could mount the existing jenkins maven secret into the forge pod? That's probably better all round?

[jimmidyson]

If the shared secret is required then yes mount it in both pods. I think you'll still need to mount to correct location & if you see https://github.com/fabric8io/fabric8-forge/blob/ddd88c79e7125507527a9986bbf008d8abfd6817/fabric8-forge/src/main/fabric8/assembly.xml#L42-L50 this will only work for root user.

[vpiduri]

Guys, Thanks for your help. I am able to past that issue now. Now I am at a point I can check in the project into gogs. Now I am facing couple of below issues

1. Failed to pull from the remote git repo with credentials UsernamePasswordCredentialsProvider{user: gogsadmin, password length: 0} due: https://github.com/fabric8io/jenkins-pipeline-library.git: cannot open git-upload-pack. This exception is ignored.
2. Failed to command remote repo http://gogs.cp.cnapps-dev..com/gogsadmin/boot12.git due: http://gogs.cp.cnapps-dev.gbt.gbtad.com/gogsadmin/boot12.git: cannot open git-upload-pack org.eclipse.jgit.api.errors.TransportException: http://gogs.cp.cnapps-dev..com/gogsadmin/boot12.git: cannot open git-upload-pack

Do we need to pass the credentials to pull from github? Not necessary right, even I pass the credentials they would be ignored. I was able to get it working on my Vagrant box.

Is this something to do with git http.sslVerify? If that is the case how do I update the settings in the fabric8 forge. Please provide your input.

Appreciate your input on this.

[jstrachan]

1. isn't an issue - credentials are not required to clone a public http git repo
2. yes we need to pass credentials to be able to push to git repos (often http / https is free to clone though). This was recently fixed - I wonder if you can check you're using 2.2.175 of the fabric8-console (see the fabric8 pod and check its image version). See this issue: https://github.com/fabric8io/fabric8/issues/6293#issuecomment-248043071

[vpiduri]

Yes, I have updated to use 2.2.177 but still I am facing same issue. I think it could be because of some internal proxy issue, if I am using service IP to clone manually on the OpenShift Server it works fine. Do you know how can I update IP instead of address, that is like below. I tried updating on the core source page but it is not getting reflected

git clone http://172.30.178.212/gogsadmin/boot20.git

instead of http://gogs.cp.cnapps-dev.***.com/gogsadmin/boot20.git

[root@* vpiduri]# git clone http://gogs.cp.cnapps-dev..com/gogsadmin/boot20.git Cloning into 'boot20'... fatal: unable to access 'http://gogs.cp.cnapps-dev.***.com/gogsadmin/boot20.git/': Recv failure: Connection reset by peer.

[jstrachan]

@vpiduri whats the output of:

oc export route gogs

just so we can see what the Route is setup as. Can you access any other routes in your openshift install? e.g. the fabric8 console via the same domain http://fabric8..cp.cnapps-dev.***.com/?

[jstrachan]

you are running the openshift router too right?

[vpiduri]

@jstrachan yes I am running OpenShift router as well, I can access the routes from browser just fine, just not able to connect from the Openshift nodes.

Here is the output, the gogs URL would be configured some where in fabric8 forge right? I just wanted to update that to use it with cluster IP

apiVersion: v1 kind: Route metadata: creationTimestamp: null labels: provider: fabric8 name: gogs spec: host: gogs.cp.cnapps-dev.****.com to: kind: Service name: gogs status: ingress:

• conditions:
  • lastTransitionTime: 2016-09-05T19:58:40Z status: "True" type: Admitted host: gogs.cp.cnapps-dev.****.com routerName: router-1

[jstrachan]

@vpiduri sounds like a DNS issue if containers inside openshift cannot access external DNS names from the Routes. You might wanna raise an openshift issue to help them figure out how to get your openshift install working properly with DNS

[vpiduri]

@jstrachan Thanks for the inputs - I am excited to say that I am getting closer, Yes recently our F5 team made a change on the load balancer which caused issues to resolve the DNS for *.gbtad.com. They have fixed issue now and I am able to past that issue.

I just don't see the pipelines on the main page once I configure the project, I see following log lines please see if you can point me in the right direction.

How do I disable username/password for the github pull? I cant figure out where this username getting added. The weird thing is that I dont see any issues running on my local vagrant opensshift.

Payload: {"namespace":"cicd","projectName":"boot21","resource":"","inputList":[{"copyPipelineToProject":true}]} Sep 20, 2016 9:28:16 PM io.fabric8.forge.devops.DevOpsEditStep getPipelines WARNING: No jenkinsWorkflowFolder! jar jar Sep 20, 2016 9:28:16 PM io.fabric8.forge.devops.DevOpsEditStep getPipelines WARNING: No jenkinsWorkflowFolder! No merge spec for branch.master.merge in the git repository at /tmp/fabric8-forge/jenkinsWorkflows/.git so not doing a pull Stashing local changes to the repo Performing a pull in git repository /tmp/fabric8-forge/jenkinsWorkflows/.git on remote URL: https://github.com/fabric8io/jenkins-pipeline-library.git Using UsernamePasswordCredentialsProvider{user: gogsadmin, password length: 0} jar Failed to pull from the remote git repo with credentials UsernamePasswordCredentialsProvider{user: gogsadmin, password length: 0} due: null. This exception is ignored. java.lang.NullPointerException jar

[jstrachan]

@vpiduri no user/pwd is required to clone the git repo at https://github.com/fabric8io/jenkins-pipeline-library.git - I wonder if the HTTP proxy is causing the issue? Could you show the rest of that NullPointerException stack trace to figure out whats failing & where?

So you can create projects OK on the Team Dashboard via the Create App -> Create New App UI right? Its just once you've created the project you don't see anything showing on the Pipeline wizard page right?

Note that the first time you ever install fabric8 it can take a little while for that page to populate itself; as the underlying nexus / fabric8-forge often have to download a bunch of stuff. I'm wondering if the HTTP proxy stuff isn't working inside fabric8-forge / nexus to download the necessary stuff for the project.

Could you attach the log of the fabric8-forge pod?

[vpiduri]

@jstrachan - OK I restarted forge pod and configured a spring boot project on the team dash board using create new app UI. Surprisingly now I dont see any issues and I get the pipelines. I selected one of the pipeline and triggered build. But I am facing issues checkout the project from gogs

Would be an issue because the build triggered as anonymous user? Do I need to change some secret to tigger build by a specific user? PFA the build logs below. The build spins for more than 15 mins or so.. Also find the entire forge logs. forge.txt

**Started by user anonymous

Cloning the remote Git repository

Cloning repository http://gogs.cp.cnapps-dev.gbt.gbtad.com/gogsadmin/boot25.git

/usr/bin/git init /var/jenkins_home/workspace/boot25@script # timeout=10

Fetching upstream changes from http://gogs.cp.cnapps-dev.gbt.gbtad.com/gogsadmin/boot25.git

/usr/bin/git --version # timeout=10

/usr/bin/git -c core.askpass=true fetch --tags --progress http://gogs.cp.cnapps-dev.gbt.gbtad.com/gogsadmin/boot25.git +refs/heads/:refs/remotes/origin/

/usr/bin/git config remote.origin.url http://gogs.cp.cnapps-dev.gbt.gbtad.com/gogsadmin/boot25.git # timeout=10

/usr/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10

/usr/bin/git config remote.origin.url http://gogs.cp.cnapps-dev.gbt.gbtad.com/gogsadmin/boot25.git # timeout=10

Fetching upstream changes from http://gogs.cp.cnapps-dev.gbt.gbtad.com/gogsadmin/boot25.git

/usr/bin/git -c core.askpass=true fetch --tags --progress http://gogs.cp.cnapps-dev.gbt.gbtad.com/gogsadmin/boot25.git +refs/heads/:refs/remotes/origin/

/usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10

/usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10

Checking out Revision d52fc0c071911bc5b1e25b20ee62ac6a9e3670ee (refs/remotes/origin/master)

/usr/bin/git config core.sparsecheckout # timeout=10

/usr/bin/git checkout -f d52fc0c071911bc5b1e25b20ee62ac6a9e3670ee

First time build. Skipping changelog.

[Pipeline] node

Still waiting to schedule task

Waiting for next available executor**

[vpiduri]

@jstrachan It looks like the source code has been pulled down by Jenkins, but while executing the pipeline I have issues.

Here is the pipeline for the project

#!/usr/bin/groovy def failIfNoTests = "" try { failIfNoTests = ITEST_FAIL_IF_NO_TEST } catch (Throwable e) { failIfNoTests = "false" } def localItestPattern = "" try { localItestPattern = ITEST_PATTERN } catch (Throwable e) { localItestPattern = "*KT" } def versionPrefix = "" try { versionPrefix = VERSION_PREFIX } catch (Throwable e) { versionPrefix = "1.0" } def canaryVersion = "${versionPrefix}.${env.BUILD_NUMBER}" def utils = new io.fabric8.Utils() node { def envStage = utils.environmentNamespace('staging') git 'http://gogs.cp.cnapps-dev.gbt.gbtad.com/gogsadmin/boot26.git' echo 'NOTE: running pipelines for the first time will take longer as build and base docker images are pulled onto the node' kubernetes.pod('buildpod').withImage('fabric8/maven-builder') .withPrivileged(true) .withHostPathMount('/var/run/docker.sock','/var/run/docker.sock') .withHostPathMount('/root/.mvnrepository','/var/lib/maven/repository') .withEnvVar('DOCKER_CONFIG','/home/jenkins/.docker/') .withSecret('jenkins-docker-cfg','/home/jenkins/.docker') .withSecret('jenkins-maven-settings','/root/.m2') .withServiceAccount('jenkins') .inside { stage 'Canary Release' mavenCanaryRelease{ version = canaryVersion } stage 'Integration Test' mavenIntegrationTest{ environment = 'Testing' failIfNoTests = localFailIfNoTests itestPattern = localItestPattern } stage 'Rolling Upgrade Staging' kubernetesApply(environment: envStage) } }

[vpiduri]

@jstrachan Did you get a chance to look at this? Appreciate your help

[rawlingsj]

@vpiduri can you check the jenkins logs under http://jenkins.your.domain.io/log/all and see if there's any errors?

The last line Waiting for next available executor** suggests there's an issue with jenkins asking kubernetes for the build pod to be scheduled.

[vpiduri]

@rawlingsj Thanks for the input. I see a lot of , OpenShift or Kubernetes running on this host, how do I update from default one? Thanks again for your help

https://gtwtdlaposhv01.***.com:8443

Caused by: java.net.UnknownHostException: kubernetes.default

Sep 24, 2016 2:34:45 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision Excess workload after pending Spot instances: 3 Sep 24, 2016 2:34:45 AM WARNING org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision Failed to count the # of live instances on Kubernetes io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred. at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53) at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:369) at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:52) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:420) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:301) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:700) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:305) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:58) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:795) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.ExecutionException: java.net.UnknownHostException: kubernetes.default at com.ning.http.client.providers.netty.future.NettyResponseFuture.abort(NettyResponseFuture.java:231) at com.ning.http.client.providers.netty.request.NettyRequestSender.abort(NettyRequestSender.java:420) at com.ning.http.client.providers.netty.request.NettyRequestSender.sendRequestWithNewChannel(NettyRequestSender.java:288) at com.ning.http.client.providers.netty.request.NettyRequestSender.sendRequestWithCertainForceConnect(NettyRequestSender.java:140) at com.ning.http.client.providers.netty.request.NettyRequestSender.sendRequest(NettyRequestSender.java:115) at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.execute(NettyAsyncHttpProvider.java:87) at com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:517) at com.ning.http.client.AsyncHttpClient$BoundRequestBuilder.execute(AsyncHttpClient.java:229) at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:364) ... 15 more Caused by: java.net.UnknownHostException: kubernetes.default at java.net.InetAddress.getAllByName0(InetAddress.java:1280) at java.net.InetAddress.getAllByName(InetAddress.java:1192) at java.net.InetAddress.getAllByName(InetAddress.java:1126) at java.net.InetAddress.getByName(InetAddress.java:1076) at com.ning.http.client.NameResolver$JdkNameResolver.resolve(NameResolver.java:28) at com.ning.http.client.providers.netty.request.NettyRequestSender.remoteAddress(NettyRequestSender.java:356) at com.ning.http.client.providers.netty.request.NettyRequestSender.connect(NettyRequestSender.java:367) at com.ning.http.client.providers.netty.request.NettyRequestSender.sendRequestWithNewChannel(NettyRequestSender.java:281) ... 21 more

[vpiduri]

@rawlingsj I was able to resolve this issue by adding KUBERNETES_MASTER and KUBERNETES_DOMAIN in the jenkins runtime Config and Deployment config.

However I am getting below exception for the following pipelines. Install : Event created, where do I find what is the correct tool name? ERROR: No tool named maven-3.3.1 found Finished: FAILURE Deploy : Could not pull Dependencies from Nexus, to fix this I would need to update /root/.m2/settings.xml with Proxy. But I cant change it as it was created by root user, and I cant sudo as root from JBOSS user, any input how can I update? BuildImage : Found Elasticsearch server, sending:{"timestamp":"2016-09-24T21:51:29.841+0000","build_number":1,"app":"boot4","build_result":"FAILURE","start_time":"2016-09-24T21:51:29.842+0000","duration":0,"env_vars":{"BUILD_DISPLAY_NAME":"#1","BUILD_ID":"1","BUILD_NUMBER":"1","BUILD_TAG":"jenkins-boot4-1","CLASSPATH":"","HUDSON_HOME":"/var/jenkins_home","HUDSON_SERVER_COOKIE":"7066ae5f9813e873","JENKINS_HOME":"/var/jenkins_home","JENKINS_SERVER_COOKIE":"7066ae5f9813e873","JOB_NAME":"boot4"},"build_url":"job/boot4/1/","causes":[{"short_description":"Started by user anonymous","user_name":"anonymous"}]} Event created java.net.ConnectException: Connection refused Canary Release and Stage: Found Elasticsearch server, sending:{"timestamp":"2016-09-24T21:06:29.740+0000","build_number":1,"app":"boot1","build_result":"FAILURE","start_time":"2016-09-24T21:06:29.743+0000","duration":0,"env_vars":{"BUILD_DISPLAY_NAME":"#1","BUILD_ID":"1","BUILD_NUMBER":"1","BUILD_TAG":"jenkins-boot1-1","CLASSPATH":"","HUDSON_HOME":"/var/jenkins_home","HUDSON_SERVER_COOKIE":"7066ae5f9813e873","JENKINS_HOME":"/var/jenkins_home","JENKINS_SERVER_COOKIE":"7066ae5f9813e873","JOB_NAME":"boot1"},"build_url":"job/boot1/1/","causes":[{"short_description":"Started by user anonymous","user_name":"anonymous"}]} Event created java.net.ConnectException: Connection refused

I was following this issue https://github.com/fabric8io/fabric8/issues/5930

Thanks for the help

[vpiduri]

@rawlingsj Did you get a chance to look into this? Appreciate your inputs on this

[rawlingsj]

Let me see if I can break it down a little:

Install : Event created, where do I find what is the correct tool name? ERROR: No tool named maven-3.3.1 found

Which pipeline are you selecting? We don't use maven tools in this way, we include the build tools in docker images that are run as part of the pipeline. Here's an example, CanaryReleaseAndStage pipeline which you can select from the new project wizard.

Deploy : Could not pull Dependencies from Nexus, to fix this I would need to update /root/.m2/settings.xml with Proxy. But I cant change it as it was created by root user

The maven settings.xml is mounted as a secret so you should be able to copy this maven settings.xml and add in your proxy details then cat mvnsettings.xml | openssl base64. Copy the result and replace the data value in the secret using

kubectl edit jenkins-maven-settings

The last two errors, I'm wondering, is elasticsearch running in the namespace? It's not needed BTW, if it's found then it will post these events. Also worth checking if you have an old elasticsearch kubernetes service but no running pods.

kubectl get svc; kubectl get pods; 

If you have delete the elasticsearch service or deploy the logging app so that elasticsearch runs.

[jstrachan]

BTW this error looks pretty serious:

Caused by: java.net.UnknownHostException: kubernetes.default

that sounds like DNS can't find the kubernetes service in the default namespace; which should always work really. Are you sure you've setup OpenShift DNS OK?

[vpiduri]

@jstrachan That has been fixed by adding the KUBERNETES_MASTER and KUBERNETES_DOMAIN in the jenkins runtime Config and Deployment config.

[rawlingsj]

@vpiduri out of interest what value did you change KUBERNETES_MASTER to? We originally had kubernetes.default.svc but with clusters provisioned by stackpoint cloud we had a problem with a cert so as a workaround changed to use kubernetes.default - does this not work for you?

[vpiduri]

@rawlingsj I changed this value to https://gtwtdlaposhv01.***.com:8443 on this server the OpenShift/Kubernetes master has been running. I tried kubernetes.default as well that does not work, do we need to import a certificate or something if we are calling https URL

[lovoni]

Really appreciate the effort and great job you are doing to help people. But I just cannot understand why common things like configuring HTTP proxies are not documented for tools like fabric8 forge? Just let us know what environment variables or system properties to set. Just that!