Containers are terminated with exit codes: {jnlp-slave=127}

[xianfengsong]

trafficstars

{ "name": "jnlp-slave", "state": { "terminated": { "exitCode": 127, "reason": "Error", "startedAt": "2017-11-23T12:53:13Z", "finishedAt": "2017-11-23T12:53:13Z", "containerID": "docker://d24c4181c514913ff4db68eac901871af3b300ea218a32549a99c272ebebb933" } }, "lastState": {}, "ready": false, "restartCount": 0, "image": "docker.io/jenkins/jnlp-slave:latest", "imageID": "docker-pullable://docker.io/jenkins/jnlp-slave@sha256:d73576772018c593de90ceb755fa34c4f135d7fb4fc2e111a0a4f8f814d4c2a5", "containerID": "docker://d24c4181c514913ff4db68eac901871af3b300ea218a32549a99c272ebebb933" }

[xianfengsong]

Nov 23, 2017 12:52:58 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch WARNING: Error in provisioning; slave=KubernetesSlave name: jnlp-l80z4, template=org.csanchez.jenkins.plugins.kubernetes.PodTemplate@5d7f5d41 java.lang.IllegalStateException: Containers are terminated with exit codes: {jnlp-slave=127} at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:184) at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:262) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

[pictolearn]

Did this get resolved yet? I am getting the same issue.

[rsilvareberth]

Import image and build again. The problem disappeared. Root cause unknow.

[devcharted]

Could be the same:

kubectl -n jenkins describe pod build-24691-utest-j0szw-9kz3p
jnlp:
    Container ID:   containerd://beb7b02d213a91d804d4093b2b4dbdad15658b146a5ed481bef8e0e1ba24faba
    Image:          jenkins/inbound-agent:4.3-4-alpine
    Image ID:       docker.io/jenkins/inbound-agent@sha256:578ab6f78b038897820392c3a37725acc82cd792b0d80c8afadebcf3a1f7ced1
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Wed, 13 May 2020 21:32:40 +0200
      Finished:     Wed, 13 May 2020 21:46:27 +0200
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:     500m
      memory:  512Mi
    Environment:
      JENKINS_SECRET:         0bfb6ddcd2dd4e2d132eb28d05d2163878dee1503fb69c5ebbd1695ad01743f5
      JENKINS_TUNNEL:         jenkins:50000
      JENKINS_AGENT_NAME:    build-24691-utest-j0szw-9kz3p
      JAVA_OPTS:              -Xrs -Xms512m -Xmx512m -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90
      JENKINS_NAME:          build-24691-utest-j0szw-9kz3p
      JENKINS_AGENT_WORKDIR:  /home/jenkins/agent
      JENKINS_URL:            https://jenkins
    Mounts:
      /etc/ssl/certs/java from cm-java-cacerts (rw)
      /home/jenkins/agent from workspace-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ffqzp (ro)

$ kubectl -n jenkins logs build-24691-utest-j0szw-9kz3p jnlp                                                                                             [21:55:52]
May 13, 2020 7:32:41 PM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: build-24691-utest-j0szw-9kz3p
May 13, 2020 7:32:41 PM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
May 13, 2020 7:32:41 PM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 4.3
May 13, 2020 7:32:41 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/jenkins/agent/remoting as a remoting work directory
May 13, 2020 7:32:41 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to /home/jenkins/agent/remoting
May 13, 2020 7:32:41 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [https://jenkins/]
May 13, 2020 7:32:41 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
May 13, 2020 7:32:41 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check
May 13, 2020 7:32:41 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
  Agent address: jenkins
  Agent port:    50000
  Identity:      15:ff:8e:1d:7c:f3:fe:e0:dc:f3:66:88:a5:84:42:36
May 13, 2020 7:32:41 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
May 13, 2020 7:32:41 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins:50000
May 13, 2020 7:32:41 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
May 13, 2020 7:32:41 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: 15:ff:8e:1d:7c:f3:fe:e0:dc:f3:66:88:a5:84:42:36
May 13, 2020 7:32:42 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected

[mbarbero]

@devcharted, IIRC exitCode: 127 can be triggered when the jnlp container gets killed by the OOMKiller. You specified -Xms512m -Xmx512m while the container gets memory: 512Mi. This is very risky: it means that container's max memory == JVM max heap. It can trigger the OOM killer as the JVM does not only need heap memory.

My advise: ensure that you run with jdk8u191 or later and use -XX:InitialRAMPercentage, -XX:MaxRAMPercentage, and -XX:MinRAMPercentage to specify your memory limits instead of the the -Xms/-Xmx. jdk8u191 ensures that the container memory will be "read" by the JVM instead of the host's memory (without any parameters to give). -XX:***RAMPercentage will configure the JVM with the % of that memory you want to allocate to heap. Something between 50 and 75 are usually the safe spot. Above is risky.

[micha-eleric]

@mbarbero , please explain "-XX:InitialRAMPercentage, -XX:MaxRAMPercentage, and -XX:MinRAMPercentage"

[MarkEWaite]

@micha-eleric refer to https://dzone.com/articles/difference-between-initialrampercentage-minramperc for a description

[micha-eleric]

@MarkEWaite, thank you, got it, still did not fix my 127 error

[vasily-v-ryabov]

Code 137 is OOM killer. Please make sure you have enough memory in your pod limits/requests. Code 127 relates to line endings: https://stackoverflow.com/q/27820268/3648361

[vasily-v-ryabov]

General way to google it is "docker exit code 127" or similar.