Private Image Repository, Kubernetes, ImagePullBackOff

[tjdurant]

Expected behavior and actual behavior

Hello,

We are attempting to run a pipeline on our k8s cluster.

The code and the docker images are hosted in an enterprise GitLab instance in private repositories.

main.nf:

#!/usr/bin/env nextflow

nextflow.enable.dsl=1

params.greeting  = 'Hello world!' 
greeting_ch = Channel.from(params.greeting) 

process splitLetters { 

    container = "<URL to image in private gitlab registry>"

    input: 
    val x from greeting_ch 

    output: 
    file 'chunk_*' into letters_ch 

    """ 
    printf '$x' | split -b 6 - chunk_  
    """ 
} 

nextflow.config:

k8s {
    namespace                   = "nextflow"
    storageClaimName            = "tommy-pvc"
    storageMountPath            = "/workspace"    
}

command:

nextflow kuberun <URL to repo> -r <revision specified>

output:

Pod started: determined-noyce
N E X T F L O W  ~  version 22.04.0
Launching `<URL to code repo>` [determined-noyce] DSL1 - revision: <revision specified>
[15/04fb61] Submitted process > splitLetters (1)
Error executing process > 'splitLetters (1)'

Caused by:
  K8s pod image cannot be pulled -- rpc error: code = Unknown desc = Error response from daemon: Head <URL API call to image repo>: denied: access forbidden

Command executed:

  printf 'Hello world!' | split -b 6 - chunk_

Command exit status:
  -

Command output:
  (empty)

Work dir:
  /workspace/<user name>/work/15/04fb612f2600f69871e289fac3683f

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

We think this is because we are not authenticating the k8s cluster to access the GitLab instance.

If we do a docker login, either locally or on one of the k8s nodes, we can pull the image.

We might be missing something, a config somewhere, that would authenticate at the cluster-level for access to a private SCM server. For most package managers we have to specify the secret to use to pull this container into k8s. So, we are wondering if there is an option to specify a secret to use across a xnode k8s cluster for nextflow to reference and use.

Happy to provide more information if needed.

Thanks, Tom

Environment

• Nextflow version: 22.04.0

• Java version: openjdk version "11.0.13" 2021-10-19 OpenJDK Runtime Environment JBR-11.0.13.7-1751.21-jcef (build 11.0.13+7-b1751.21) OpenJDK 64-Bit Server VM JBR-11.0.13.7-1751.21-jcef (build 11.0.13+7-b1751.21, mixed mode)

• Local Operating system: macOS

• Local Bash version: zsh 5.8 (x86_64-apple-darwin20.0)

[bentsherman]

Hey Thomas. I've never actually had to pull private images in Nextflow myself, but I believe we just recently implemented this feature.

Going off the following K8s docs: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

It looks like you just need to create a secret with your Docker creds and then use the imagePullSecrets pod option. In Nextflow you can specify this option through the pod directive.

Let me know if that works for you.

[tjdurant]

Hi @bentsherman ,

Thanks for the response, we hadn't seen the pod directive. We gave it a shot this AM and its still giving us issues.

main.nf (we tried it with and without 'secret/'):

#!/usr/bin/env nextflow
nextflow.enable.dsl=2

params.str = 'Hello world!'

process splitLetters {

  pod = [ [imagePullSecret: 'secret/<team name>-gitlab-<username>'], [imagePullPolicy: 'Always'] ]

  container = "<URL to image repository>"

  output:
    path 'chunk_*'

  """
  printf '${params.str}' | split -b 6 - chunk_
  """
}

Output:

❯ nextflow kuberun <url to repo>.git -r <commit sha>
Pod started: stoic-stonebraker
N E X T F L O W  ~  version 22.04.0
Pulling <project name>-5 ...
 downloaded from <url to repo>.git
Launching `<url to repo>` [stoic-stonebraker] DSL2 - revision: <commit sha>
[d5/c77187] Submitted process > splitLetters
Error executing process > 'splitLetters'

Caused by:
  K8s pod image cannot be pulled -- rpc error: code = Unknown desc = Error response from daemon: Head <api call to image repo>: denied: access forbidden

Command executed:

  printf 'Hello world!' | split -b 6 - chunk_

Command exit status:
  -

Command output:
  (empty)

Work dir:
  <workdir path>

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line


Pod running: stoic-stonebraker ... waiting for pod to stop running
Pod stoic-stonebraker has changed from running state [terminated:[exitCode:1, reason:Error, startedAt:2022-04-29T15:28:01Z, finishedAt:2022-04-29T15:28:09Z, containerID:docker://0323af09b491b8fdcc92b3f677b0d7105c61889b5fd7bc0dd6dd82db0a01f9c5]]

Here is the get secret output:

kubectl get secret -n nextflow
NAME TYPE DATA AGE
<team name>-gitlab-<username> kubernetes.io/dockerconfigjson 1 7m54s

Here is the kubectl describe output:

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 16s default-scheduler Successfully assigned nextflow/nf-d5c771878fd3ca1f58512667d40f6f1b to <node>
Normal Pulling 15s kubelet Pulling image "<URL to image repo>"
Warning Failed 15s kubelet Failed to pull image "<URL to image repo>": rpc error: code = Unknown desc = Error response from daemon: Head <API call to image repo>: denied: access forbidden
Warning Failed 15s kubelet Error: ErrImagePull
Normal BackOff 13s kubelet Back-off pulling image "<URL to image repo>"
Warning Failed 13s kubelet Error: ImagePullBackOff

[bentsherman]

Okay, so either we are configuring something incorrectly or Nextflow is not mapping to the correct k8s config. Can you confirm that the task pod has the correct secrets config? It should match the example:

apiVersion: v1
kind: Pod
metadata:
  name: private-reg
spec:
  containers:
  - name: private-reg-container
    image: <your-private-image>
  imagePullSecrets:
  - name: regcred

So it looks like you don't need to prepend secret/. You can obtain the pod yaml with kubectl get pod -o yaml <pod-name>

[bentsherman]

Hi @tjdurant did you figure out your problem? Is this issue resolved?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.