Unable to update to kops 1.23.2 from 1.22 due to private registry configuration

[ryan-dyer-sp]

/kind bug

1. What kops version are you running? The command kops version, will display this information. 1.23.2

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag. 1.21

3. What cloud provider are you using? AWS

4. What commands did you run? What is the simplest way to reproduce this issue? kops replace -f cluster.yaml kops get assets --copy kops update cluster --yes kops rolling-update cluster --yes

5. What happened after the commands executed? First master terminated. New master starts but is unable to start any pods as containerd does not have permissions to pull the private/pause:3.6 image (unauthorized)

6. What did you expect to happen? Works as before

7. Please provide your cluster manifest. Execute kops get --name my.example.com -o yaml to display your cluster manifest. You may want to remove your cluster name and other sensitive information.

8. Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know? I believe the key here is in that we use a private registry and copy assets from the public to our private registry. With kops 1.22, the pause image is not listed as an asset. I'm assuming this means that with these clusters we still pull the pause image from (whichever) public registry instead of our private one. With 1.23, the pause image now shows up as an asset and gets copied to our registry.

  assets:
    containerRegistry: private.domain:5555

[hakman]

@ryan-dyer-sp Most likely, this change will not be reverted and works as intended. The problem is the lack of contained mirror repo authentication. I see the following solutions in this case:

1. Allow pulling the pause image from the private registry without authentication.
2. Use a small AdditionalUserData script that pulls the image using crictl or ctr after kops-configuration has run. This way, containerd should see that it's already there and move on.

We are trying to get to add containerd repo mirror authentication, but not sure when we will get to it. Some discussion about it was in https://github.com/kubernetes/kops/pull/12562#issuecomment-962887836. If you feel that's something you could contribute with a PR would be very welcomed.

[olemarkus]

I think after we have moved to the new host containerd format, we should parse the docker secret and write the credentials to the host file instead of writing out the docker.json. Kubelet has the code for this as that is basically what it does when passing the credentials over CRI. So those interested in this can take a look at the k/k credentialsprovider package, which contains the code for parsing docker files (docker.go) and passing it on to containerd over CRI. Then see how this can be ported to nodeup.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.