Docker Rate limit on docker.io/library/nginx:1.23.0-alpine imagepull

[turbodeploy]

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"): Linux 3.10.0-1160.80.1.el7.x86_64 x86_64 NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Version of Ansible (ansible --version): ansible [core 2.11.11] config file = /opt/kubespray/ansible.cfg configured module search path = ['/opt/kubespray/library'] ansible python module location = /usr/local/lib/python3.6/site-packages/ansible ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections executable location = /usr/local/bin/ansible python version = 3.6.8 (default, Nov 16 2020, 16:55:22) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] jinja version = 2.11.3 libyaml = True

• Version of Python (python --version): Python 3.6.8

Kubespray version (commit) (git rev-parse --short HEAD): Tag: v2.20.0

Output of ansible run:

TASK [download : Set image save/load command for containerd on localhost] ****** ok: [node1] Tuesday 06 December 2022 22:38:23 +0000 (0:00:00.410) 0:12:37.239 ****** Tuesday 06 December 2022 22:38:23 +0000 (0:00:00.232) 0:12:37.472 ******

TASK [download : download_container | Prepare container download] ************** included: /opt/kubespray/roles/download/tasks/check_pull_required.yml for node1 Tuesday 06 December 2022 22:38:23 +0000 (0:00:00.428) 0:12:37.901 ******

TASK [download : check_pull_required | Generate a list of information about the images on a node] *** ok: [node1] Tuesday 06 December 2022 22:38:25 +0000 (0:00:01.895) 0:12:39.796 ******

TASK [download : check_pull_required | Set pull_required if the desired image is not yet loaded] *** ok: [node1] Tuesday 06 December 2022 22:38:25 +0000 (0:00:00.337) 0:12:40.134 ****** Tuesday 06 December 2022 22:38:26 +0000 (0:00:00.126) 0:12:40.261 ******

TASK [download : debug] ******************************************************** ok: [node1] => { "msg": "Pull docker.io/library/nginx:1.23.0-alpine required is: True" } Tuesday 06 December 2022 22:38:26 +0000 (0:00:00.183) 0:12:40.444 ****** Tuesday 06 December 2022 22:38:26 +0000 (0:00:00.091) 0:12:40.536 ****** Tuesday 06 December 2022 22:38:26 +0000 (0:00:00.144) 0:12:40.680 ****** Tuesday 06 December 2022 22:38:26 +0000 (0:00:00.169) 0:12:40.849 ****** FAILED - RETRYING: download_container | Download image if required (4 retries left). FAILED - RETRYING: download_container | Download image if required (3 retries left). FAILED - RETRYING: download_container | Download image if required (2 retries left). FAILED - RETRYING: download_container | Download image if required (1 retries left).

TASK [download : download_container | Download image if required] ************** fatal: [node1 -> node1]: FAILED! => {"attempts": 4, "changed": true, "cmd": ["/usr/local/bin/nerdctl", "-n", "k8s.io", "pull", "--quiet", "docker.io/library/nginx:1.23.0-alpine"], "delta": "0:00:00.990523", "end": "2022-12-06 22:38:47.889778", "msg": "non-zero return code", "rc": 1, "start": "2022-12-06 22:38:46.899255", "stderr": "time="2022-12-06T22:38:47Z" level=fatal msg="failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/nginx/manifests/sha256:4a846cc240449c53c8ae24269ba6bcaee5167d8ad75cd2a8d8ba422b7c726979: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"", "stderr_lines": ["time="2022-12-06T22:38:47Z" level=fatal msg="failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/nginx/manifests/sha256:4a846cc240449c53c8ae24269ba6bcaee5167d8ad75cd2a8d8ba422b7c726979: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit""], "stdout": "", "stdout_lines": []}

[turbodeploy]

In the past, I was able to use docker login. But it seems that is not an option using crictl. I am not able to find documentation/help on how to get around this. Any help would be greatly appreciated.

[jwitko]

This isn't really a bug for kubespray is it?

Also, this should help you: https://github.com/kubernetes-sigs/cri-tools/issues/482

[turbodeploy]

How to explain this. Ok, on the first run for kubespray, crictl is not yet available, so that link you provided, does not help me, as I have tried that. If I run the playbook the 1st time and it errors out where I am seeing it, then I can use crictl with the creds to pull in that image from the command line, run kubespray again, and it works as expected, since the dockerhub image is already available locally.

What I don't see, and I can be missing it, is I was hoping there was a way to set those creds as part of the playbook run, or, for example in the inventory/x/group_vars/all/containerd.yml

# containerd_registries:
#   "docker.io": "https://registry-1.docker.io"

# containerd_registry_auth:
#   - registry: 10.0.0.2:5000
#     username: user
#     password: pass

Where I can feed in the auth for dockerhub, or through the cli when running the playbook.

[jwitko]

Is containerd your chosen runtime?
You'll have to forgive me but I'm having a hard time following. Are you using the above variable configuration to set the docker.io registry and provide auth to it , and it's simply not working?

[turbodeploy]

containerd is the chosen runtime.

Initially, without the auth being set in the inventory, I got the docker pull error. When I try to set it (in the example above, but uncommented), it seems to be ignoring the creds.

I am just not sure what I am missing.

Here is my current containerd.yml

 containerd_registry_auth:
   - registry: "docker.io"
     username: username
     password: passwd

[caruccio]

I'm hitting this same issue. Also tried with config below:

 containerd_registry_auth:
   - registry: "registry-1.docker.io"
     username: username
     password: passwd
   - registry: "docker.io"
     username: username
     password: passwd

[chanyshev]

I get the same error, If you do docker pull docker.io/library/nginx:1.23.2-alpine The image is being downloaded Why does he need nginx ?

[chanyshev]

I think this is a problem kubespray. I have a pro subscription on docker.io, so I have 5000 connections. config file: inventory/local/group_vars/all/containerd.yml

containerd_registry_auth:
 - registry: docker.io
    username: ****
    password: ****

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[sebastienbonami]

I had the same issue on my side (with Kubespray v2.21.0) and I fixed it with the following:

containerd_registry_auth:
  - registry: registry-1.docker.io
    username: user
    password: pass

So it seems that by default Kubespray sets endpoint https://registry-1.docker.io for docker.io: https://github.com/kubernetes-sigs/kubespray/blob/v2.21.0/roles/container-engine/containerd/defaults/main.yml#L50-L51

And the registry value for the auth configuration needs to be the endpoint minus the protocol (https://): registry-1.docker.io.

Somebody can confirm?

/remove-lifecycle stale

[behzadev]

Same issue here

TASK [download : debug] *****************************************************************************************************************************************************************
ok: [node1] => {
    "msg": "Pull docker.io/library/nginx:1.23.2-alpine required is: True"
}
Tuesday 04 July 2023  20:17:35 +0330 (0:00:00.031)       0:05:35.258 **********
Tuesday 04 July 2023  20:17:35 +0330 (0:00:00.019)       0:05:35.277 **********
Tuesday 04 July 2023  20:17:35 +0330 (0:00:00.017)       0:05:35.295 **********
Tuesday 04 July 2023  20:17:35 +0330 (0:00:00.023)       0:05:35.318 **********
FAILED - RETRYING: [node1]: download_container | Download image if required (4 retries left).
FAILED - RETRYING: [node1]: download_container | Download image if required (3 retries left).
FAILED - RETRYING: [node1]: download_container | Download image if required (2 retries left).
FAILED - RETRYING: [node1]: download_container | Download image if required (1 retries left).

TASK [download : download_container | Download image if required] ***********************************************************************************************************************
fatal: [node1]: FAILED! => {"attempts": 4, "changed": true, "cmd": ["/usr/local/bin/nerdctl", "-n", "k8s.io", "pull", "--quiet", "docker.io/library/nginx:1.23.2-alpine"], "delta": "0:00:00.448304", "end": "2023-07-04 16:48:05.573012", "msg": "non-zero return code", "rc": 1, "start": "2023-07-04 16:48:05.124708", "stderr": "time=\"2023-07-04T16:48:05Z\" level=fatal msg=\"failed to resolve reference \\\"docker.io/library/nginx:1.23.2-alpine\\\": unexpected status from HEAD request to https://registry-1.docker.io/v2/library/nginx/manifests/1.23.2-alpine: 403 Forbidden\"", "stderr_lines": ["time=\"2023-07-04T16:48:05Z\" level=fatal msg=\"failed to resolve reference \\\"docker.io/library/nginx:1.23.2-alpine\\\": unexpected status from HEAD request to https://registry-1.docker.io/v2/library/nginx/manifests/1.23.2-alpine: 403 Forbidden\""], "stdout": "", "stdout_lines": []}

[hellt]

I am hitting this issue as well and specifying containerd registry auth like this:

containerd_registry_auth:
  - registry: registry-1.docker.io
    username: <name>
    password: <pat>

doesn't seem to help.

Who knows how can I redefined the variable to point to a ghcr mirror where nginx is located to alleviate pull rate limits?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten