Pull images from others than k8s.gcr.io

[lostwater]

Hi, I am in China, because of GFW, you can not pull images from k8s.gcr.io (google is banned).

Is there any solution that allows me to pull the images from the mirror GCR repo on docker.io ? for example hub.docker.com/u/googlecontainer

I found some solutions that you can use 'docker pull' to pull the images, then 'docker tag' to re-tag image to k8s.gcr.io namespace, but the problem is I have to pull and tag every time image updates, and the most important is microk8s.ctr image does not have 'tag' command.

[ktsakalozos]

I hear that some are using proxies. See the "Deploy behind a proxy" section in https://microk8s.io/docs/ on how to set MicroK8s with a proxy.

[balchua]

Hi @lostwater, maybe this can help. To get your cluster working, you need to have your pause container using the registry that you can access from china. You can edit the /var/snap/microk8s/current/args/containerd-template.toml. You should be able to find the pause image there. After that you need to restart containerd snap.microk8s.daemon-containerd.

But for addons, unfortunately the yamls are mostly located in $SNAP, which is a readonly fs.

if i remember right, china have a registry where they get the images mirrored from gcr. And they use that to pull images. But that was sometime ago.

[lostwater]

Hi @lostwater, maybe this can help. To get your cluster working, you need to have your pause container using the registry that you can access from china. You can edit the /var/snap/microk8s/current/args/containerd-template.toml. You should be able to find the pause image there. After that you need to restart containerd snap.microk8s.daemon-containerd.

But for addons, unfortunately the yamls are mostly located in $SNAP, which is a readonly fs.

if i remember right, china have a registry where they get the images mirrored from gcr. And they use that to pull images. But that was sometime ago.

thanks, I have found the 'pause' yaml, of course, I can modify that file. but for addons, as you said, there are mirrors in China, but the namespace is different, how can i let microk8s use that images, since i can not re-tag images using micork8s.ctr

[ktsakalozos]

Would this https://github.com/ubuntu/microk8s/issues/396 work for you?

[lostwater]

Would this #396 work for you?

I am not able to set a proxy server

[ktsakalozos]

@lostwater as already mentioned the yaml manifests for addons are on a read only file system. You could microk8s.enable "whatever_addons_you_need" and as soon as you see them failing to pull the images edit the respective deployments so they point to the right repository. Do a microk8s.kubectl edit "failing_deployment"` and you should see the reference to the images under the containers section.

Of course this requires you edit all yamls by hand. Unfortunately we do not have a single place to change all image references.

[lostwater]

@ktsakalozos thanks very much, but I ended up with setting up a proxy server. I do not want to edit each yaml every time I deploy microk8s. please consider supporting custom image namespace, for example, I have a mirror docker.io/mirrorgooglecontainers/ = k8s.gcr/

[ktsakalozos]

Thank you for the suggestion @lostwater. I am going t leave this issue open and mark it as a feature request.

[IsvaraWoo]

@ktsakalozos I running a http-proxy listen 8118 locally, and set HTTPS_PROXY=http://127.0.0.1:8118 in /var/snap/microk8s/current/args/containerd-env and restart snap.microk8s.daemon-containerd, but no effect on pull any images. I tried again and again, but couldn't execute microk8s.enable dns correctly. The status of coredns-f7867545d-qxxdq always is pending. So I doubt this proxy setting.

[dxas90]

@lostwater hey man I'm a cuban and I am dealing with the same problem even docker.io it is blocked to us so I bring a tor proxy with me just apt install tor and I changed the the file /var/snap/microk8s/current/args/containerd-env to use the tor proxy

happy coding

[lizebang]

@ktsakalozos Thank you for your work. Maybe we can add a flag just like kubeadm did.

kubeadm init --image-repository=gcr.azk8s.cn/google_containers

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[gaochundong]

+1 Same issue.

[ZheGuangZeng]

+1 Same issue.

[HarborZeng]

+1 feature request

[huang12zheng]

+1 feature request

I set it for mk

sed -i "s#k8s.gcr.io#registry.cn-hangzhou.aliyuncs.com/google_containers#g" /etc/containerd/config.toml

- name: proxy https
      lineinfile:
        path: /var/snap/microk8s/current/args/containerd-env
        regexp: 'HTTPS_PROXY='
        line: 'HTTPS_PROXY=http://127.0.0.1:7890'

but it no work for something like

➜  ~ mk get pod -A
NAMESPACE            NAME                                         READY   STATUS              RESTARTS   AGE
container-registry   registry-9b57d9df8-txm68                     1/1     Running             10         16h
kube-system          hostpath-provisioner-5c65fbdb4f-jmf49        1/1     Running             10         16h
kube-system          calico-node-v7v2m                            1/1     Running             0          6m46s
kube-system          coredns-86f78bb79c-5sw4r                     1/1     Running             0          2m54s
kube-system          calico-kube-controllers-847c8c99d-x5r2m      1/1     Running             0          6m46s
kube-system          calico-node-44njh                            1/1     Running             0          6m46s
kube-system          metrics-server-8bbfb4bdb-x28nl               0/1     ContainerCreating   0          17s
kube-system          calico-node-ng4wf                            1/1     Running             0          6m46s
kube-system          kubernetes-dashboard-7ffd448895-b5sqd        0/1     CrashLoopBackOff    13         13h
default              aptsync-6cdb6846d5-kgbwz                     0/1     CrashLoopBackOff    159        13h
kube-system          dashboard-metrics-scraper-6c4568dc68-jz9cc   1/1     Running             1          13h

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  33s   default-scheduler  Successfully assigned kube-system/metrics-server-8bbfb4bdb-x28nl to prisma
  Normal   Pulling    32s   kubelet            Pulling image "k8s.gcr.io/metrics-server-amd64:v0.3.6"
  Warning  Failed     2s    kubelet            Failed to pull image "k8s.gcr.io/metrics-server-amd64:v0.3.6": rpc error: code = Unknown desc = failed to pull and unpack image "k8s.gcr.io/metrics-server-amd64:v0.3.6": failed to resolve reference "k8s.gcr.io/metrics-server-amd64:v0.3.6": failed to do request: Head "https://k8s.gcr.io/v2/metrics-server-amd64/manifests/v0.3.6": dial tcp 64.233.189.82:443: i/o timeout
  Warning  Failed     2s    kubelet            Error: ErrImagePull
  Normal   BackOff    1s    kubelet            Back-off pulling image "k8s.gcr.io/metrics-server-amd64:v0.3.6"
  Warning  Failed     1s    kubelet            Error: ImagePullBackOff

and I snap remove microk8s(2264), and reinstall it((2262). it is work

NAMESPACE            NAME                                         READY   STATUS              RESTARTS   AGE
kube-system          calico-kube-controllers-f7868dd95-hkc72      1/1     Running             3          19m
kube-system          calico-node-65pfz                            1/1     Running             3          19m
kube-system          coredns-7f9c69c78c-bs9hx                     1/1     Running             0          12m
kube-system          dashboard-metrics-scraper-78d7698477-rtzgj   1/1     Running             0          11m
kube-system          metrics-server-8bbfb4bdb-9qr2b               1/1     Running             0          11m
kube-system          kubernetes-dashboard-85fd7f45cb-ljrpr        1/1     Running             1          11m
kube-system          hostpath-provisioner-5c65fbdb4f-8dn7x        1/1     Running             1          14m
container-registry   registry-9b57d9df8-r28c8                     0/1     ContainerCreating   0          14m

But I still want to use a custom Registry

[HarborZeng]

@huang12zheng

sed -i "s#k8s.gcr.io#registry.cn-hangzhou.aliyuncs.com/google_containers#g" /etc/containerd/config.toml

sed replacement won't work, because there are temporary images whose tag is hash string, and they can't be save and imported, so you may refer to a specialized install manifest like https://github.com/shikanon/kubeflow-manifests

[huang12zheng]

@HarborZeng ~~Thank you. I would find out~~

gcr.io/ml-pipeline/cache-deployer
registry.cn-shenzhen.aliyuncs.com/tensorbytes/ml-pipeline-cache-deployer@sha256:a13d49a4bee754f221697957d8491469bf2f958bbaac3d09707f053c8a4adf83
registry.cn-shenzhen.aliyuncs.com/tensorbytes/ml-pipeline-cache-deployer:1.5.0-rc.2-deb1e


k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea
registry.cn-hangzhou.aliyuncs.com/google_containers/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea

From the download link,My case is simple to you.

`registry.config_path` specifies a directory to look for registry hosts
configuration. When resolving an image name during pull operations, the CRI
plugin will look in the `<registry.config_path>/<image hostname>/` directory
for host configuration. An optional `hosts.toml` file in that directory may be
used to configure which hosts will be used for the pull operation as well
host-specific configurations. Updates under that directory do not require
restarting the containerd daemon.

https://github.com/containerd/containerd/blob/main/releases/v1.5.0.toml#L100-L106 It may be relevant

[zhhuabj]

@huang12zheng

sed -i "s#k8s.gcr.io#registry.cn-hangzhou.aliyuncs.com/google_containers#g" /etc/containerd/config.toml

sed replacement won't work, because there are temporary images whose tag is hash string, and they can't be save and imported, so you may refer to a specialized install manifest like https://github.com/shikanon/kubeflow-manifests

how about trying sed -i "s#k8s.gcr.io#registry.cn-hangzhou.aliyuncs.com/google_containers#g" /var/snap/microk8s/current/args/containerd-template.toml sudo systemctl restart snap.microk8s.daemon-containerd.service

[neoaggelos]

Hi @zhhuabj, this does not sound like a solution that would work in all cases. Can you check if the documentation in https://microk8s.io/docs/registry-private#configure-registry-mirrors-7 helps you resolve the issue? Thanks!

[bioinformatist]

Hi @zhhuabj, this does not sound like a solution that would work in all cases. Can you check if the documentation in https://microk8s.io/docs/registry-private#configure-registry-mirrors-7 helps you resolve the issue? Thanks!

@neoaggelos Sir, this way cannot solve the issue. The inspected report file /var/snap/microk8s/current/inspection-report/snap.microk8s.daemon-containerd/systemctl.log records errors like:

Sep 24 17:29:25 an microk8s.daemon-containerd[1083659]: time="2022-09-24T17:29:25.773315903+08:00" level=info msg="trying next host" error="pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed" host=registry.aliyuncs.com

However, this way works well:

sed -i -e 's#k8s.gcr.io#registry.aliyuncs.com/google_containers#g' /var/snap/microk8s/current/args/containerd-template.toml
sudo snap restart microk8s

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.