projectcalico
Images in custom registry: CreateContainerError
Here we are using calico as network plugin for k8s based on KB: https://docs.tigera.io/calico/latest/operations/image-options/alternate-registry by using local registry as calico dockers images. We adjust based on that KB in tigera-operator.yaml and custom-resources.yaml. But calico related pods can't be in normal status after apply custom-resources.yaml. Below are the detail error info, any suggestion?
[root@k8sma manifests]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
calico-system calico-kube-controllers-576f6b4cc-kzbdp 0/1 Pending 0 11m
calico-system calico-node-vh5j4 0/1 Init:CreateContainerError 0 11m
calico-system calico-typha-c8775554c-lrd6c 0/1 CreateContainerError 0 11m
calico-system csi-node-driver-6fblr 0/2 ContainerCreating 0 11m
kube-system coredns-7b5944fdcf-rmgxx 0/1 Pending 0 11m
kube-system coredns-7b5944fdcf-spc2h 0/1 Pending 0 11m
kube-system etcd-k8sma 1/1 Running 38 12m
kube-system kube-apiserver-k8sma 1/1 Running 43 12m
kube-system kube-controller-manager-k8sma 1/1 Running 46 12m
kube-system kube-proxy-tbcxw 1/1 Running 0 11m
kube-system kube-scheduler-k8sma 1/1 Running 45 12m
tigera-operator tigera-operator-5847fb7754-f67jj 1/1 Running 0 11m
[root@k8sma manifests]# kubectl describe pod calico-typha-c8775554c-lrd6c -n calico-system
Name: calico-typha-c8775554c-lrd6c
Namespace: calico-system
Priority: 2000000000
Priority Class Name: system-cluster-critical
Service Account: calico-typha
Node: k8sma/192.168.31.111
Start Time: Wed, 17 Jul 2024 14:26:37 +0800
Labels: app.kubernetes.io/name=calico-typha
k8s-app=calico-typha
pod-template-hash=c8775554c
Annotations: hash.operator.tigera.io/system: fdde45054a8ae4f629960ce37570929502e59449
tigera-operator.hash.operator.tigera.io/tigera-ca-private: bc1830f9e6cb4590e3f0da8a12e44e08ecb12eaa
tigera-operator.hash.operator.tigera.io/typha-certs: 84e83e8f72477d38dcd30a197b85919b25f006ed
Status: Pending
IP: 192.168.31.111
IPs:
IP: 192.168.31.111
Controlled By: ReplicaSet/calico-typha-c8775554c
Containers:
calico-typha:
Container ID:
Image: localhost:5000/calico/typha:v3.28.0
Image ID:
Port: 5473/TCP
Host Port: 5473/TCP
SeccompProfile: RuntimeDefault
State: Waiting
Reason: CreateContainerError
Ready: False
Restart Count: 0
Liveness: http-get http://localhost:9098/liveness delay=0s timeout=10s period=60s #success=1 #failure=3
Readiness: http-get http://localhost:9098/readiness delay=0s timeout=10s period=30s #success=1 #failure=3
Environment:
TYPHA_LOGSEVERITYSCREEN: info
TYPHA_LOGFILEPATH: none
TYPHA_LOGSEVERITYSYS: none
TYPHA_CONNECTIONREBALANCINGMODE: kubernetes
TYPHA_DATASTORETYPE: kubernetes
TYPHA_HEALTHENABLED: true
TYPHA_HEALTHPORT: 9098
TYPHA_K8SNAMESPACE: calico-system
TYPHA_CAFILE: /etc/pki/tls/certs/tigera-ca-bundle.crt
TYPHA_SERVERCERTFILE: /typha-certs/tls.crt
TYPHA_SERVERKEYFILE: /typha-certs/tls.key
TYPHA_FIPSMODEENABLED: false
TYPHA_SHUTDOWNTIMEOUTSECS: 300
TYPHA_CLIENTCN: typha-client
KUBERNETES_SERVICE_HOST: 10.96.0.1
KUBERNETES_SERVICE_PORT: 443
Mounts:
/etc/pki/tls/cert.pem from tigera-ca-bundle (ro,path="ca-bundle.crt")
/etc/pki/tls/certs from tigera-ca-bundle (ro)
/typha-certs from typha-certs (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ptcm5 (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
tigera-ca-bundle:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: tigera-ca-bundle
Optional: false
typha-certs:
Type: Secret (a volume populated by a Secret)
SecretName: typha-certs
Optional: false
kube-api-access-ptcm5:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: kubernetes.io/os=linux
Tolerations: :NoSchedule op=Exists
:NoExecute op=Exists
CriticalAddonsOnly op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 12m default-scheduler Successfully assigned calico-system/calico-typha-c8775554c-lrd6c to k8sma
Warning FailedMount 12m kubelet MountVolume.SetUp failed for volume "tigera-ca-bundle" : failed to sync configmap cache: timed out waiting for the condition
Warning Failed 12m kubelet Error: failed to generate container "9a30eed862e93fc8248b64e0bd5d7de2fe090d24a618f810476f1e20508451e3" spec: failed to generate spec: no command specified
Warning Failed 12m kubelet Error: failed to generate container "43d0f89a95da40b10f82636bfc37e7da5767643495df669dd6bc8acafbca829c" spec: failed to generate spec: no command specified
Warning Failed 12m kubelet Error: failed to generate container "a67cbab6ceec0ed297105cdac05275306e06cad1b46480775c5c5cf82b88bcc8" spec: failed to generate spec: no command specified
Warning Failed 12m kubelet Error: failed to generate container "47551a347907b903f4d840a06f49366013147f985940c4dfa798d81b8c898f01" spec: failed to generate spec: no command specified
Warning Failed 11m kubelet Error: failed to generate container "59093a5b73bafab6fb02c4d1bbae407600ecb074bb68ecd7e9b19ead8605db69" spec: failed to generate spec: no command specified
Warning Failed 11m kubelet Error: failed to generate container "83ff673c79fb6437472347b8448e73bdff761f0a3a90ff112c1990c0a9a87197" spec: failed to generate spec: no command specified
Warning Failed 11m kubelet Error: failed to generate container "3071ec15367613b8e0ba5092edbb008124a2df6b5595cc217970b82f8e41cd6d" spec: failed to generate spec: no command specified
Warning Failed 11m kubelet Error: failed to generate container "77b8c8428e6a2b1499c5461cc57ae79dc1aae6cac16371c5125d056fce94892c" spec: failed to generate spec: no command specified
Warning Failed 11m kubelet Error: failed to generate container "ff4b01d90db08f46f2a0f24b1cd9e66f2bcad7091b1eee1ef6e38996b605c1c6" spec: failed to generate spec: no command specified
Warning Failed 10m (x3 over 10m) kubelet (combined from similar events): Error: failed to generate container "37eafe5c181ee2edb9cc0559622ed142c75064ba72ba37ffc395d57cf4c43a43" spec: failed to generate spec: no command specified
Normal Pulled 2m25s (x48 over 12m) kubelet Container image "localhost:5000/calico/typha:v3.28.0" already present on machine
[root@k8sma manifests]# kubectl describe pod csi-node-driver-6fblr -n calico-system
Name: csi-node-driver-6fblr
Namespace: calico-system
Priority: 2000001000
Priority Class Name: system-node-critical
Service Account: default
Node: k8sma/192.168.31.111
Start Time: Wed, 17 Jul 2024 14:26:39 +0800
Labels: app.kubernetes.io/name=csi-node-driver
controller-revision-hash=74fbb6df98
k8s-app=csi-node-driver
name=csi-node-driver
pod-template-generation=1
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: DaemonSet/csi-node-driver
Containers:
calico-csi:
Container ID:
Image: localhost:5000/calico/csi:v3.28.0
Image ID:
Port: <none>
Host Port: <none>
SeccompProfile: RuntimeDefault
Args:
--nodeid=$(KUBE_NODE_NAME)
--loglevel=$(LOG_LEVEL)
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment:
LOG_LEVEL: warn
KUBE_NODE_NAME: (v1:spec.nodeName)
Mounts:
/csi from socket-dir (rw)
/var/lib/kubelet from kubelet-dir (rw)
/var/run from varrun (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rjd5s (ro)
csi-node-driver-registrar:
Container ID:
Image: localhost:5000/calico/node-driver-registrar:v3.28.0
Image ID:
Port: <none>
Host Port: <none>
SeccompProfile: RuntimeDefault
Args:
--v=5
--csi-address=$(ADDRESS)
--kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment:
ADDRESS: /csi/csi.sock
DRIVER_REG_SOCK_PATH: /var/lib/kubelet/plugins/csi.tigera.io/csi.sock
KUBE_NODE_NAME: (v1:spec.nodeName)
Mounts:
/csi from socket-dir (rw)
/registration from registration-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rjd5s (ro)
Conditions:
Type Status
PodReadyToStartContainers False
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
varrun:
Type: HostPath (bare host directory volume)
Path: /var/run
HostPathType:
kubelet-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet
HostPathType: Directory
socket-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins/csi.tigera.io
HostPathType: DirectoryOrCreate
registration-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins_registry
HostPathType: Directory
kube-api-access-rjd5s:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: kubernetes.io/os=linux
Tolerations: :NoSchedule op=Exists
:NoExecute op=Exists
CriticalAddonsOnly op=Exists
node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 13m default-scheduler Successfully assigned calico-system/csi-node-driver-6fblr to k8sma
Warning NetworkNotReady 3m45s (x302 over 13m) kubelet network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
@huanghaiqing1 are you using custom built images? Or are these the official Calico images we host?
Warning Failed 10m (x3 over 10m) kubelet (combined from similar events): Error: failed to generate container "37eafe5c181ee2edb9cc0559622ed142c75064ba72ba37ffc395d57cf4c43a43" spec: failed to generate spec: no command specified
This suggests potentially a problem with the container, as the image we provide has a CMD directive to provide a default command.
On Jul 23, 2024, at 1:01 AM, Casey Davenport @.> wrote: Hello, here I'm using official Calico images. And based on official KB: https://docs.tigera.io/calico/latest/operations/image-options/alternate-registry , I need to modify yaml: tigera-operator.yaml and custom-resources.yaml. But the instructions is too short or maybe I didn't get its point. Here I modify according to the BK's suggestion, but related pods about calico part don't startup normally. @huanghaiqing1 are you using custom built images? Or are these the official Calico images we host? Warning Failed 10m (x3 over 10m) kubelet (combined from similar events): Error: failed to generate container "37eafe5c181ee2edb9cc0559622ed142c75064ba72ba37ffc395d57cf4c43a43" spec: failed to generate spec: no command specified This suggests potentially a problem with the container, as the image we provide has a CMD directive to provide a default command. — Reply to this email directly, view it on GitHub , or unsubscribe . You are receiving this because you were mentioned. Message ID: @.>
Could you run the following for me to see the output:
docker inspect localhost:5000/calico/typha:v3.28.0
Specifically looking to see what the "Cmd" section of the output shows.
Also, to be sure - have you specified the --pod-infra-container-image argument on the kubelet by chance?
Hello, below is the output. CMD session shows null. I'm not sure the download calico package for private registry is complete or not. Because based on official KB, it requires. But my unzip calico folder about images only gives 7 tar ball.
docker tag quay.io/tigera/operator:v1.34.0 $REGISTRY/tigera/operator:v1.34.0 docker tag calico/typha:v3.28.0 $REGISTRY/calico/typha:v3.28.0 docker tag calico/ctl:v3.28.0 $REGISTRY/calico/ctl:v3.28.0 docker tag calico/node:v3.28.0 $REGISTRY/calico/node:v3.28.0 docker tag calico/cni:v3.28.0 $REGISTRY/calico/cni:v3.28.0 docker tag calico/apiserver:v3.28.0 $REGISTRY/calico/apiserver:v3.28.0 docker tag calico/kube-controllers:v3.28.0 $REGISTRY/calico/kube-controllers:v3.28.0 docker tag calico/dikastes:v3.28.0 $REGISTRY/calico/dikastes:v3.28.0 docker tag calico/pod2daemon-flexvol:v3.28.0 $REGISTRY/calico/pod2daemon-flexvol:v3.28.0 docker tag calico/csi:v3.28.0 $REGISTRY/calico/csi:v3.28.0 docker tag calico/node-driver-registrar:v3.28.0 $REGISTRY/calico/node-driver-registrar:v3.28.0 ***@***.*** images]# docker images | grep -i localhost localhost:5000/calico/pod2daemon v3.28.0 651b8c0ee75e 14 minutes ago 13.4MB localhost:5000/calico/flannel-migration-controller v3.28.0 d0c308187ddb 15 minutes ago 128MB localhost:5000/calico/dikastes v3.28.0 3f03f0b0cf90 19 minutes ago 41.9MB localhost:5000/calico/kube-controllers v3.28.0 2bd71868d777 22 minutes ago 79.2MB localhost:5000/calico/cni v3.28.0 b144a54fe61f 23 minutes ago 209MB localhost:5000/calico/node v3.28.0 5bc4d7581211 26 minutes ago 355MB localhost:5000/calico/typha v3.28.0 337aa4a7808a 29 minutes ago 71.2MB ***@***.*** images]# docker images | grep -i "^calico" calico/pod2daemon v3.28.0 651b8c0ee75e 14 minutes ago 13.4MB calico/flannel-migration-controller v3.28.0 d0c308187ddb 16 minutes ago 128MB calico/dikastes v3.28.0 3f03f0b0cf90 20 minutes ago 41.9MB calico/kube-controllers v3.28.0 2bd71868d777 22 minutes ago 79.2MB calico/cni v3.28.0 b144a54fe61f 23 minutes ago 209MB calico/node v3.28.0 5bc4d7581211 26 minutes ago 355MB calico/typha v3.28.0 337aa4a7808a 29 minutes ago 71.2MB ***@***.*** images]
# docker inspect localhost:5000/calico/typha:v3.28.0 [ { "Id": "sha256:337aa4a7808a28ab0c30f9348c9157600e1ed0882b2e780fcbd7f1b4e63626b5", "RepoTags": [ "calico/typha:v3.28.0", "localhost:5000/calico/typha:v3.28.0" ], "RepoDigests": [], "Parent": "", "Comment": "Imported from -", "Created": "2024-07-24T00:55:07.84429967Z", "DockerVersion": "26.1.1", "Author": "", "Config": { "Hostname": "", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": null, "Cmd": null, "Image": "", "Volumes": null, "WorkingDir": "", "Entrypoint": null, "OnBuild": null, "Labels": null }, "Architecture": "amd64", "Os": "linux", "Size": 71173819, "GraphDriver": { "Data": { "MergedDir": "/var/lib/docker/overlay2/9b965a65bcffe20919974634876e91d53ed0f939cf5bf4f3933c9674dbbe6106/merged", "UpperDir": "/var/lib/docker/overlay2/9b965a65bcffe20919974634876e91d53ed0f939cf5bf4f3933c9674dbbe6106/diff", "WorkDir": "/var/lib/docker/overlay2/9b965a65bcffe20919974634876e91d53ed0f939cf5bf4f3933c9674dbbe6106/work" }, "Name": "overlay2" }, "RootFS": { "Type": "layers", "Layers": [ "sha256:5f6c59d25589c88a42e4bafba0dd87729c844bf77be55c5e22847dcd0e65e240" ] }, "Metadata": { "LastTagTime": "2024-07-24T09:24:41.47737321+08:00" } } ]
Jul 24, 202412:59 AM，Casey Davenport @.> 写道： Also, to be sure - have you specified the --pod-infra-container-image argument on the kubelet by chance? — Reply to this email directly, view it on GitHub , or unsubscribe . You are receiving this because you were mentioned. Message ID: @.>
I didn't include " --pod-infra-container-image" in my kubectl realetd service and I also didn't include it manually. Is there a way to provide an complete calico download tar-ball, which can be used for setup private registray and deploy from local. I'm in China and technically I can't get these images from calico directly, if not use VPN. Jul 24, 202412:59 AM，Casey Davenport @.> 写道： Also, to be sure - have you specified the --pod-infra-container-image argument on the kubelet by chance? — Reply to this email directly, view it on GitHub , or unsubscribe . You are receiving this because you were mentioned. Message ID: @.>
@huanghaiqing1 I downloaded and loaded the images from release-v3.28.0.tar from here: https://github.com/projectcalico/calico/releases/tag/v3.28.0
This is the output I get for the calico/typha image:
[
{
"Id": "sha256:a9372c0f51b54c589e5a16013ed3049b2a052dd6903d72603849fab2c4216fbc",
"RepoTags": [
"calico/typha:v3.28.0"
],
"RepoDigests": [],
"Parent": "",
"Comment": "buildkit.dockerfile.v0",
"Created": "2024-05-11T00:15:52.055728329Z",
"DockerVersion": "",
"Author": "",
"Config": {
"Hostname": "",
"Domainname": "",
"User": "999",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/usr/bin/calico-typha"
],
"ArgsEscaped": true,
"Image": "",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": null,
"OnBuild": null,
"Labels": {
"description": "Calico Typha is a fan-out datastore proxy",
"maintainer": "[email protected]",
"name": "Calico Typha",
"release": "1",
"summary": "Calico Typha is a fan-out datastore proxy",
"vendor": "Project Calico",
"version": "v3.28.0"
}
},
"Architecture": "amd64",
"Os": "linux",
"Size": 71143933,
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/e4a8269bcc993b26c93df9883fa4f2386e26673d1cf6b73c5dc3d69a6f2c286a/diff",
"MergedDir": "/var/lib/docker/overlay2/7c4bf0a7dee90c680d744ff2c0ec6476f661281ae6a28e5b1a9bd9ceac7ddd55/merged",
"UpperDir": "/var/lib/docker/overlay2/7c4bf0a7dee90c680d744ff2c0ec6476f661281ae6a28e5b1a9bd9ceac7ddd55/diff",
"WorkDir": "/var/lib/docker/overlay2/7c4bf0a7dee90c680d744ff2c0ec6476f661281ae6a28e5b1a9bd9ceac7ddd55/work"
},
"Name": "overlay2"
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:29ebc113185d6c09c2f84abf7d4fbbb49e2d6e4a169c0a5f9d14d1358d70827e",
"sha256:31719a1450dd2c929a14d1bd057b3b663cb7b99286198effa4e9e10b62007641"
]
},
"Metadata": {
"LastTagTime": "0001-01-01T00:00:00Z"
}
}
]
Note that the "Cmd" section is not null. I think the images you have loaded into your registry have been modified in some way to remove the embedded command.
Hello, I compared the release-v3.28.0.tar with my current downloaded one. It’s same size. And I use "docker import calico-typha.tar calico/typha:v3.28.0” to import it and other tar.ball as docker images and then push them to my private registry. But inspect calico/typha, the cmd becomes “null”. What’s the problem? If you think this is the cause for invalid applying tigera-operator.yaml and custom-resources.yaml based on KB: https://docs.tigera.io/calico/latest/operations/image-options/alternate-registry. How should I bypass?


2024年7月25日 01:13，Casey Davenport @.***> 写道：
@huanghaiqing1 https://github.com/huanghaiqing1 I downloaded and loaded the images from release-v3.28.0.tar from here: https://github.com/projectcalico/calico/releases/tag/v3.28.0
This is the output I get for the calico/typha image:
[ { "Id": "sha256:a9372c0f51b54c589e5a16013ed3049b2a052dd6903d72603849fab2c4216fbc", "RepoTags": [ "calico/typha:v3.28.0" ], "RepoDigests": [], "Parent": "", "Comment": "buildkit.dockerfile.v0", "Created": "2024-05-11T00:15:52.055728329Z", "DockerVersion": "", "Author": "", "Config": { "Hostname": "", "Domainname": "", "User": "999", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ], "Cmd": [ "/usr/bin/calico-typha" ], "ArgsEscaped": true, "Image": "", "Volumes": null, "WorkingDir": "/", "Entrypoint": null, "OnBuild": null, "Labels": { "description": "Calico Typha is a fan-out datastore proxy", "maintainer": @.***", "name": "Calico Typha", "release": "1", "summary": "Calico Typha is a fan-out datastore proxy", "vendor": "Project Calico", "version": "v3.28.0" } }, "Architecture": "amd64", "Os": "linux", "Size": 71143933, "GraphDriver": { "Data": { "LowerDir": "/var/lib/docker/overlay2/e4a8269bcc993b26c93df9883fa4f2386e26673d1cf6b73c5dc3d69a6f2c286a/diff", "MergedDir": "/var/lib/docker/overlay2/7c4bf0a7dee90c680d744ff2c0ec6476f661281ae6a28e5b1a9bd9ceac7ddd55/merged", "UpperDir": "/var/lib/docker/overlay2/7c4bf0a7dee90c680d744ff2c0ec6476f661281ae6a28e5b1a9bd9ceac7ddd55/diff", "WorkDir": "/var/lib/docker/overlay2/7c4bf0a7dee90c680d744ff2c0ec6476f661281ae6a28e5b1a9bd9ceac7ddd55/work" }, "Name": "overlay2" }, "RootFS": { "Type": "layers", "Layers": [ "sha256:29ebc113185d6c09c2f84abf7d4fbbb49e2d6e4a169c0a5f9d14d1358d70827e", "sha256:31719a1450dd2c929a14d1bd057b3b663cb7b99286198effa4e9e10b62007641" ] }, "Metadata": { "LastTagTime": "0001-01-01T00:00:00Z" } } ] Note that the "Cmd" section is not null. I think the images you have loaded into your registry have been modified in some way to remove the embedded command.
— Reply to this email directly, view it on GitHub https://github.com/projectcalico/calico/issues/9017#issuecomment-2248521013, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHFAPJE222UBI4N4OZFSBOLZN7ODLAVCNFSM6AAAAABK76WBSWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBYGUZDCMBRGM. You are receiving this because you were mentioned.
@huanghaiqing1 I think the problem is the docker import command.
docker import simply loads the filesystem contained within the tar file, but does not retain any metadata from the original image.
I would recommend using docker load instead, which will load the image as well as the metadata (including the Cmd).
For example:
docker load < calico-typha.tar
Very thanks for your remind. Here I use docker load to push imanges to local from downlaod folder release-v3.28.0/images. And I also create tag and push to my local private registry. And I can use "docker inspect" to see related local dockers have conten in "cmd". But when I follow KB: https://docs.tigera.io/calico/latest/operations/image-options/alternate-registry and use customized yamls: tigera-operator.yaml and custom-resources.yaml. But there are still different warnings during calico pods setup in k8s.
my steps:
kubeadm init --pod-network-cidr=192.1.0.0/16 --image-repository registry.aliyuncs.com/google_containers 2. adjust tigera-operator.yaml and custom-resources.yaml then create calico pods from local registry. You can also refer to attached modifed yaml files.
@.*** manifests]# diff tigera-operator.yaml tigera-operator.yaml.orig 25450c25450 < image: localhost:5000/tigera/operator:v1.34.0
image: quay.io/tigera/operator:v1.34.0
@.*** manifests]# diff custom-resources.yaml custom-resources.yaml.orig 13c13 < cidr: 192.1.0.0/16
cidr: 192.168.0.0/16
17,19d16 < variant: Calico < imagePath: calico < registry: localhost:5000 29a27
kubectl create -f ./tigera-operator.yaml kubectl create -f ./custom-resources.yaml
kubectl get pod -n calico-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-b7fd956f5-xpqgg 0/1 Pending 0 15s calico-node-c975x 0/1 Init:CreateContainerError 0 16s calico-typha-55458bc957-crc4k 0/1 CreateContainerError 0 18s csi-node-driver-2gbzt 0/2 ContainerCreating 0 16s
4. I output pod descrption and local docker inspect info to output.log for your reference. Maybe you can give some additional advice. Thanks.
change in tiger-operator.yaml:
Configure use of your image registry | Calico Documentationhttps://docs.tigera.io/calico/latest/operations/image-options/alternate-registry Configure Calico to pull images from a public or private registry. docs.tigera.io
From: Casey Davenport @.> Sent: Friday, July 26, 2024 11:06 PM To: projectcalico/calico @.> Cc: Huang Haiqing @.>; Mention @.> Subject: Re: [projectcalico/calico] can't setup calico related pods in k8s (Issue #9017)
@huanghaiqing1https://github.com/huanghaiqing1 I think the problem is the docker import command.
docker import simply loads the filesystem contained within the tar file, but does not retain any metadata from the original image.
I would recommend using docker load instead, which will load the image as well as the metadata (including the Cmd).
For example:
docker load < calico-typha.tar
— Reply to this email directly, view it on GitHubhttps://github.com/projectcalico/calico/issues/9017#issuecomment-2252961278, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHFAPJH33OJRH5NQ4PKJ2MLZOJQX5AVCNFSM6AAAAABK76WBSWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJSHE3DCMRXHA. You are receiving this because you were mentioned.
@huanghaiqing1 I don't see any diagnostics on your last comment - you likely need to embed them as links into the GitHub post rather than via email.
Here I submit related attachments here for your reference.
I use docker load to push imanges to local from downlaod folder release-v3.28.0/images. And I also create tag and push to my local private registry. And I can use "docker inspect" to see related local dockers have conten in "cmd". But when I follow KB: https://docs.tigera.io/calico/latest/operations/image-options/alternate-registry and use customized yamls: tigera-operator.yaml and custom-resources.yaml. But there are still different warnings during calico pods setup in k8s. What I'm curious is even I switch to docker load mode, I still see "spec: failed to generate spec: no command specified" when use "kubectl describe pod calico-typha-67f78bf575-2d6jz -n calico-system". But when I use "docker inspect calico/typha:v3.28.0", I can see "cmd="/usr/bin/calico-typha" in image defination.
my steps:
1.kubeadm init --pod-network-cidr=192.1.0.0/16 --image-repository registry.aliyuncs.com/google_containers 2.adjust tigera-operator.yaml and custom-resources.yaml, then create calico pods from local registry. You can also refer to attached modifed yaml files from here.
diff tigera-operator.yaml tigera-operator.yaml.orig 25450c25450 < image: localhost:5000/tigera/operator:v1.34.0
image: quay.io/tigera/operator:v1.34.0
diff custom-resources.yaml custom-resources.yaml.orig 13c13 < cidr: 192.1.0.0/16
cidr: 192.168.0.0/16
17,19d16 < variant: Calico < imagePath: calico < registry: localhost:5000 29a27
3.kubectl create -f ./tigera-operator.yaml kubectl create -f ./custom-resources.yaml
kubectl get pod -n calico-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-b7fd956f5-xpqgg 0/1 Pending 0 15s calico-node-c975x 0/1 Init:CreateContainerError 0 16s calico-typha-55458bc957-crc4k 0/1 CreateContainerError 0 18s csi-node-driver-2gbzt 0/2 ContainerCreating 0 16s
- I output pod descrption and local docker inspect info to output.log for your reference. Maybe you can give some additional advice. Thanks.
Hello, I have updated in github's comment. You can check there.
From: Casey Davenport @.> Sent: Wednesday, July 31, 2024 5:01 AM To: projectcalico/calico @.> Cc: Huang Haiqing @.>; Mention @.> Subject: Re: [projectcalico/calico] can't setup calico related pods in k8s (Issue #9017)
@huanghaiqing1https://github.com/huanghaiqing1 I don't see any diagnostics on your last comment - you likely need to embed them as links into the GitHub post rather than via email.
— Reply to this email directly, view it on GitHubhttps://github.com/projectcalico/calico/issues/9017#issuecomment-2259202265, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHFAPJFJIW4XWRIDV3T4HNLZO75LDAVCNFSM6AAAAABK76WBSWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJZGIYDEMRWGU. You are receiving this because you were mentioned.Message ID: @.***>
when use "kubectl describe pod calico-typha-67f78bf575-2d6jz -n calico-system". But when I use "docker inspect calico/typha:v3.28.0", I can see "cmd="/usr/bin/calico-typha" in image defination.
What about docker inspect localhost:5000/calico/typha:v3.28.0?
I think your custom-resources.yaml and tigera-operator.yaml both look correct to me. I suspect that this is a problem with the way the images are loaded into the registry, and it not pushing the manifest contents correct.
[root@k8sma ~]# docker inspect localhost:5000/calico/typha:v3.28.0
[
{
"Id": "sha256:a9372c0f51b54c589e5a16013ed3049b2a052dd6903d72603849fab2c4216fbc",
"RepoTags": [
"calico/typha:v3.28.0",
"localhost:5000/calico/typha:v3.28.0"
],
"RepoDigests": [
"localhost:5000/calico/typha@sha256:dc37e0ef67d141bea4bccee6f6488007b5ce7f560768a654246fa65ccf157b63"
],
"Parent": "",
"Comment": "buildkit.dockerfile.v0",
"Created": "2024-05-11T00:15:52.055728329Z",
"DockerVersion": "",
"Author": "",
"Config": {
"Hostname": "",
"Domainname": "",
"User": "999",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/usr/bin/calico-typha"
],
"ArgsEscaped": true,
"Image": "",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": null,
"OnBuild": null,
"Labels": {
"description": "Calico Typha is a fan-out datastore proxy",
"maintainer": "[email protected]",
"name": "Calico Typha",
"release": "1",
"summary": "Calico Typha is a fan-out datastore proxy",
"vendor": "Project Calico",
"version": "v3.28.0"
}
},
"Architecture": "amd64",
"Os": "linux",
"Size": 71143933,
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/cc4b4480639c02341b04e33d0e4b840838fcf66973876738235bfd75e036d628/diff",
"MergedDir": "/var/lib/docker/overlay2/b005e6ed593572d788e0ae6b18d2c7d6b6ea7a5c24986e5dd96eac2063d37cfa/merged",
"UpperDir": "/var/lib/docker/overlay2/b005e6ed593572d788e0ae6b18d2c7d6b6ea7a5c24986e5dd96eac2063d37cfa/diff",
"WorkDir": "/var/lib/docker/overlay2/b005e6ed593572d788e0ae6b18d2c7d6b6ea7a5c24986e5dd96eac2063d37cfa/work"
},
"Name": "overlay2"
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:29ebc113185d6c09c2f84abf7d4fbbb49e2d6e4a169c0a5f9d14d1358d70827e",
"sha256:31719a1450dd2c929a14d1bd057b3b663cb7b99286198effa4e9e10b62007641"
]
},
"Metadata": {
"LastTagTime": "2024-07-27T07:50:01.289194125+08:00"
}
}
]
[root@k8sma ~]#
@huanghaiqing1 do you change the image contents/Dockerfile in any way before pushing to your custom registry?
Hello, not clear about your detail intention. I modifed tigera-operator.yaml and custom-resources.yaml. based on KB: https://docs.tigera.io/calico/latest/operations/image-options/alternate-registry. Then try create and failed to setup calico pods inside k8s cluster. Do you have a example tigera-operator.yaml and custom-resources.yaml for my reference?
The only example I have is the one from the documentation, which I believe to be correct and I know has worked successfully for other users.
I'm still pretty sure this has something to do with how you are loading the images into your private registry somehow removing the Cmd section of the manifest.