loki
loki copied to clipboard
Failed to fetch images: prometheus-config-reloader and grafana-agent
Hello!
Problem fetching the grafana-agent and prometheus-config-reloader, when installing loki. I'm in isolated env. I have pointed other image's to our own registry and it works fine. But how can i point those images that won't work to our own registry? I can't find the endpoints in the chart's. The endpoints that loki-logs "pod" is trying to fetch is = "https://quay.io/v2/" + "https://registry-1.docker.io/v2/" but failing as you can see below in the logs.
The grafana-operator starts but when the loki-logs "Pod" is trying to start it cannot fetch. Below is a "Describe" of one of the loki-logs "pod" Error: ErrImagePull
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 102s default-scheduler Successfully assigned loki/loki-logs-948r4 to test
Warning Failed 87s kubelet Failed to pull image "quay.io/prometheus-operator/prometheus-config-reloader:v0.67.1": rpc error: code = Unknown desc = Error response from daemon: Get "https://quay.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning Failed 56s kubelet Failed to pull image "quay.io/prometheus-operator/prometheus-config-reloader:v0.67.1": rpc error: code = Unknown desc = Error response from daemon: Get "https://quay.io/v2/": dial tcp xx.xx.xx.xx:443: i/o timeout
Warning Failed 41s (x2 over 72s) kubelet Error: ErrImagePull
Warning Failed 41s kubelet Error: ImagePullBackOff
Warning Failed 41s (x2 over 72s) kubelet Failed to pull image "grafana/agent:v0.38.1": rpc error: code = Unknown desc = Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Normal BackOff 41s kubelet Back-off pulling image "grafana/agent:v0.38.1"
Warning Failed 41s kubelet Error: ImagePullBackOff
Normal BackOff 41s kubelet Back-off pulling image "quay.io/prometheus-operator/prometheus-config-reloader:v0.67.1"
Normal Pulling 28s (x3 over 102s) kubelet Pulling image "quay.io/prometheus-operator/prometheus-config-reloader:v0.67.1"
Normal Pulling 13s (x3 over 87s) kubelet Pulling image "grafana/agent:v0.38.1"
Warning Failed 13s (x3 over 87s) kubelet Error: ErrImagePull
Warning Failed 13s kubelet Failed to pull image "quay.io/prometheus-operator/prometheus-config-reloader:v0.67.1": rpc error: code = Unknown desc = Error response from daemon: Get "https://quay.io/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Here below you can see that the grafana-agent-operator is running fine, it can fetch the img from our own registry. But the loki-logs "pod" having problem pulling the IMG as error-log above.
loki-canary-wp2jw 1/1 Running 0 37s
loki-gateway-74d74f9984-l9m8j 1/1 Running 0 37s
loki-grafana-agent-operator-54bd4b9674-59lns 1/1 Running 0 37s
loki-logs-2nsrb 0/2 ErrImagePull 0 34s
The following block below is working for the grafana-agent-operator. I want to do the same thing for the agent and the config-reloader IMG, if it is the correct way of doing it. It overwrites and points to our own registry.
grafana-agent-operator:
image:
# -- Image registry
registry: harbor.test.com
# -- Image repo
repository: docker_registry.hub.docker.com-r/grafana/agent-operator
# -- Image tag
tag: main
Chart=5.41.4
anyway I got the same issue with the loki-5.39.0 chart version
Questions have a better chance of being answered if you ask them on the community forums.
Hi, I opened a thread in the community forum.
any update?