loki
loki copied to clipboard
grafana
failed to flush user; store put chunk: NoCredentialProviders
Describe the bug I am running Loki/Promtail via Helm Charts von GCP/GKE:
- Loki 2.6.1 Helm 1.8.11: https://bityl.co/Echb
- Promtail 2.6.1 Helm 6.4.0: https://bityl.co/EaaJ
I noticed via Google Notification that my current billing is reaching its defined maximum and I was shocked why that is. In the billing console I was able to track down some "strange" Logging behaviour that has eaten up my whole account. I drilled deeped and found out that on all of my GKE clusters I have installed loki/promtail, I was getting a ton (7 million log entries per day) of following log entries:
level=error ts=2022-11-09T15:15:08.356391217Z caller=flush.go:146 org_id=fake msg="failed to flush user" err="store put chunk: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
I saw via kubectl that the writer pods where running, but not ready:
NAME READY STATUS RESTARTS AGE
pod/loki-gateway-6558c495d4-qfd8p 1/1 Running 0 4d5h
pod/loki-grafana-agent-operator-79867d6656-577xs 1/1 Running 0 4d6h
pod/loki-logs-898c6 2/2 Running 0 4d6h
pod/loki-logs-8xnfd 2/2 Running 0 4d7h
pod/loki-logs-c6dgs 2/2 Running 0 4d6h
pod/loki-logs-ccz64 2/2 Running 0 4d6h
pod/loki-logs-ffztb 2/2 Running 0 4d5h
pod/loki-logs-gqcwt 2/2 Running 0 4d7h
pod/loki-logs-qzf54 2/2 Running 0 4d7h
pod/loki-logs-w52jh 2/2 Running 0 4d5h
pod/loki-read-0 1/1 Running 0 4d5h
pod/loki-read-1 1/1 Running 0 4d6h
pod/loki-read-2 1/1 Running 0 4d7h
pod/loki-write-0 0/1 Running 0 4d4h
pod/loki-write-1 0/1 Running 0 4d6h
pod/loki-write-2 0/1 Running 0 4d5h
pod/promtail-2fqg7 1/1 Running 0 4d5h
pod/promtail-6w4q4 1/1 Running 0 4d5h
pod/promtail-d9zkv 1/1 Running 0 4d7h
pod/promtail-gqlbr 1/1 Running 0 4d7h
pod/promtail-jwdfk 1/1 Running 0 4d6h
pod/promtail-n64gw 1/1 Running 0 4d6h
pod/promtail-smb2b 1/1 Running 0 4d7h
pod/promtail-z7lgw 1/1 Running 0 4d6h
There was no change from my side since days and Loki/Promtail was up and running fine. I was able to search logs via Grafana, no issues.
To Reproduce Steps to reproduce the behavior:
- Install Loki via Helm Chart: https://bityl.co/Echb with following values.yaml:
loki:
auth_enabled: false
commonConfig:
replication_factor: 3
monitoring:
selfMonitoring:
enabled: true
write:
replicas: 3
resources:
limits:
cpu: 600m
requests:
cpu: 300m
memory: 384Mi
persistence:
storageClass: premium-rwo
size: "50Gi"
read:
replicas: 3
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 5
resources:
limits:
cpu: 550m
memory: 256Mi
requests:
cpu: 350m
memory: 128Mi
persistence:
storageClass: premium-rwo
gateway:
enabled: true
replicas: 1
verboseLogging: true
autoscaling:
enabled: true
minReplicas: 1
maxReplicas: 3
resources:
limits:
cpu: 150m
memory: 64Mi
requests:
cpu: 150m
memory: 64Mi
- Install promtail via Helm chart: https://bityl.co/EaaJ with following values.yaml:
resources:
requests:
cpu: 50m
memory: 128Mi
limits:
cpu: 100m
memory: 128Mi
Expected behavior
Environment:
- GKE:
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.8-gke.1900", GitCommit:"79209257257c051b27df67c567755783eda93353", GitTreeState:"clean", BuildDate:"2022-07-15T09:23:51Z", GoVersion:"go1.17.11b7", Compiler:"gc", Platform:"linux/amd64"} - Deployment tool: helm, helmfile, gitlab ci/cd
Questions
- what does this log entry actually means and why does it need to be repeated like 7M times a day? I am not sure why
aws.Config.CredentialsChainVerboseErrorsactually is showing up, since I am not running on AWS or have configured anything but the values.yaml listed above - Is it possible to reduce the amount of loki log entry?
Sorry if I am showing a bit of unprofessional feelings here but I am a bit nervous about upcomming GCP bill produced by the unnecessary amount of loki logs.
Cheers
I experienced this when using a Loki component that could not use credentials being injected via the Helm extraEnvFrom key.
So maybe you need to configure your global storage to use credentials directly.
@mrszop have you figured it out? I have exactly the same issue. I am not using AWS at all, though I see this issue that mentioned AWS, which makes no sense.
i also noticed that error message at startup
level=error ts=2024-03-05T08:20:46.798163605Z caller=reporter.go:205 msg="failed to delete corrupted cluster seed file, deleting it" err="NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors
I am using official helm chart
NAME CHART VERSION APP VERSION DESCRIPTION
grafana/loki 5.42.0 2.9.3 Helm chart
upd might me connected with https://github.com/grafana/loki/issues/5941
