kubernetes-operator icon indicating copy to clipboard operation
kubernetes-operator copied to clipboard

Set Loglevel

Open jhoelzel opened this issue 1 year ago • 4 comments

Dear Team,

how can i set the loglevel of the operator so my kibana is not spammed with info logs that do nothing for me?

Thanks in advance.

to put this into persepective, here are the last 24h for an active doppler system:

grafik

jhoelzel avatar Jan 12 '24 09:01 jhoelzel

Thanks for writing in about this, @jhoelzel!

We don't currently have any mechanisms for adjusting the log level of the operator. The logging library that we're using under the hood sends all logs to stderr by default, which we've had our eye on fixing. Is this the primary issue for you? Or would you prefer to simply disable all logs below a certain log level?

For Doppler internal reference, we're also tracking this issue under ENG-7553.

nmanoogian avatar Jan 29 '24 15:01 nmanoogian

+1 to this.

Hundreds of thousands of useless log messages per day. I think in production it should be possible to set it to only log warnings and errors.

The only use of info logs is seeing as the secrets are being synced, but the status of when a secret was last synced should be exposed either on the DopplerSecret (as a status field?) or via a metrics API of the operator instead of logging it every time.

miskr-instructure avatar Mar 05 '24 12:03 miskr-instructure

+1 Would like to see environment variables or command arguments that we can set in the helm chart values.yaml file to move LOG_LEVEL to different levels as needed.

gooseops avatar Mar 15 '24 19:03 gooseops

Bonus note: we should be able to mute rate limit warnings too, or they should be logged much less frequently. Even though they are warnings, if they happen once they'll keep reoccurring a lot and cause a lot of spam in the logs. It's not informative to see them dozens of times a minute.

miskr-instructure avatar Mar 18 '24 09:03 miskr-instructure

Also having this issue in GKE. Logging to stderr so showing as errors in GCP logging.

pnispel avatar Oct 02 '24 14:10 pnispel