k8s-spot-rescheduler icon indicating copy to clipboard operation
k8s-spot-rescheduler copied to clipboard

Is the moving of pods logged?

Open max-rocket-internet opened this issue 5 years ago • 6 comments

Because all I see is this:

I0502 14:44:16.294689       1 rescheduler.go:247] No nodes to process.
I0502 14:44:26.443325       1 rescheduler.go:247] No nodes to process.
I0502 14:44:36.574006       1 rescheduler.go:247] No nodes to process.
I0502 14:44:46.724541       1 rescheduler.go:247] No nodes to process.
I0502 14:44:56.940652       1 rescheduler.go:247] No nodes to process.
I0502 14:45:07.118794       1 rescheduler.go:247] No nodes to process.
I0502 14:45:17.301454       1 rescheduler.go:247] No nodes to process.
I0502 14:45:27.464235       1 rescheduler.go:247] No nodes to process.

And sometimes this:

I0502 14:46:28.288639       1 rescheduler.go:205] Waiting for unschedulable pods to be scheduled.

max-rocket-internet avatar May 02 '19 14:05 max-rocket-internet

@max-rocket-internet did you get to the bottom of this?

sercanacar avatar May 28 '19 13:05 sercanacar

It's not logged but it should definitely be.

max-rocket-internet avatar May 28 '19 13:05 max-rocket-internet

Hey @max-rocket-internet and @sercanacar, which log setting do you have? The rescheduler uses glog and we typically run with level 2 -v=2 at @pusher. You are right that there is not specific log line for each individual pod being evicted, though i don't see why we couldn't add one, probably somewhere within this method https://github.com/pusher/k8s-spot-rescheduler/blob/b46cd362a3fb07159226b8d5a9872ad6f0de8dab/scaler/scaler.go#L71

JoelSpeed avatar May 30 '19 15:05 JoelSpeed

which log setting do you have?

I'm just using the chart: https://github.com/helm/charts/tree/master/stable/k8s-spot-termination-handler

there is not specific log line for each individual pod being evicted

I would say this is very important to log at default log level. Much more important than No nodes to process. 😄

max-rocket-internet avatar May 31 '19 08:05 max-rocket-internet

Yeah you're probably right... Levelled logging decisions are hard 😅

JoelSpeed avatar May 31 '19 08:05 JoelSpeed

Any update on this please?

aimanparvaiz avatar Mar 20 '20 14:03 aimanparvaiz