the reason behind 20 minutes kubelet executor suicide timeout default ?

[ravilr]

@jdef Observed in kubernetes v1.0.3 release.

why is the default kubelet executor's suicide timeout so large (20 minutes) ? kubelet executor seem to commit suicide after some period of "inactivity":

• no running tasks
• no task launch requests in the last X minutes

what we are seeing is, after an executor is asked to kill pod tasks, it kills them and enters into suicide timeout timer wait. meanwhile, the kube node controller sees the pods running on that slave node as healthy, as the kubelet seems to still report node status during this timer wait interval. Eventually, when the executor exits, node controller's pod evictor eventually kicks in (again after a default podEvictionTimeout of 5 mins!) and deletes the pod assignments to that slave node, after which Replication controller kicks in, recreates the pod and k8sm scheduler schedules the pod task.

Can you please validate the above analysis. Are there any downsides to running with lower executor timeout ? we are planning to run with a executor-suicide-timeout of 2 mins and implicit reconcile-interval down to 60secs.

[jdef]

thanks for the analysis! this part sounds like a bug in the scheduler:

meanwhile, the kube node controller sees the pods running on that slave node as healthy

running with a lower value for the executor suicide timeout is probably fine. the default of 20m was selected arbitrarily.

implicit reconciliation running every 60s sounds like it won't scale very well with large number of tasks in the cluster.

[jdef]

possibly fixed by #730 (and if so, fix is available in v0.7.2-v1.1.5) -- can you re-test?