restart policy for local_resource(serve_cmd)

[nicks]

In the #tilt channel, buzzedword asks:

i have a local_resource defined as follows:

  local_resource('next dev server', 
    serve_cmd='npm run build:watch', 
    deps=['package.json'], 
    labels='frontend', 
    trigger_mode=TRIGGER_MODE_MANUAL,
    readiness_probe=probe(
      initial_delay_secs=120,
      period_secs=15,
      failure_threshold=4,
      http_get=http_get_action(port=3003, path="/")
    )
  )

every once in a while, the npm task may crash, and needs to be restarted. before i start barking down disassembling the cmd itself, do y'all have any tilt native recommendations to restart a local_resource if it sees a non-zero exit code

I've definitely felt this tension before: local_resource(serve_cmd) is supposed to be the host process equivalent of Deployment or a Pod. It always makes sure that exactly one instance of a server is running. But it doesn't have any restart semantics. If the host process dies, tilt just marks the resource failed.

I think the right way to handle this is to have some sort of serve_restart_policy that tells tilt what to do when a serve process fails, similar to Pod's RestartPolicy

https://pkg.go.dev/k8s.io/api/core/v1#RestartPolicy

where possible values could be Never, OnFailure, Always

[nicks]

(the current workaround is to have serve_cmd invoke a script that handles restart semantics, like entr or nodemon)

[arik-so]

Seconding this issue. It would be really nice to have some sort of built-in retry/restart semantics for serve commands.

Going beyond the K8s restart policy referenced above, it might be prudent to have a restart attempt interval, i. e. every couple minutes as opposed to immediately upon failure, and perhaps a restart attempt limit.