Retryability for live updates on what seems to be a recoverable error

[krijoh92]

Expected Behavior

Live update should retry on recoverable errors, or at least be configurable to do so.

Current Behavior

Currently live update doesn't retry on the following error:

Will copy 1 file(s) to container: [<podname>/path]
- 'path/to/file.tsx' --> '/remote/path/to/file.tsx'
Build Failed: Updating pod <podname>: copying changed files: error dialing backend: EOF
LiveUpdate "name:registry.example.com_stuff_here" UpdateFailed: Updating pod <podname>: copying changed files: error dialing backend: EOF

While I'm still able to exec into the pod, get logs from it and interact with it like any other time. Might just be a hickup in my home-network that causes this, so it stings having to rebuild the image for the service whenever this happens.

Context

tilt doctor Output

-> tilt doctor
Tilt: v0.30.4, built 2022-06-16
System: linux-amd64
---
Docker
- Host: unix:///var/run/docker.sock
- Server Version: 20.10.12
- API Version: 1.41
- Builder: 2
---
Kubernetes
- Env: unknown
- Context: rpi8s
- Cluster Name: rpi8s
- Namespace: ---
- Container Runtime: containerd
- Version: v1.23.8+k3s2
- Cluster Local Registry: none

[padzikm]

I have the same issue - sample repo: https://github.com/padzikm/TiltSample

Output from tilt ui logs:

1 time file changed: Will copy 4 file(s) to container: [hello-tilt-back2-5894c86bb6-82bwg/hello-tilt-back2]

• '/Users/majki/Code/tilt-demo/backend2/out/TiltDemoApi2' --> '/app/out/TiltDemoApi2'
• '/Users/majki/Code/tilt-demo/backend2/out/TiltDemoApi2.dll' --> '/app/out/TiltDemoApi2.dll'
• '/Users/majki/Code/tilt-demo/backend2/out/TiltDemoApi2.pdb' --> '/app/out/TiltDemoApi2.pdb'
• '/Users/majki/Code/tilt-demo/backend2/out/web.config' --> '/app/out/web.config' Build Failed: Updating pod hello-tilt-back2-5894c86bb6-82bwg: copying changed files: error dialing backend: EOF LiveUpdate "hello-tilt-back2:hello-tilt-back2" UpdateFailed: Updating pod hello-tilt-back2-5894c86bb6-82bwg: copying changed files: error dialing backend: EOF

2 time file changed: full image rebuild and redeploy succeeded

3 time file changed: livereload succeeded

4 time file changed: Will copy 4 file(s) to container: [hello-tilt-back2-7955fc4958-v9jpz/hello-tilt-back2]

• '/Users/majki/Code/tilt-demo/backend2/out/TiltDemoApi2' --> '/app/out/TiltDemoApi2'
• '/Users/majki/Code/tilt-demo/backend2/out/TiltDemoApi2.dll' --> '/app/out/TiltDemoApi2.dll'
• '/Users/majki/Code/tilt-demo/backend2/out/TiltDemoApi2.pdb' --> '/app/out/TiltDemoApi2.pdb'
• '/Users/majki/Code/tilt-demo/backend2/out/web.config' --> '/app/out/web.config' [CMD 1/1] sh -c date > /tmp/.restart-proc → Container hello-tilt-back2-7955fc4958-v9jpz/hello-tilt-back2 updated! info: Microsoft.Hosting.Lifetime[0] Application is shutting down... info: Microsoft.Hosting.Lifetime[14] Now listening on: http://[::]:80 info: Microsoft.Hosting.Lifetime[0] Application started. Press Ctrl+C to shut down. info: Microsoft.Hosting.Lifetime[0] Hosting environment: Production info: Microsoft.Hosting.Lifetime[0] Content root path: /app/out/ Reconnecting... Error port-forwarding hello-tilt-back2 (8082 -> 80): error upgrading connection: error dialing backend: EOF Reconnecting... Error port-forwarding hello-tilt-back2 (8082 -> 80): error upgrading connection: error dialing backend: x509: certificate is valid for k3d-k3s-default-server-0, localhost, not k3d-k3s-default-server-1 Error streaming hello-tilt-back2-7955fc4958-v9jpz logs: unexpected EOF Reconnecting... Error port-forwarding hello-tilt-back2 (22001 -> 22): error upgrading connection: error sending request: Post "https://0.0.0.0:53115/api/v1/namespaces/default/pods/hello-tilt-back2-7955fc4958-v9jpz/portforward": EOF Reconnecting... Error port-forwarding hello-tilt-back2 (8082 -> 80): error upgrading connection: error dialing backend: x509: certificate is valid for k3d-k3s-default-server-2, localhost, not k3d-k3s-default-server-1 Reconnecting... Error port-forwarding hello-tilt-back2 (22001 -> 22): error upgrading connection: error dialing backend: x509: certificate is valid for k3d-k3s-default-server-0, localhost, not k3d-k3s-default-server-1 Reconnecting... Error port-forwarding hello-tilt-back2 (8082 -> 80): error upgrading connection: error dialing backend: x509: certificate is valid for k3d-k3s-default-server-0, localhost, not k3d-k3s-default-server-1 Reconnecting... Error port-forwarding hello-tilt-back2 (22001 -> 22): error upgrading connection: error dialing backend: x509: certificate is valid for k3d-k3s-default-server-2, localhost, not k3d-k3s-default-server-1 Reconnecting... Error port-forwarding hello-tilt-back2 (22001 -> 22): error upgrading connection: error dialing backend: x509: certificate is valid for k3d-k3s-default-server-2, localhost, not k3d-k3s-default-server-1 Reconnecting... Error port-forwarding hello-tilt-back2 (8082 -> 80): error upgrading connection: error dialing backend: x509: certificate is valid for k3d-k3s-default-server-2, localhost, not k3d-k3s-default-server-1 Reconnecting... Error port-forwarding hello-tilt-back2 (22001 -> 22): error upgrading connection: error dialing backend: x509: certificate is valid for k3d-k3s-default-server-2, localhost, not k3d-k3s-default-server-1 Reconnecting... Error port-forwarding hello-tilt-back2 (8082 -> 80): error upgrading connection: error dialing backend: x509: certificate is valid for k3d-k3s-default-server-2, localhost, not k3d-k3s-default-server-1 Reconnecting... Error port-forwarding hello-tilt-back2 (22001 -> 22): error upgrading connection: error dialing backend: x509: certificate is valid for k3d-k3s-default-server-2, localhost, not k3d-k3s-default-server-1 Error streaming hello-tilt-back2-7955fc4958-v9jpz logs: Get "https://k3d-k3s-default-server-1:10250/containerLogs/default/hello-tilt-back2-7955fc4958-v9jpz/hello-tilt-back2?follow=true&sinceTime=2022-08-09T10%3A21%3A48Z": x509: certificate is valid for k3d-k3s-default-server-2, localhost, not k3d-k3s-default-server-1 Reconnecting... Error port-forwarding hello-tilt-back2 (8082 -> 80): error upgrading connection: error dialing backend: x509: certificate is valid for k3d-k3s-default-server-2, localhost, not k3d-k3s-default-server-1 Reconnecting... Error port-forwarding hello-tilt-back2 (22001 -> 22): error upgrading connection: error dialing backend: x509: certificate is valid for k3d-k3s-default-server-2, localhost, not k3d-k3s-default-server-1 Error streaming hello-tilt-back2-7955fc4958-v9jpz logs: Get "https://k3d-k3s-default-server-1:10250/containerLogs/default/hello-tilt-back2-7955fc4958-v9jpz/hello-tilt-back2?follow=true&sinceTime=2022-08-09T10%3A21%3A50Z": x509: certificate is valid for k3d-k3s-default-server-2, localhost, not k3d-k3s-default-server-1 Reconnecting... Error port-forwarding hello-tilt-back2 (8082 -> 80): error upgrading connection: error dialing backend: x509: certificate is valid for k3d-k3s-default-server-2, localhost, not k3d-k3s-default-server-1 Reconnecting... Error port-forwarding hello-tilt-back2 (22001 -> 22): error upgrading connection: error dialing backend: x509: certificate is valid for k3d-k3s-default-server-2, localhost, not k3d-k3s-default-server-1 Reconnecting... Error port-forwarding hello-tilt-back2 (8082 -> 80): error upgrading connection: error dialing backend: EOF Error streaming hello-tilt-back2-7955fc4958-v9jpz logs: Get "https://k3d-k3s-default-server-1:10250/containerLogs/default/hello-tilt-back2-7955fc4958-v9jpz/hello-tilt-back2?follow=true&sinceTime=2022-08-09T10%3A21%3A54Z": EOF Reconnecting... Error port-forwarding hello-tilt-back2 (22001 -> 22): error upgrading connection: error dialing backend: proxy error from 127.0.0.1:6443 while dialing k3d-k3s-default-server-1:10250, code 500: 500 Internal Server Error [event: pod hello-tilt-back2-7955fc4958-v9jpz] Container image "demoregistry:5000/hello-tilt-back2:tilt-f8c288f5a6f12348" already present on machine Detected container restart. Pod: hello-tilt-back2-7955fc4958-v9jpz. Container: hello-tilt-back2.

Oftentimes there is one livereload success and on next file changes error: copying changed files: error dialing backend: EOF Container restarts and especially rebuilding and redeploying is time consuming to watch what failed when working with larger project

tilt doctor: Tilt: v0.30.6, built 2022-08-04 System: darwin-amd64 Docker

• Host: unix:///var/run/docker.sock
• Server Version: 20.10.16
• API Version: 1.41
• Builder: 2
• Compose Version: v2.5.1 Kubernetes
• Env: k3d
• Context: k3d-k3s-default
• Cluster Name: k3d-k3s-default
• Namespace: default
• Container Runtime: containerd
• Version: v1.23.8+k3s1
• Cluster Local Registry: &RegistryHosting{Host:localhost:53116,HostFromClusterNetwork:demoregistry:5000,HostFromContainerRuntime:demoregistry:5000,Help:https://k3d.io/usage/guides/registries/#using-a-local-registry,SingleName:,}