runtime logs are not present

[chaporgin]

Expected Behavior

Runtime logs are seen

Current Behavior

Runtime logs are sometimes not seen after long time of running tilt (10s of hours)

Steps to Reproduce

1. Have docker-compose file, no kubernetes
2. Have the docker_build directive
3. Change some code
4. Watch build logs finish successfully
5. No runtime logs are seen in the tilt console, but they are seen with docker logs -n10 docker ps | grep container_name | cut -d" " -f1`

Context

tilt doctor Output

$ tilt doctor
Tilt: v0.30.13, built 2022-12-01
System: darwin-arm64
---
Docker
- Host: unix:///var/run/docker.sock
- Server Version: 20.10.20
- API Version: 1.41
- Builder: 2
- Compose Version: v2.12.1
---
Kubernetes
- Env: unknown
- Context: removed manually
- Cluster Name: removed manually
- Namespace: default
- Container Runtime: docker
- Version: v1.21.14-eks-fb459a0
- Cluster Local Registry: none
---
Thanks for seeing the Tilt Doctor!
Please send the info above when filing bug reports. 💗

The info below helps us understand how you're using Tilt so we can improve,
but is not required to ask for help.
---
Analytics Settings
--> (These results reflect your personal opt in/out status and may be overridden by an `analytics_settings` call in your Tiltfile)
- User Mode: opt-out
- Machine: c5fb1592e4eab9c0bd2124b58977e568
- Repo: tb987Z6kvKu+U4t+WzVbyw==

[nicksieger]

Thanks for the report! I personally have seen this at times as well. I think #5826 might help with this. I'll see if it's something we can revive and get ready for merge.

[nicks]

ya, lemme revive that pr and see if it helps. when we hooked up our kubernetes log ingester to a reconcilation loop, a lot of weird transient issues went away (thanks to the inherent self-healing design of the reconcilation loop). hoping the same thing will happen here.