garden icon indicating copy to clipboard operation
garden copied to clipboard

Port forwards get disconnected after awhile

Open eysi09 opened this issue 2 years ago • 4 comments

Bug

Current Behavior

One of my favourite features of Garden is that it automatically creates port forwards when in watch mode.

The annoying thing is that these seem to disconnect at some points when running the command for awhile.

I often notice this when using e.g. a Postgres client tool and can no longer query my DB. At this point I'll need to restart the Garden process which can take awhile, in particular if code I've written since I started the process triggers image builds.

Expected behavior

Well, in a perfect world I run garden dev when I clock in on Monday and kill the process when I clock out on Friday.

Under the hood, Garden is detecting if the port forward connection is severed and tries to reconnect if it is.

Of course the retries should timeout at some point, and then that should be reported to the user. I should needn't to find out the "hard way" that port forwarding no longer works.

Reproducible example

It's a bit hard to give a repro because I think Kubectl does some retrying on its own (that apparently doesn't always do the trick).

But the code where the Garden level retries are missing should be easy to spot. This is where we'd also add better error logging,

Suggested solution(s)

Mostly discussed above.

Your environment

  • OS: macOS
  • How I'm running Kubernetes: EKS

Running latest master as of writing.

eysi09 avatar Aug 26 '22 07:08 eysi09

Thanks for reporting this @eysi09! We'll take a look after SGv2 changes (0.13).

vvagaytsev avatar Aug 29 '22 09:08 vvagaytsev

+1 on this! I also believe something similar happens when using garden exec. I often have to restart my terminal because when the connection is lost while shelling into a container, garden becomes unresponsive and my zsh tab freezes completely. Please let me know if I should open a separate issue for this. ❤️

10ko avatar Aug 29 '22 09:08 10ko

+1 on this! I also believe something similar happens when using garden exec. I often have to restart my terminal because when the connection is lost while shelling into a container, garden becomes unresponsive and my zsh tab freezes completely. Please let me know if I should open a separate issue for this. ❤️

I've been seeing the same. I first thought it was my tmux setup but looks like it happens in other terminals as well. I think this should be a separate issue though. Would you mind opening one, @10ko?

eysi09 avatar Sep 12 '22 08:09 eysi09

Of course, no problem.

10ko avatar Sep 12 '22 09:09 10ko

Closing this as stale. Feel free to reopen this if there is a reproducible example.

vvagaytsev avatar Jul 11 '23 11:07 vvagaytsev