SSH & port forwarding: connections aren't closed in the server

[ItalyPaleAle]

• VSCode Version: 1.48.1
• Local OS Version: macOS 10.15.6
• Remote OS Version: Debian 10
• Remote Extension/Connection Type: SSH

This is a tricky one. Inside a remote-SSH, run the following Go code:

package main

import (
	"bytes"
	"fmt"
	"net/http"
)

func main() {
	http.HandleFunc("/test", func(w http.ResponseWriter, req *http.Request) {
		fmt.Println("Request starting")
		p := bytes.Repeat([]byte("x"), 1000)
		for i := 0; i < 100000; i++ {
			// Goroutine hangs here
			_, err := w.Write(p)
			if err != nil {
				fmt.Println("error:", err)
				return
			}
		}
		fmt.Println("Request done")
	})
	http.ListenAndServe("127.0.0.1:8090", nil)
}

Launch it using the terminal with:

go run .

Then forward port 8090 to the client.

1. In another terminal inside the remote server (recommend using an external terminal, as using the integrated terminal might make VS Code hang), run: curl http://localhost:8090/test then abort the request (CTRL+C). In the terminal that is running the app, you'll see an error: error: write tcp 127.0.0.1:8090->127.0.0.1:55264: write: connection reset by peer
2. In the local machine, run the same command: curl http://localhost:8090/test then like before abort the request (CTRL+C) before it's done. You'll see that nothing happens in the app that is running, no error message (it only shows Request starting).

This seems to be caused by the fact that the tunnel does not transmit the closing of the connection. In fact, if you add fmt.Println(i) before _, err := w.Write(p), you see that the app keeps transmitting data even after curl is stopped. That data probably goes into some stream in the system or in VS Code, and creates backpressure. At a certain point, because there's nothing consuming that stream, the stream is full and the app hangs, unable to add more data.

[roblourens]

Remote-SSH uses VS Code's tunnel forwarding, so I assume this is an issue with that.

[alexr00]

Yes, this is a problem somewhere in the port forwarding. We aren't handling the case where local socket gets an error (which is what is happening here). This results in an unhandled exception and all the behavior in the bug.

I've learned how to handle the error (by destroying the tunnel), but not how to pass the error back to remote (in this case, so that err != nil is true). When the error is handled but not passed back to the remote, the server successfully finished the request and doesn't get stuck as happens in the bug. This, at least, is progress.

To be continued once I learn more.

[ItalyPaleAle]

Thanks, Alex! Having the tunnel closing is a big improvement already, so the server doesn't hang (and the developer isn't left hanging too, trying to figure out what's happening :) )

[alexr00]

@alexdima and I took a long look at the issue and determined that:

• We were only "half" closing the connection before, which is what caused the server to hang. This is fixable.
• We cannot pass the Ctrl+C signal through the tunnel so the server will not get the error. Nothing to be done about this.

[alexdima]

Thanks @alexr00 !

[ItalyPaleAle]

Thank you @alexr00 and @alexdima !

[alexr00]

For verifier, follow the original steps in the bug (the try-go sample is a good place to start). In (2), the expected behavior is that the go app prints Request starting and then Request done. There will be no error like in (1).

[roblourens]

I see "Request starting", and a few minutes later, it's not done

[roblourens]

I don't think it's working... maybe I'm not interpreting it correctly but it seems hung.

[alexr00]

I'm still seeing this work as expected... let's try verifying again.

[alexdima]

I have checked via a docker container and it appears to work as expected for me. I see:

Request starting
Request done

I don't see an error, but that is expected, since the tunneling cannot simulate a broken socket.

@alexr00 Maybe remote ssh plugs in a different implementation for port forwarding? (I remember we discussed this some time ago, I'm not sure where it landed).

[alexr00]

@alexdima thanks for checking, I will also try with SSH.

[alexr00]

SSH is using the core port forwarding mechanism, but it still doesn't work. I have no idea why this is the case. I will investigate further.

[alexr00]

Moving to October since the solution is unclear.

[thediveo]

"moving to October" ... which October?

[Hadatko]

I hope this October :D

[Angrite]

I hope this October :D

Look like not ;)