I/O failure when trying to talk to `localhost`

[tronje]

When I try to connect to localhost, I get this error, but when I use 127.0.0.1 instead, everything is fine. I guess websocat can't resolve names or something? curl can talk to the server running on localhost just fine. Would be cool to have this fixed for ease of use. And thank you for the tool, it does seem to work quite well otherwise :+1:

E.g.:

user@hostname $ websocat ws://localhost:8080/ws
websocat: WebSocketError: I/O failure
websocat: error running

user@hostname $ websocat ws://127.0.0.1:8080/ws
# no error

user@hostname $ curl localhost:8080/ws
# no error

[vi]

websockat ws://... should resolve the name. Names are not resolved when specifying ip address for listening in server more or when using low-level tcp: connector.

Plese post what strace -e trace=network -s 4096 websocat ws://localhost:8080/ws outputs.

For me it is obvious that it resolved localhost to 127.0.0.1 and tried to establish connection. I suspect that the problem may be at server side: host part of URL is used not only for getting the IP address, but is also part of Host: header, depending on which web server may decide to serve something else.

[tronje]

strace.txt output of the strace call you asked for.

I think what you're saying makes sense -- I am using a simple echoing websocket built with actix-web. I also did not compile websocat with ssl support, if that helps.

Thanks for looking into this!

[vi]

It tries connecting to IPv6, make sure it is also being listened on server side.

You can use websocat as a websocket server as well. IPv4: websocat -s 127.0.0.1:8080, IPv6: websocat -s [::1]:8080. Does it connect to such servers using ws://localhost:8080/?

[peteclark3]

@vi - Not for me:

Run server:

ip-192-168-1-216:~$ websocat -s 127.0.0.1:8989
Listening on ws://127.0.0.1:8989/

Try to connect:

ip-192-168-1-216:~$ websocat ws://localhost:8989
websocat: WebSocketError: I/O failure
websocat: error running

[vi]

@peteclark3

• Is error message the same if you deliberately specify the wrong port, e.g. websocat ws://localhost:11111
• If you run nc -nvlp 8989 instead of websocat -s 127.0.0.1:8989 and try to connect to it with websocat ws://localhost:8989, do you see the HTTP request?
• Have you tried combinations of 127.0.0.1, 0.0.0.0, localhost? All fail?
• Maybe check if TCP works in general on this port: nc -nvlp 8989 as a server, nc -nv 127.0.0.1 8989 as a client?

[peteclark3]

I moved on to wsd

[vi]

@peteclark3 Does wsd work where websocat fails to connect?

[pshirshov]

I have the same issue on macOS:

❯ websocat -vvv ws://localhost:8081/ws
[INFO  websocat::lints] Auto-inserting the line mode
[DEBUG websocat] Done third phase of interpreting options.
[DEBUG websocat] Done fourth phase of interpreting options.
[DEBUG websocat] Preparation done. Now actually starting.
[INFO  websocat::sessionserve] Serving Line2Message(Stdio) to Message2Line(WsClient("ws://localhost:8081/ws")) with Options { websocket_text_mode: true, websocket_protocol: None, websocket_reply_protocol: None, udp_oneshot_mode: false, unidirectional: false, unidirectional_reverse: false, exit_on_eof: false, oneshot: false, unlink_unix_socket: false, exec_args: [], ws_c_uri: "ws://0.0.0.0/", linemode_strip_newlines: false, linemode_strict: false, origin: None, custom_headers: [], custom_reply_headers: [], websocket_version: None, websocket_dont_close: false, one_message: false, no_auto_linemode: false, buffer_size: 65536, broadcast_queue_len: 16, read_debt_handling: Warn, linemode_zero_terminated: false, restrict_uri: None, serve_static_files: [], exec_set_env: false, reuser_send_zero_msg_on_disconnect: false, process_zero_sighup: false, process_exit_sighup: false, socks_destination: None, auto_socks5: None, socks5_bind_script: None, tls_domain: None, tls_insecure: false, headers_to_env: [], max_parallel_conns: None, ws_ping_interval: None, ws_ping_timeout: None }
[INFO  websocat::stdio_peer] get_stdio_peer (async)
[INFO  websocat::stdio_peer] Setting stdin to nonblocking mode
[INFO  websocat::stdio_peer] Installing signal handler
[DEBUG websocat::sessionserve] Underlying connection established
[INFO  websocat::ws_client_peer] get_ws_client_peer
[DEBUG websocat::stdio_peer] restore_blocking_status
[INFO  websocat::stdio_peer] Restoring blocking status for stdin
websocat: WebSocketError: I/O failure
[DEBUG websocat::stdio_peer] restore_blocking_status
[INFO  websocat::stdio_peer] Restoring blocking status for stdin
websocat: error running

Wrong port:

❯ websocat -vvv ws://localhost:32121/ws
[INFO  websocat::lints] Auto-inserting the line mode
[DEBUG websocat] Done third phase of interpreting options.
[DEBUG websocat] Done fourth phase of interpreting options.
[DEBUG websocat] Preparation done. Now actually starting.
[INFO  websocat::sessionserve] Serving Line2Message(Stdio) to Message2Line(WsClient("ws://localhost:32121/ws")) with Options { websocket_text_mode: true, websocket_protocol: None, websocket_reply_protocol: None, udp_oneshot_mode: false, unidirectional: false, unidirectional_reverse: false, exit_on_eof: false, oneshot: false, unlink_unix_socket: false, exec_args: [], ws_c_uri: "ws://0.0.0.0/", linemode_strip_newlines: false, linemode_strict: false, origin: None, custom_headers: [], custom_reply_headers: [], websocket_version: None, websocket_dont_close: false, one_message: false, no_auto_linemode: false, buffer_size: 65536, broadcast_queue_len: 16, read_debt_handling: Warn, linemode_zero_terminated: false, restrict_uri: None, serve_static_files: [], exec_set_env: false, reuser_send_zero_msg_on_disconnect: false, process_zero_sighup: false, process_exit_sighup: false, socks_destination: None, auto_socks5: None, socks5_bind_script: None, tls_domain: None, tls_insecure: false, headers_to_env: [], max_parallel_conns: None, ws_ping_interval: None, ws_ping_timeout: None }
[INFO  websocat::stdio_peer] get_stdio_peer (async)
[INFO  websocat::stdio_peer] Setting stdin to nonblocking mode
[INFO  websocat::stdio_peer] Installing signal handler
[DEBUG websocat::sessionserve] Underlying connection established
[INFO  websocat::ws_client_peer] get_ws_client_peer
[DEBUG websocat::stdio_peer] restore_blocking_status
[INFO  websocat::stdio_peer] Restoring blocking status for stdin
websocat: WebSocketError: I/O failure
[DEBUG websocat::stdio_peer] restore_blocking_status
[INFO  websocat::stdio_peer] Restoring blocking status for stdin
websocat: error running

IP Address:

❯ websocat -vvv ws://127.0.0.1:8080/ws
[INFO  websocat::lints] Auto-inserting the line mode
[DEBUG websocat] Done third phase of interpreting options.
[DEBUG websocat] Done fourth phase of interpreting options.
[DEBUG websocat] Preparation done. Now actually starting.
[INFO  websocat::sessionserve] Serving Line2Message(Stdio) to Message2Line(WsClient("ws://127.0.0.1:8080/ws")) with Options { websocket_text_mode: true, websocket_protocol: None, websocket_reply_protocol: None, udp_oneshot_mode: false, unidirectional: false, unidirectional_reverse: false, exit_on_eof: false, oneshot: false, unlink_unix_socket: false, exec_args: [], ws_c_uri: "ws://0.0.0.0/", linemode_strip_newlines: false, linemode_strict: false, origin: None, custom_headers: [], custom_reply_headers: [], websocket_version: None, websocket_dont_close: false, one_message: false, no_auto_linemode: false, buffer_size: 65536, broadcast_queue_len: 16, read_debt_handling: Warn, linemode_zero_terminated: false, restrict_uri: None, serve_static_files: [], exec_set_env: false, reuser_send_zero_msg_on_disconnect: false, process_zero_sighup: false, process_exit_sighup: false, socks_destination: None, auto_socks5: None, socks5_bind_script: None, tls_domain: None, tls_insecure: false, headers_to_env: [], max_parallel_conns: None, ws_ping_interval: None, ws_ping_timeout: None }
[INFO  websocat::stdio_peer] get_stdio_peer (async)
[INFO  websocat::stdio_peer] Setting stdin to nonblocking mode
[INFO  websocat::stdio_peer] Installing signal handler
[DEBUG websocat::sessionserve] Underlying connection established
[INFO  websocat::ws_client_peer] get_ws_client_peer
[INFO  websocat::ws_client_peer] Connected to ws
fasd
[DEBUG websocat::ws_peer] incoming text
xxx
fsad
[DEBUG websocat::ws_peer] incoming text
xxx
fasd
[DEBUG websocat::ws_peer] incoming text
xxx
^C[DEBUG websocat::stdio_peer] restore_blocking_status
[INFO  websocat::stdio_peer] Restoring blocking status for stdin

At the same time I can connect without an issue to non-localhost servers:

❯ websocat -vvv ws://echo.websocket.org/
[INFO  websocat::lints] Auto-inserting the line mode
[DEBUG websocat] Done third phase of interpreting options.
[DEBUG websocat] Done fourth phase of interpreting options.
[DEBUG websocat] Preparation done. Now actually starting.
[INFO  websocat::sessionserve] Serving Line2Message(Stdio) to Message2Line(WsClient("ws://echo.websocket.org/")) with Options { websocket_text_mode: true, websocket_protocol: None, websocket_reply_protocol: None, udp_oneshot_mode: false, unidirectional: false, unidirectional_reverse: false, exit_on_eof: false, oneshot: false, unlink_unix_socket: false, exec_args: [], ws_c_uri: "ws://0.0.0.0/", linemode_strip_newlines: false, linemode_strict: false, origin: None, custom_headers: [], custom_reply_headers: [], websocket_version: None, websocket_dont_close: false, one_message: false, no_auto_linemode: false, buffer_size: 65536, broadcast_queue_len: 16, read_debt_handling: Warn, linemode_zero_terminated: false, restrict_uri: None, serve_static_files: [], exec_set_env: false, reuser_send_zero_msg_on_disconnect: false, process_zero_sighup: false, process_exit_sighup: false, socks_destination: None, auto_socks5: None, socks5_bind_script: None, tls_domain: None, tls_insecure: false, headers_to_env: [], max_parallel_conns: None, ws_ping_interval: None, ws_ping_timeout: None }
[INFO  websocat::stdio_peer] get_stdio_peer (async)
[INFO  websocat::stdio_peer] Setting stdin to nonblocking mode
[INFO  websocat::stdio_peer] Installing signal handler
[DEBUG websocat::sessionserve] Underlying connection established
[INFO  websocat::ws_client_peer] get_ws_client_peer
[INFO  websocat::ws_client_peer] Connected to ws

Though curl works:

❯ curl -v http://localhost:8080/http
*   Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 8080 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /http HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Connection: keep-alive
< Content-Type: text/plain
< Content-Length: 11
< Date: Tue, 22 Oct 2019 15:38:00 GMT
<
* Connection #0 to host localhost left intact
Hello World* Closing connection 0

❯ curl -v http://localhost:8080/ws
*   Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 8080 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /ws HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Connection: keep-alive
< Content-Length: 0
< Date: Tue, 22 Oct 2019 15:38:12 GMT
<
* Connection #0 to host localhost left intact
* Closing connection 0

[pshirshov]

An observation: I put a breakpoint into my connection handler - and there are no incoming requests at all. So, seems like it really fails on name resolution but I have no idea how may it happen.

[vi]

@pshirshov , Does name resolution from websockat ws:// URLs work in general? Does websocat ws://echo.websocket.org work?

[pshirshov]

Yeah, as you can see from my logs.

[pshirshov]

I think I know what happened there. localhost has been resolved to two addresses - v6 and v4. websocat tried to connect to v6, got rejected and didn't try v4.

By the way, I'm using macOS in case that matters.

[vi]

Websocat 2.0 is going to have Happy Eyeballs and try to connect to both, whichever is faster.

[JetForMe]

I'm having this same issue (works with IP address, not with localhost) on macOS. Just installed it today via brew. v1.9.0

[vi]

Unfortunately, it is likely to be resolved only with Websocat 3.0 (2.0 branch is abandoned now).

There is a workaround with current version 1.9 although: with explicit TCP connection specifier, like this websocat -t - ws-c:tcp:localhost:80, it should properly set up race between multiple IP addresses and use the first successfull. Hovewer, I have just experimented and found out that it is be buggy as well:

$ /opt/websocat -s 1234
Listening on ws://127.0.0.1:1234/
websocat: WebSocketError: I/O failure

---

$ /opt/websocat -v -v -t - ws-c:tcp:localhost:1234
[INFO  websocat::lints] Auto-inserting the line mode
[DEBUG websocat] Done third phase of interpreting options.
[INFO  websocat::net_peer] Resolving hostname to IP addresses
[INFO  websocat::net_peer] Got IP: [::1]:1234
[INFO  websocat::net_peer] Got IP: 127.0.0.1:1234
[DEBUG websocat] Done fourth phase of interpreting options.
[DEBUG websocat] Preparation done. Now actually starting.
...
[INFO  websocat::stdio_threaded_peer] get_stdio_peer (threaded)
[DEBUG websocat::sessionserve] Underlying connection established
[DEBUG websocat::net_peer] Setting up a race between multiple TCP client sockets. Who connects the first?
websocat: Connection refused (os error 111)
websocat: error running

Maybe I'll fix it to only return error if all attempts failed, not one.

[JetForMe]

No worries. I used brew to install it, so whenever that gets updated for 3.0, I can upgrade. Thanks!

[vi]

However, I have just experimented and found out that it is be buggy as well.

Fixed the bug in the workaround.

[TrueGit]

Thank you for making websocat. It's a great tool that has helped me a lot.

If you haven't already, I suggest you indicate its current "localhost" connection limitation relatively prominently in the documentation. Perhaps in the "limitations" section at the bottom of the current README.md file? It's trivial to work around if you know about it. It can be a scare-inducing trap for the unwary if you don't. Asking for a friend.

[msdrigg]

First of all I would like to say that websocat is awesome! It has been really helpful in my server dev.

But this is a consistently annoying pain point with using it. My most common use case for websocat is for localhost ipv4, so this issue comes up consistently. And judging by issue reactions, it's the most common problem for other users as well. Is there anything I (or others) could do to help get this resolved?

[vi]

Simplest thing is to introduce a message explaining the failure when "localhost" is detected in the URL and it fails to connect.

Other ways are:

• Fixing the legacy rust-websocket library to handle happy eyeballs properly. Unlikely to be worth it.
• Automatically applying the workaround with ws-c: instead of letting rust-websocket connect to TCP directly. It may lead to lower performance and/or introduce other bugs for all users.
• Implement a hack that auto-retries the connection attempt with localhost substituted to 127.0.0.1 if first one fails. Maybe only in the simple client mode to avoid breaking somebody's scripts.
• Migrate Websocat to modern dependencies and better architecture. I have already abandoned two such attempts, maybe the third one will be luckier? In any case it is a long-term project and Websocat v1 will need to be maintained for a while even when redesigned one becomes useful.

There are no real, extensive tests for Websocat, so making any changes may break things for users.

[msdrigg]

Migrate Websocat to modern dependencies and better architecture. I have already abandoned two such attempts, maybe the third one will be luckier?

That is sad and I feel your pain.

I didn't realize that websocat was in such a tough place dependency-wise. Looking at the rust-websocket recent commits I see you are the sole active maintainer of that project as well.

I wouldn't be capable of helping with such a rewrite unfortunately, but honestly the hack (option 3) would be likely to alleviate the pain. Also the tech debt wouldn't be too much of a problem especially considering how any major changes would likely involve a whole rewrite anyway.

[vi]

I wouldn't be capable of helping with such a rewrite unfortunately ...

I plan to (someday) rewrite it myself. After that it should be also more open to external contributions.

If you want to help, enhancing the testsuite be more beneficial. Testing framework can even be implemented in some other programming language. Having tests would both inspire a rewrite and bring confidence that it is successful and can be called the official Websocat.

honestly the hack (option 3) would be likely to alleviate the pain

Maybe I'll get round and just do it, as this ws://localhost issue gets mentioned again and again.

With the hack it would probably still print error message before connecting.

[msdrigg]

On macos, here's an interesting point of comparison:

me@myhostname ~ % nc 127.0.0.1 8899 -vvv
Connection to 127.0.0.1 port 8899 [tcp/*] succeeded!
^C
me@myhostname ~ % nc localhost 8899 -vvv   
nc: connectx to localhost port 8899 (tcp) failed: Connection refused
Connection to localhost port 8899 [tcp/*] succeeded!
^C
me@myhostname ~ % nc localhost -6 8899 -vvv 
nc: connectx to localhost port 8899 (tcp) failed: Connection refused
me@myhostname ~ % nc localhost -4 8899 -vvv 
Connection to localhost port 8899 [tcp/*] succeeded!
^C

So it seems nc uses the same try-till-it-works approach? I mean the extra log isn't an issue really.

Just throwing ideas out there, but maybe this could be made address-independent and less of a hack by using something like https://doc.rust-lang.org/std/net/trait.ToSocketAddrs.html to get a list of addresses to try for a given host:port pair.

I couldn't find it explicitly in the docs, but looking at the source, it looks like when calling ToSocketAddrs on localhost:12345, it would return both 127.0.0.1:12345 and the ipv6 version as long as getaddrinfo does (https://doc.rust-lang.org/src/std/sys_common/net.rs.html#207). Then even if rust-websocket can't connect to a SocketAddr explicitly websocat could convert each one back to a string with SocketAddr::to_string.

I don't know if this would be viable in websocat's architecture, but I just wanted to offer a possible solution.

[vi]

try-till-it-works approach

Proper approach is called "happy eyeballs" and it should try both connections methods simultaneously.

Websocat also supports it:

$ websocat -v -b - tcp:localhost:1234
[INFO  websocat::net_peer] Resolving hostname to IP addresses
[INFO  websocat::net_peer] Got IP: [::1]:1234
[INFO  websocat::net_peer] Got IP: 127.0.0.1:1234
[INFO  websocat::stdio_threaded_peer] get_stdio_peer (threaded)
[INFO  websocat::net_peer] Failure during connecting TCP: Connection refused (os error 111)
[INFO  websocat::net_peer] Connected to TCP 127.0.0.1:1234

and can use it for connecting to a WebSocket:

$ websocat -v -b - ws-c:tcp:localhost:1234
[INFO  websocat::net_peer] Resolving hostname to IP addresses
[INFO  websocat::net_peer] Got IP: [::1]:1234
[INFO  websocat::net_peer] Got IP: 127.0.0.1:1234
[INFO  websocat::stdio_threaded_peer] get_stdio_peer (threaded)
[INFO  websocat::net_peer] Failure during connecting TCP: Connection refused (os error 111)
[INFO  websocat::net_peer] Connected to TCP 127.0.0.1:1234
[INFO  websocat::ws_client_peer] get_ws_client_peer_wrapped
[INFO  websocat::ws_client_peer] Connected to ws

But that requires specifying WebSocket layer and TCP layer separately and requires specifying WebSocket URL separately.

Switching handler of plain simple $ websocat ws:// to this scheme is what I meaned by "Automatically applying the workaround with ws-c: ..." point above. It can degrade performance (not tested) or something else could regress (if there were tests I would be more confident).

Maybe using this method only when ws://localhost* or wss://localhost is found (not just with other hostnames) can make sense.

I.e. something like (pseudocode)

if (specified argument is ws[s]://localhost) then
   rewrite "ws[s]://localhost/path" with "ws-c:[tls:]tcp:localhost:80 --ws-c-url=ws://localhost/path [--tls-domain=localhost?]" in the command line
   continue with rewritten command line
else
  continue like usual
endif

We also need to make sure that --ws-c-url is not already used explicitly - ability to use the same node/overlay multiple times with distinct options is also a feature that should come with the redesign.

It can be extended with all hostnames (non-IP-address URLs), so it could fix and/or accelerate other client connections. But I don't know how to make sure everybody's scripts keep working well after that - it brings rarely used parts of Websocat to the spotlight and increase severity of bugs. Also the whole substitution scheme looks a bit fragile, especially if also apply it for complex command lines, not just simple client mode.