libssh2_channel_read() would block after libssh2_channel_request_auth_agent()

[z-image]

I first stumbled into this when trying to use the Rust bindings (ssh2-rs): https://github.com/alexcrichton/ssh2-rs/issues/200

Now I'm trying to see if it works in libssh2 directly, and I'm getting seemingly the same behaviour.

I've tried example/ssh2_agent_forwarding.c and replaced commandline to read

    const char *commandline = "echo $SSH_AUTH_SOCK; ssh-add -l";

I expected to see the output from the ssh-add -l but it just hangs there...

Here's the output with some additional debug prints added (also timestamps):

zimage@damage72:/tmp/libssh2/example$ ./ssh2_agent_forwarding 213.145.98.12 root 2>&1 |ts
ное 15 15:52:34         Authentication with username root and public key /home/zimage/.ssh/id_rsa succeeded!
ное 15 15:52:34 calling libssh2_channel_read()...
ное 15 15:52:34 libssh2_channel_read returned -37
ное 15 15:52:34 out of read() loop
ное 15 15:52:34 waitsocket() after EAGAIN
ное 15 15:52:34 calling libssh2_channel_read()...
ное 15 15:52:34 We read:
ное 15 15:52:34 /tmp/ssh-yWHEOIpmmc/agent.23377
ное 15 15:52:34
ное 15 15:52:34 calling libssh2_channel_read()...
ное 15 15:52:34 libssh2_channel_read returned -37
ное 15 15:52:34 out of read() loop
ное 15 15:52:34 waitsocket() after EAGAIN
ное 15 15:52:34 calling libssh2_channel_read()...
ное 15 15:52:34 libssh2_channel_read returned -37
ное 15 15:52:34 out of read() loop
ное 15 15:52:34 waitsocket() after EAGAIN
ное 15 15:52:44 calling libssh2_channel_read()...
ное 15 15:52:44 libssh2_channel_read returned -37
ное 15 15:52:44 out of read() loop
ное 15 15:52:44 waitsocket() after EAGAIN
ное 15 15:52:54 calling libssh2_channel_read()...
ное 15 15:52:54 libssh2_channel_read returned -37
ное 15 15:52:54 out of read() loop
ное 15 15:52:54 waitsocket() after EAGAIN

Is this example wrong or is there bug somewhere? What additional information would you need from me?

Version (please complete the following information):

• OS: Linux (Ubuntu 20.10)
• libssh2 version 1.9.0_DEV (git master cfe0bf64985fd6a5db3b45ffc31a2fe3b8fd9948)

[z-image]

I see in Guacamole they set a special libssh2 callback:

https://github.com/apache/guacamole-server/blob/master/src/protocols/ssh/ssh.c#L307

But there's no such thing as LIBSSH2_CALLBACK_AUTH_AGENT in current libssh2

[z-image]

And finally I tried with debug enabled libssh2 and here are the last few things that happen:

[libssh2] 0.328471 Conn: starting request(exec) on channel 0/0, message=ssh-add -l
=> libssh2_transport_write plain (18 bytes)
=> libssh2_transport_write plain2 (10 bytes)
[libssh2] 0.328751 Socket: Sent 80/80 bytes at 0x55958b80c278
=> libssh2_transport_write send() (80 bytes)
...
[libssh2] 0.328907 Transport: Looking for packet of type: 99
[libssh2] 0.328924 Transport: Looking for packet of type: 100
[libssh2] 0.333637 Socket: Recved 112/16384 bytes to 0x55958b808238+0
=> libssh2_transport_read() raw (112 bytes)
...
=> libssh2_transport_read() plain (9 bytes)
[libssh2] 0.333830 Transport: Packet type 93 received, length=9
[libssh2] 0.333848 Conn: Window adjust for channel 0/0, adding 2097152 bytes, new window_size=2097152
=> libssh2_transport_read() plain (5 bytes)
[libssh2] 0.333896 Transport: Packet type 99 received, length=5
[libssh2] 0.333913 Transport: Looking for packet of type: 99
[libssh2] 0.333936 Conn: channel_read() wants 16384 bytes from channel 0/0 stream #0
[libssh2] 0.371884 Conn: channel_read() wants 16384 bytes from channel 0/0 stream #0
[libssh2] 0.371990 Socket: Recved 80/16384 bytes to 0x55958b808238+0
=> libssh2_transport_read() raw (80 bytes)
...
=> libssh2_transport_read() plain (39 bytes)
0000: 5A 00 00 00 16 61 75 74  68 2D 61 67 65 6E 74 40 : Z....auth-agent@
0010: 6F 70 65 6E 73 73 68 2E  63 6F 6D 00 00 00 02 00 : openssh.com.....
0020: 01 00 00 00 00 40 00                             : .....@.
[libssh2] 0.372174 Transport: Packet type 90 received, length=39
[libssh2] 10.380144 Conn: channel_read() wants 16384 bytes from channel 0/0 stream #0

So the very last thing is Transport: Packet type 90 received with type 90 being SSH_MSG_CHANNEL_OPEN

Is the openssh server trying to open a new channel back to the example client? But then the client is ignoring its request and is trying to read data from channel 0/0 instead...

[z-image]

IMHO, it should catch this case like that (and also update documentation) until it is actually supported:

index 9897f77..f4ae950 100644
--- a/src/packet.c
+++ b/src/packet.c
@@ -984,6 +984,19 @@ _libssh2_packet_add(LIBSSH2_SESSION * session, unsigned char *data,
                 rc = packet_x11_open(session, data, datalen,
                                      &session->packAdd_x11open_state);
             }
+            else if((datalen >= (sizeof("[email protected]") + 4)) &&
+                     ((sizeof("[email protected]") - 1) ==
+                      _libssh2_ntohu32(data + 1))
+                     &&
+                     (memcmp(data + 5, "[email protected]",
+                             sizeof("[email protected]") - 1) == 0)) {
+                _libssh2_error(session, LIBSSH2_ERROR_METHOD_NOT_SUPPORTED,
+                               "CHANNEL_OPEN not supported for [email protected]");
+            }
+            else {
+                _libssh2_error(session, LIBSSH2_ERROR_METHOD_NOT_SUPPORTED,
+                               "Unknown CHANNEL_OPEN type received");
+            }
             if(rc == LIBSSH2_ERROR_EAGAIN)
                 return rc;

[pkittenis]

Agent forwarding should work, it was added in #219

Complete code to reproduce would be good. The example script is handling libssh2 sending EAGAIN.

[z-image]

On 1.12.20 г. 18:54, Panos wrote:

Agent forwarding should work, it was added in #219 https://github.com/libssh2/libssh2/pull/219

Complete code to reproduce would be good. The example script is handling libssh2 sending EAGAIN.

I'm just trying the example code from https://github.com/libssh2/libssh2/blob/master/example/ssh2_agent_forwarding.c

The only thing I changed is:

-const char *commandline = "uptime"; +const char *commandline = "ssh-add -l";

Does this ssh2_agent_forwarding.c example code work for you? It just hangs and never completes for me. With openssh client I get the expected result.

[z-image]

I'll try to explain what I think is the problem, maybe as a non-native speaker I can't get the idea across.

My understanding of how ssh agent forwarding works is this:

1. ssh client informs ssh server that agent forwarding is needed
2. ssh server creates an auth agent proxy: creates unix socket + sets up $SSH_AUTH_AGENT env
3. ssh server tries to open a new channel back to the client
4. ssh client must accept this new channel
5. ssh client has to somehow ensure data exchange between the new auth channel and the unix socket of the real local agent (proxy)
6. finally, a ssh client executed remotely on the server can access the ssh agent on the client through this new agent forwarding channel

So, in the example/ssh2_agent_forwarding.c there are all steps from 1 to 3. I can't see how one is supposed to do steps 4 & 5 neither in this example, nor in documentation, nor in the libssh2 code... there is for example x11 handling code, but no auth agent forwarding.

Are we supposed to somehow handle steps 4 & 5 by ourselves with some custom code called from libssh2_session_callback_set()? I was expecting this to work automatically OR a note in documentation that this is left as an exercise to the reader.

At least high level guidance would be appreciated.

[pkittenis]

Do not think examples or tests for handing the remote agent were ever made.

In your example the client is blocking on EAGAIN, presumably from trying to read from SSH_AUTH_SOCK on the remote while it has had no data written to it.

Meaning there would need to be code that handles '[email protected]' open channel requests from server with a handler function, or at least opening a channel for client to handle proxying.

So yes, the implementation looks incomplete at this point.

[z-image]

I found the set of patches implementing auth agent forwarding used by the apache guacamole server:

https://jira.glyptodon.com/browse/GUAC-432

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[pkittenis]

Bump.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[migueldeicaza]

I think that @z-image is correct, this functionality does not appear to exist.

When the remote end needs to access the agent, it sends a SSH_MSG_CHANNEL_OPEN back, with the type set to [email protected] which is currently ignored in _libssh2_packet_add. So this would likely need to accept the connection (Step #4) and then something would need to be wired up to callback into the calling code to handle the authentication channel.

On top of this, it might be necessary to have some helper code to fulfill the authentication request, rather than just passing it to a local agent.

[MichaelBuckley]

We have a working implementation of this in our copy of libssh2. It looks similar to the code in Guacamole. I believe the provinence of this work was patches discussed on the mailing list circa 2015 and earlier, which were never integrated into the main repository.

I'll work on getting an MR set up and will have something in a few days.

[migueldeicaza]

Music to my ears Michael!

[MichaelBuckley]

Still working on this. My week has been busier than expected, and I'll be in a lot of mettings tomorrow. Looks like I should also write man pages for the new functions, so it may take a bit longer.

[MichaelBuckley]

I've got a WIP PR for this https://github.com/libssh2/libssh2/pull/752 It contains what I believe are the relevant changes betwene our internal copy of libssh2 and this repository. We were pretty good about calling out in comments where we differ from the official release.

One thing that became clear when putting this together is that it's not a full implementation of agent forwarding. What we did was copy ssh-agent.c from the OpenSSH project into our soruce repostiory, along with other files from that project that it includes. We then modified authagent_process_input() to call the new libssh2 callbacks. In our LIBSSH2_CALLBACK_AUTHAGENT callback, we save off the channel, and in the code where we call libssh2_channel_read() on our data channel, we also call libssh2_channel_read() on this auth channel, passing the buffer to authagent_process_input(), then call authagent_getoutput(), pass that data to the auth channel using libssh2_channel_write(), and finally call libssh2_channel_write().

This PR would benefit by someone using it to confirm that it works, and possibly adding or editing some of the tests. (We have our own internal tests, and the docker commands used by the tests in this project have never worked on my machines.)

[vszakats]

Thanks everyone and in particular @MichaelBuckley for the PR. It has now been merged in bc4e619e76071393e466c29220fc4ef5764c2820.

[migueldeicaza]

I can confirm it works for those of us implementing our own agents