`triton ssh INST COMMAND` missing stdout with node >=0.12 and ControlMaster

[bahamat]

triton ssh <instance> appears to hang because there's no output from the command. Logging into the instance directly with ssh shows me that the triton nested session is in fact logged in, and that commands are being executed.

Typing exit, logout, or ^D returns me to my local shell.

---

Update (by @trentm 2016-03-11): adding a known issue for anyone hitting thise or curious about the workaround added in triton version 4.8.0.

known issue

triton instance ssh works by spawning ssh (with node's child_process.spawn).

There is a known issue with SSH connection multiplexing (a.k.a. ControlMaster, mux) where stdout/stderr is lost. As a workaround, ssh is spawned with options disabling ControlMaster.

If you want to use ControlMaster here is an alternative:

ssh root@$(triton ip <inst>)

[cehoffman]

I have a similar problem on node.js 5.4.0 and triton ssh. I can see that the ssh command has executed successfully, but it doesn't appear to be connected or piped through to the invoking shell.

[trentm]

Repro of the bug (at least for me):

$ node --version
v0.10.42
$ triton ssh vm1 hostname
20633805-9e11-4996-8b19-90424946db2d

But with node 4:

$ node --version
v4.3.0
$ ./bin/triton ssh vm1 hostname

No output

[trentm]

With some help from @misterdjules: He suspects that node >=4.x changes things so that when you use spawn with {stdio: 'inherit'} that the node process won't wait around for the std handles to flush before exiting. Haven't verified this though.

To work around this we above 'inherit' and manually pass through stdout/stderr content.

[bahamas10]

this is why i wish node had built in process.exec to just become a new process, and not have to fork and manage a child process

[trentm]

ditto on the real exec option.

FWIW, we can repro the error when ssh'ing to an IP for which I have ssh ControlMaster enabled. If I disable that, then no issue (or at least get luckier on a possible race).

[bahamas10]

@trentm we could check out https://github.com/jprichardson/node-kexec. I've never used it, but based on the docs it seems legit.

If ssh itself hangs because of a user configuration or option that's out of our jurisdiction.

[trentm]

binary module, not going there

[trentm]

ssh isn't hanging here, it is node stdio handle flushing

[bahamas10]

@trentm that's my only reservation... based on the nature of this desire though that's the only way it'll be possible - native extension.

I made an issue with node core a couple years ago saying there should be process.exec... idk whatever happened with it

[bahamas10]

ah my mistake then

[trentm]

Re-opening because the commit above is a workaround. This isn't fully understood.

[trentm]

@misterdjules's suspicion now is that it is node's console.log lazy mucking with standard handles interacting poorly with ssh's ControlMaster doing something with fd's

[bahamat]

FWIW, I do have ControlMaster auto in my ~/.ssh/config.

[trentm]

see PUBAPI-1275 (apparently the change above can cause problems?)

[trentm]

Problem is there is no TTY when we aren't using 'inherit' with node's spawn. So that breaks interactive ssh session. tl;dr: we need to back out the change above.

Then back to old situation: you don't get (all) the std output from triton ssh INST COMMAND when (a) using node >=4 and (b) using ControlMaster with ssh.

[bahamat]

I know it would be a hack to do so, but what about adding -o ControlMaster=none to the command line flags?

[trentm]

Brian, that would potentially force re-auth, which could fail. But yes that is the only option right now

A quick kexec attempt didn't work for me either. Would need to look into exec options there

--Trent

On Mar 10, 2016, at 6:43 PM, Brian Bennett [email protected] wrote:

I know it would be a hack to do so, but what about adding -o ControlMaster=none to the command line flags?

— Reply to this email directly or view it on GitHub.

[trentm]

Notes from discussion with @misterdjules about what might be going on:

---

jgilli [4:53 PM] so basically the problem seems to be that when /dev/tty is reopened by node when process.stdout is initialized (when the console global getter is called), the close-on-exec flag is set on that new fd that replaces fd 1, which means that any forked process will have its stdout closed

[4:54] and I tried a fix that seemed straightforward but it didn't fix the problem

jgilli [4:57 PM] trentm: the part where it gets tricky is that on OSX it seems that async I/O with kqueue doesn't work on /dev/tty, so node does something a bit complex where it creates a thread to call select on that new fd. And that makes it a bit trick to debug/trace what's going on :disappointed: (edited)

[5:02] trentm: this is basically the code that reopens /dev/tty: https://github.com/nodejs/node/blob/master/deps/uv/src/unix/tty.c#L64-L87

[5:02] trentm: and this is the OSX workaround for /dev/tty not being pollable via kqueue: https://github.com/nodejs/node/blob/master/deps/uv/src/unix/tty.c#L96-L104

[5:03] trentm: which is implemented like this: https://github.com/nodejs/node/blob/master/deps/uv/src/unix/stream.c#L134-L386

[5:04] trentm: not calling dup2 and not setting close on exec on the fd 1 here: https://github.com/nodejs/node/blob/master/deps/uv/src/unix/tty.c#L76 "fixes" the problem

[5:05] trentm: but it's gross, at least that's what I thought at the time

jgilli [5:06 PM] trentm: because you end up with two different fds for writing to the tty, and process.stdout is not fd 1 but the next available fd when you first use console.log or process.stdout

[5:08] trentm: my original attempt at fixing it was to use the standard dup2 call, which does not set close on exec, and not using uv__open_cloexec to open /dev/tty but the standard open syscall, but that didn't fix the problem, and I didn't spend more time than that

---

Note that it still isn't clear why disabling ControlMaster makes the difference here.

[misterdjules]

Note that it still isn't clear why disabling ControlMaster makes the difference here.

Right, because it seems that if the issue was about the standard output/error fds' having their close-on-exec flag et to true, then it would be an issue even when not using ControlMaster. One of the differences between ssh running with and without ControlMaster is that when ControlMaster is used, the standard output/error fds are passed from the ssh client process to the control master process through a UNIX socket. I don't know why that would make a difference so.

In other words, I need to spend more time investigating that issue, probably on SmartOS so that I can DTrace it without getting the noise from the separate thread used on OSX to poll /dev/tty.

[trentm]

triton 4.8.0 is published to npm with the workaround