Forwarding numpy calculations over SSH results in single-core operation

[spencertipping]

If I run from my laptop and use s to forward a by'' section to a 64-core server, it runs single-threaded.

On the other hand, if I run by'' directly on the server, it runs multicore. I suspect this has to do with environment variables, but I use the same .bashrc on both systems so I'm not sure. We may be breaking stuff if we forward and overwrite $PATH (are we getting the wrong python3 and libs?).

[spencertipping]

Ok, we are not forwarding $PATH (which is good). However, we also aren't evaluating ~/.bashrc or even ~/.profile on the remote system when logging in. So if we have, say, custom pip or conda stuff, I doubt it's being used.

That explains that much. Now the question is what we should do about it. Should we use bash -l to wrap remote ni?

[spencertipping]

Ohhh, OK I see what's going on.

First, bash -l is basically the right idea if we want initialization -- however it doesn't set interactive-mode, which means we won't get pip/conda path settings since those are appended to .bashrc, not .profile. Normally that wouldn't matter, but .bashrc typically begins with a bailout if it isn't running interactively. Even ssh server 'export PS1=1; bash -l -c env' fails to evaluate .bashrc; bash itself seems to understand that it isn't interactive and clears PS1 for us.

Why this bug exists

...because pip and conda do it the wrong way. They should put their setup into .profile instead of .bashrc. You can see the problem if you just ssh machine 'bash -l python3'; you won't have the right Python if you've got local miniconda.

A solution might exist

bashrc normally begins with [ -z "$PS1" ] && return to bailout if the shell is non-interactive. I've just updated bashrc-tmux to repeat that check, which frees the normal bashrc logic up to do something more general: [ -t 0 ] || return (bailout unless stdin is a terminal).

I'm trying some variations on the following:

1. Pull the latest bashrc-tmux if you're using it (if using dotbash, you can get it with dotbash pull)
2. Replace [ -z "$PS1" ] && return with [ -t 0 ] || return in .bashrc
3. Use the obscure multi-option syntax for the s operator: ni s[-t -t machine] e[bash -l -c env]

This almost works but doesn't quite. First, any stdout messages from .bashrc are echoed into the output stream, which isn't right; and second, ni itself is echoed, possibly due to something about the way it forwards to stdin (i.e. maybe bash is echoing what it assumes we've typed).

Moving the conda/pip init code into .profile solves half the problem, but even then we still need bash -l somewhere. I could bake that into the SSH operator but I'm not sure it's the right approach.