How to undo a threadable prediction

[AstraLuma]

(This should probably be in the documentation.)

I'm using xonsh for some systems analysis, which involves grabbing the output of ssh commands.

Which xonsh won't let me do, because ssh is an interactive command (predicted threadable as False).

Please document a way to override predictions so I can actually do my bulk sysadmin stuff.

For community

⬇️ Please click the 👍 reaction instead of leaving a +1 or 👍 comment

[anki-code]

Hi @AstraLuma! Yeah, capturing requires improvements, you know it better than me )

I've tested this on Arch:

__xonsh__.commands_cache.threadable_predictors['ssh'] = lambda *a, **kw: True

!(ssh host -T "echo 1")                         # Auth by key without password
#CommandPipeline(
#  returncode=0,                                    # Correct return code
#  output='1\n',                                    # Correct stdout
#  errors=None
#)

!(ssh host -T "ls -error")
#ls: invalid option -- 'e'                             # Issue #4021
#Try 'ls --help' for more information.                 # Issue #4021
#
#CommandPipeline(
#  returncode=2,                                    # Correct return code
#  output='',
#  errors="ls: invalid option -- 'e'\nTry 'ls --help' for more information.\n"  # Correct stderr
#)

Also the way to avoid predictors manipulation:

!(bash -c 'ssh host -T "echo 123"')                         # Auth by key without password
#CommandPipeline(
#  returncode=0,                                            # Correct return code
#  output='123\n',                                          # Correct output
#  errors=None
#)


!(bash -c 'ssh host -T "ls -error" 2>&1')      # 2>&1 is to avoid uncaptured stderr output (#4021)
#CommandPipeline(
#  returncode=2,                                            # Correct return code
#  output="ls: invalid option -- 'e'\nTry 'ls --help' for more information.\n",  # Correct output
#  errors=None
#)

# Prepared by xontrib-hist-format

[anki-code]

I want to put here the pointer to the code that related of how threading flags work:

https://github.com/xonsh/xonsh/blob/5a792a6eec4889cbbc50619da6b26f7471d7ce5d/xonsh/procs/specs.py#L705-L728

As we can see the threading check is checking the last command and this is the cause why we have a workaround for some cases where we add | cat to the end of command. By doing this we switch threading to True.

How to trace the xonsh code

I've found that python-hunter is one of the tracing tools that works with xonsh out of the box:

pip install hunter

# Trace all
$PYTHONHUNTER='depth_lt=10,stdlib=False' $XONSH_DEBUG=1 xonsh -c 'echo 1'

# Trace the code around executing the subprocess commands
$PYTHONHUNTER='depth_lt=100,stdlib=False,module_startswith=\'xonsh.procs\'' $XONSH_DEBUG=1 xonsh -c 'echo 1'

[AstraLuma]

Right, that's what I did, but

1. This is a significant gotcha for a shell
2. How to un/redo it involves some serious insider knowledge
3. Even I had to do some serious digging to get this right.

Until the need for threadable predictors is no longer needed, I suggest a more friendly way to disable and enable.

[anki-code]

UPD: Outdated see #5424

Originally posted by @seanfarley in https://github.com/xonsh/xonsh/issues/3273#issuecomment-526719569:

Is there any easier way to run curses in this unthreadable mode? e.g. unthreadable python curses_script.py?

I like this idea. I made a draft for a threadable and unthreadable callable aliases as wrappers:

def _threading(args):
    if len(args) == 0:
        text = 'threadable' if $DO_THREADABLE else 'unthreadable'
        print(f'Run command with {text} predictor.')
        print(f'Usage: {text} <command with arguments>')
        return
        
    cmd = args[0]
    
    if p'{cmd}'.exists():
        cmd = p'{cmd}'.name
    
    prev_predictor = None

    if cmd in __xonsh__.commands_cache.threadable_predictors:
        prev_predictor = __xonsh__.commands_cache.threadable_predictors[cmd]

    __xonsh__.commands_cache.threadable_predictors[cmd] = lambda *a, **kw: $DO_THREADABLE

    ret = None
    try:
        if $DO_THREADABLE:
            ret = !(@(args))
        else:
            $[@(args)]
    finally:
        if prev_predictor is None:
            del __xonsh__.commands_cache.threadable_predictors[cmd]
        else:
            __xonsh__.commands_cache.threadable_predictors[cmd] = prev_predictor

    if $DO_THREADABLE:
        if ret is None:
            return 1
            
        if ret.out:
            print(ret.out, end='')
        if ret.errors:
            print(ret.errors, file=sys.stderr, end='')
        return ret.rtn
        

def _threadable(args):
    with __xonsh__.env.swap(DO_THREADABLE=True):
        return _threading(args)

        
def _unthreadable(args):
    with __xonsh__.env.swap(DO_THREADABLE=False):
        return _threading(args)
        
        
aliases['threadable'] = _threadable
aliases['unthreadable'] = _unthreadable

Now we can:

!(threadable ssh host -T "echo 1")          # Auth by key without password
#CommandPipeline(
#  returncode=0,                                    # Correct return code
#  output='1\n',                                    # Correct stdout
#  errors=None
#)

[anki-code]

I want to leave here the link to diving into issues and workaround for fzf - https://github.com/xonsh/xonsh/issues/2404#issuecomment-813056993

[djmattyg007]

I just wanted to note for the record that the same issue that affects git log and git diff (see #4243) also affects other important programs:

• systemctl when it needs to output to a pager
• journalctl when it needs to output to a pager
• journalctl when run with the --follow flag
• tail when run with the --follow flag

[daniel-shimon]

Hey, we've decided to disable interactive capturing by default and make it opt-in, see https://github.com/xonsh/xonsh/pull/4283

[djmattyg007]

Is there a timeline for that PR making its way into a release?

[daniel-shimon]

Yeah I'm working on it, it turns out to be a bit tricky since the threading code is a bit tangled and hardcoded. Currently there's an issue with capturing python aliases correctly (as you can see in the PR's failing tests). I'm pretty close to cracking it but can't promise a specific time-line My educated guess will be about ~2/3 weeks from now :)

[andrew222651]

For me, even !(echo hi) doesn't capture the output (!(echo hi).output == "").

[anki-code]

On Arch:

!(echo hi) == 'hi\n'
# True
!(echo hi).out == 'hi\n'
# True
!(echo hi).output == 'hi\n'
# False

yeah, there is some difference between output and out.

[andrew222651]

docker run --rm -it nixos/nix:2.3.16 /bin/sh
nix-channel --update
nix-shell -p xonsh
xonsh --version  
# xonsh/0.13.3
xonsh
!(echo hi).out == ""
# hi
# True

[anki-code]

I can repeat this (#5003 created). Deep dive is very welcome!