qtconsole
qtconsole copied to clipboard
jupyter
Output of os.system, subprocess.call goes to command prompt instead of qtconsole, on Windows 10, Open Suse
Problem Description
When running os.system or subprocess.call on Windows 10, no output is produced in qtconsole. Instead the output goes to the Anaconda Prompt console or, if qtconsole was started from the start menu, is simply discarded.
The same issue is reproducible with ipython qtconsole and and the sypder IDE, where this issue was originally reported (https://github.com/spyder-ide/spyder/issues/14783), but declared an depenency issue.
When running bare-bones consoles (ipython, python), the issue does not appear, presumably because everything is displayed in the same cmd.exe window.
On OpenSuse the issue occurs likewise. If the console is started from a terminal window, that terminal will receive the output of the commands. When originally posted, I thought I couldn't reproduce the issue on Linux.
Qt Console
As a component in Spyder
What steps reproduce the problem?
- Run
jupyter qtconsolefrom the Anaconda prompt. - In the qtconsole type
import os; os.system(r"mkdir C:\tmp\demo & dir C:\tmp\demo")
What is the expected output? What do you see instead?
The expectation is to see the output in the graphical console, together with the output of print or sys.stdout.write.
Instead the output is displayed in the command prompt window.
Potentially relevant package versions
- Python version: 3.8.5
- Qt version: 5.9.7
- PyQt version: 5.9.2, PyQt5: 5.12.3
- QtConsole: 5.0.2
- OS: Windows 10, Version 2004 (OS Build 19041.804)
DEPENDENCY REPORT OF SPYDER
atomicwrites >=1.2.0 : 1.4.0 (OK)
chardet >=2.0.0 : 4.0.0 (OK)
cloudpickle >=0.5.0 : 1.6.0 (OK)
diff_match_patch >=20181111 : 20200713 (OK)
intervaltree >=3.0.2 : 3.1.0 (OK)
IPython >=7.6.0 : 7.20.0 (OK)
jedi =0.17.2 : 0.17.2 (OK)
jsonschema >=3.2.0 : 3.2.0 (OK)
keyring >=17.0.0 : 22.0.1 (OK)
nbconvert >=4.0 : 6.0.7 (OK)
numpydoc >=0.6.0 : 1.1.0 (OK)
paramiko >=2.4.0 : 2.7.2 (OK)
parso =0.7.0 : 0.7.0 (OK)
pexpect >=4.4.0 : 4.8.0 (OK)
pickleshare >=0.4 : 0.7.5 (OK)
psutil >=5.3 : 5.8.0 (OK)
pygments >=2.0 : 2.7.4 (OK)
pylint >=1.0 : 2.6.0 (OK)
pyls >=0.36.2;<1.0.0 : 0.36.2 (OK)
pyls_black >=0.4.6 : 0.4.6 (OK)
pyls_spyder >=0.3.0 : 0.3.0 (OK)
qdarkstyle >=2.8 : 2.8.1 (OK)
qtawesome >=0.5.7 : 1.0.1 (OK)
qtconsole >=5.0.1 : 5.0.2 (OK)
qtpy >=1.5.0 : 1.9.0 (OK)
rtree >=0.8.3 : 0.9.4 (OK)
setuptools >=39.0.0 : 52.0.0.post20210125 (OK)
sphinx >=0.6.6 : 3.4.3 (OK)
spyder_kernels >=1.10.1;<1.11.0 : 1.10.1 (OK)
textdistance >=4.2.0 : 4.2.1 (OK)
three_merge >=0.1.1 : 0.1.1 (OK)
watchdog >=0.10.3 : 1.0.2 (OK)
zmq >=17 : 20.0.0 (OK)
cython >=0.21 : 0.29.21 (OK)
matplotlib >=2.0.0 : 3.3.4 (OK)
numpy >=1.7 : 1.19.2 (OK)
pandas >=1.1.1 : 1.2.1 (OK)
scipy >=0.17.0 : 1.6.0 (OK)
sympy >=0.7.3 : 1.7.1 (OK)
I can observe the issue on Linux too after all, with one difference: When executing the commands in Spyder's console pane, the output doesn't appear in the terminal from which Spyder was started either.
This is certainly a problem on Windows and I think it's not possible to fix it in the Jupyter architecture. I mean, there's no workaround or fix I'm aware of.
On Linux I can't reproduce it while using Spyder:
But that's because the kernels in our IPython console depend on the wurtlizer package, which takes care of redirecting stdout/stderr to the console.
Quick question (just to check): Are you using the Windows Subsystem for Linux or a real Linux installation/VM?
Quick question (just to check): Are you using the Windows Subsystem for Linux or a real Linux installation/VM?
The Linux environment is a separate desktop at work, running Open Suse with Anaconda Python and spyder/jupyter updated with pip3 install --upgrade, then starting Spyder/Jupyter QtConsole from the terminal in order to ensure that it isn't the more dated zypper version I am starting.
When I first posted this ticket though, I didn't see the issue on OpenSuse, only on Windows. I am not sure if any relevant packages have changed in between. There is a difference between the native Windows installation and the native OpenSuse installation, but I don't have a snapshot of the versions I had on OpenSuse, when the issue did not occur. I would assume they were the same, but I don't know if pip3 upgrades dependencies, that are already installed with a sufficiently but not up-to-date version.
It looks like wurlitzer isn't working the same in qtconsole as in terminals. (OpenSuse)
On Windows 10, it doesn't seem to be used at all and when installing it manually will fail, complaining about a missing package fnctl.
Output
(in ipython3 on terminal) (on terminal directly) (in qtconsole)
In [1]: %run /tmp/a.py >>> python3 /tmp/a.py In [3]: %run /tmp/a.py
____ Uncaptured output ____ ____ Uncaptured output ____ ____ Uncaptured output ____
____ Captured Stdout ____ ____ Captured Stdout ____ print function, stdout
print function, stdout print function, stdout print function, stderr
print function, stderr print function, stderr ____ Captured Stdout ____
os.system, stdout os.system, stdout os.system, stdout
subprocess.call, stdout subprocess.call, stdout subprocess.call, stdout
____ Captured Stderr ____ ____ Captured Stderr ____ ____ Captured Stderr ____
os.system, stderr os.system, stderr os.system, stderr
subprocess.call, stderr subprocess.call, stderr subprocess.call, stderr
____ end ____ ____ end ____ ____ end ____
The script /tmp/a.py
import os, wurlitzer, io, subprocess
out = io.StringIO()
err = io.StringIO()
print("____ Uncaptured output ____")
with wurlitzer.pipes(stdout=out, stderr=err):
print("print function, stdout")
print("print function, stderr")
os.system("echo os.system, stdout")
os.system("echo os.system, stderr >&2")
subprocess.call(["bash", "-c", "echo subprocess.call, stdout"])
subprocess.call(["bash", "-c", "echo subprocess.call, stderr >&2"])
print("____ Captured Stdout ____")
print(out.getvalue())
print("____ Captured Stderr ____")
print(err.getvalue())
print("____ end ____")
Version notes
>>> pip3 list | grep wurl
wurlitzer 2.0.1
Wurlitzer only works on Posix systems (sorry if I wasn't clear enough about it).
PR spyder-ide/spyder#1478 solves that problem, but only for consoles started from spyder. Also, if another client connects to the same console, it wouldn't recieve the messages either.
The ipython kernel creates a new sys.stdout. Any output that is written to sys.stdout or sys.stderr, for example using the print function, is sent to the client. The problem is that any non-python code is not aware of the new sys.stdout and uses the regular cout. On UNIX you can pipe that back to sys.stdout (wurlitzer does that) but I don't know of any way to do that on windows. In spyder we can control the process for the kernel (if it is started from spyder) and therefore the cout can be printed in the console (spyder-ide/spyder#14783) but this is not going through the regular ipython mechanisms. That means that the cout is not sent to any other clients.
