Bash cells or subprocesses show no intermediate output

[nilsvu]

I took the liberty to duplicate @MarkDunne 's issue https://github.com/jupyter/notebook/issues/2023 over, since it appears the problem lies within the IPython kernel and not the Jupyter notebook.

To reproduce, simply note that in a bash cell, the stdout stream is only displayed on termination of the process:

%%bash
echo "Hello 1" # This line is displayed only after the process has terminated.
sleep 5
echo "Hello 2"

To give a more practical use case, I suspect it is the same issue that prohibits reading the stdout stream from subprocess.Popen in a Jupyter notebook until the process has terminated:

import subprocess
p = subprocess.Popen(
    "some/long/process/with/output",
    stdout=subprocess.PIPE,
    universal_newlines=True
)
for line in p.stdout:
    print(line, end="")
# BUG: This waits until the process is done, then prints all output at once

Please correct me if there is a way to receive intermediate output in a Jupyter notebook cell I have missed.

[Icarwiz]

I found an elegant hack to this problem : https://stackoverflow.com/questions/52545512/realtime-output-from-a-shell-command-in-jupyter-notebook It's an interesting start if someone wants to integrate it directly in Notebook...

[maciejb]

Another workaround I have found is instead of a %%bash block to use !:

!echo "Hello 1"
!sleep 5
!echo "Hello 2"

or in a file:

!./hello.sh

Obviously not ideal, but hopefully useful to someone looking here for a quick fix.

[lukelbd]

This is a bit... specific, but that stackoverflow solution posted by @lcarwiz also seems to fix some bizarre and frustrating issues with the matplotlib_iterm2 backend in jupyter console (oselivanov/matplotlib_iterm2#3). After forking the project and replacing the lines subprocess.call(['imgcat', f.name]) with:

from subprocess import Popen, PIPE, STDOUT
process = Popen(
    ['imgcat', f.name],
    stdout=PIPE,
    stderr=STDOUT,
    close_fds=True,
)
for line in iter(process.stdout.readline, b''):
    print(line.rstrip().decode('utf-8'))
process.stdout.close()
process.wait()

everything works fine in both ipython and jupyter console (I also tried python-imgcat and itermplot but the resolution of the resulting images is tiny... only matplotlib_iterm2 with command-line imgcat combined with this workaround works).