Can't run kernel/client when `sys.stdout` is closed.

[jhidding]

Not sure if this is an issue with jupyter_client or panflute. Taking this minimal example (call it test-jupyter.py):

#!/usr/bin/env python3.7

import panflute
import jupyter_client
import sys

def action(elem, doc):
    pass

def finalize(doc):
    with jupyter_client.run_kernel(kernel_name="python3") as kc:
        print(kc.is_alive(), file=sys.stderr)

def main(doc=None):
    return panflute.run_filter(action, finalize=finalize)

if __name__ == "__main__":
    main()

And running it with

chmod +x jupyter_test.py
pandoc -t plain --filter jupyter_test.py

(press Ctrl-D to end input) Gives the following error:

Traceback (most recent call last):
  File "/home/johannes/.local/lib/python3.7/site-packages/traitlets/traitlets.py", line 528, in get
    value = obj._trait_values[self.name]
KeyError: 'kernel_dirs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./test-jupyter.py", line 18, in <module>
    main()
  File "./test-jupyter.py", line 15, in main
    return panflute.run_filter(action, finalize=finalize)
  File "/home/johannes/.local/lib/python3.7/site-packages/panflute/io.py", line 260, in run_filter
    return run_filters([action], *args, **kwargs)
  File "/home/johannes/.local/lib/python3.7/site-packages/panflute/io.py", line 244, in run_filters
    finalize(doc)
  File "./test-jupyter.py", line 11, in finalize
    with jupyter_client.run_kernel(kernel_name="python3") as kc:
  File "/usr/lib64/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/johannes/.local/lib/python3.7/site-packages/jupyter_client/manager.py", line 490, in run_kernel
    km, kc = start_new_kernel(**kwargs)
  File "/home/johannes/.local/lib/python3.7/site-packages/jupyter_client/manager.py", line 468, in start_new_kernel
    km.start_kernel(**kwargs)
  File "/home/johannes/.local/lib/python3.7/site-packages/jupyter_client/manager.py", line 246, in start_kernel
    kernel_cmd = self.format_kernel_cmd(extra_arguments=extra_arguments)
  File "/home/johannes/.local/lib/python3.7/site-packages/jupyter_client/manager.py", line 170, in format_kernel_cmd
    cmd = self.kernel_spec.argv + extra_arguments
  File "/home/johannes/.local/lib/python3.7/site-packages/jupyter_client/manager.py", line 82, in kernel_spec
    self._kernel_spec = self.kernel_spec_manager.get_kernel_spec(self.kernel_name)
  File "/home/johannes/.local/lib/python3.7/site-packages/jupyter_client/kernelspec.py", line 234, in get_kernel_spec
    resource_dir = self._find_spec_directory(kernel_name.lower())
  File "/home/johannes/.local/lib/python3.7/site-packages/jupyter_client/kernelspec.py", line 205, in _find_spec_directory
    for kernel_dir in self.kernel_dirs:
  File "/home/johannes/.local/lib/python3.7/site-packages/traitlets/traitlets.py", line 556, in __get__
    return self.get(obj, cls)
  File "/home/johannes/.local/lib/python3.7/site-packages/traitlets/traitlets.py", line 535, in get
    value = self._validate(obj, dynamic_default())
  File "/home/johannes/.local/lib/python3.7/site-packages/jupyter_client/kernelspec.py", line 151, in _kernel_dirs_default
    from IPython.paths import get_ipython_dir
  File "/home/johannes/.local/lib/python3.7/site-packages/IPython/__init__.py", line 55, in <module>
    from .terminal.embed import embed
  File "/home/johannes/.local/lib/python3.7/site-packages/IPython/terminal/embed.py", line 16, in <module>
    from IPython.terminal.interactiveshell import TerminalInteractiveShell
  File "/home/johannes/.local/lib/python3.7/site-packages/IPython/terminal/interactiveshell.py", line 81, in <module>
    if not _stream or not hasattr(_stream, 'isatty') or not _stream.isatty():
ValueError: I/O operation on closed file
Error running filter test-jupyter.py:
Filter returned error status 1

Just running finalize(None) and calling that outside the panflute framework works fine. Somehow the magic inside traitlets is affected by panflute, but I can't fathom how.

Any help is much appreciated. Regards, Johan

[jhidding]

I found out what triggers the error: stdout was closed when the kernel tried to start. I'll rename the issue.

[jhidding]

Example to trigger:

#!/usr/bin/env python

import jupyter_client
import sys

def main():
    sys.stdout.close()
    with jupyter_client.run_kernel(kernel_name="python3") as kc:
        print(kc.is_alive(), file=sys.stderr)

if __name__ == "__main__":
    main()

[jacobwhall]

I am having the same issue with standard input closed. Here is an example that throws the same ValueError as above:

import sys
from jupyter_client.manager import start_new_kernel

sys.stdin.close()
start_new_kernel()

This issue appears to stem from IPython. This code runs when we call start_new_kernel, which calls isatty() on stdin, stdout, and stderr. Calling isatty() on a closed stdin throws the same ValueError:

import sys

sys.stdin.close()
sys.stdin.isatty()

I wish isatty() would return False if the stream is closed. It does not, and so I think IPython should only call isatty() if the stream is open. We should probably file an issue on that repository.

Until this is resolved, I have a workaround:

setattr(sys.stdin, "isatty", lambda: False)

[jacobwhall]

@jhidding, I (hopefully!) solved this issue with ipython/ipython#13701. The fix should be released in an upcoming version of IPython.