guysv
Tab-completion crashes
Tab-completions crash ILua on Ubuntu 20.04. It makes no difference whether I install it via Pip or from sources. To minimize possible interactions with installed Pip-packages (or even apt-installed packages), I tried to install ILua into its own virtual environment.
rhaberkorn@thinkpad-x270:~/working-copies/ilua$ python3 -m venv env
rhaberkorn@thinkpad-x270:~/working-copies/ilua$ . env/bin/activate
(env) rhaberkorn@thinkpad-x270:~/working-copies/ilua$ python3 -m pip install -e .
Obtaining file:///home/rhaberkorn/working-copies/ilua
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done
Collecting twisted
Using cached twisted-23.8.0-py3-none-any.whl (3.1 MB)
Collecting pygments
Using cached Pygments-2.16.1-py3-none-any.whl (1.2 MB)
Collecting termcolor
Using cached termcolor-2.3.0-py3-none-any.whl (6.9 kB)
Collecting txzmq
Using cached txZMQ-1.0.0-py3-none-any.whl (24 kB)
Collecting jupyter-console
Using cached jupyter_console-6.6.3-py3-none-any.whl (24 kB)
Collecting jupyter-core
Using cached jupyter_core-5.4.0-py3-none-any.whl (28 kB)
Collecting attrs>=21.3.0
Using cached attrs-23.1.0-py3-none-any.whl (61 kB)
Collecting automat>=0.8.0
Using cached Automat-22.10.0-py2.py3-none-any.whl (26 kB)
Collecting typing-extensions>=3.10.0
Using cached typing_extensions-4.8.0-py3-none-any.whl (31 kB)
Collecting zope-interface>=5
Using cached zope.interface-6.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (249 kB)
Collecting incremental>=22.10.0
Using cached incremental-22.10.0-py2.py3-none-any.whl (16 kB)
Collecting constantly>=15.1
Using cached constantly-15.1.0-py2.py3-none-any.whl (7.9 kB)
Collecting hyperlink>=17.1.1
Using cached hyperlink-21.0.0-py2.py3-none-any.whl (74 kB)
Collecting pyzmq>=13
Using cached pyzmq-25.1.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)
Collecting prompt-toolkit>=3.0.30
Using cached prompt_toolkit-3.0.39-py3-none-any.whl (385 kB)
Collecting jupyter-client>=7.0.0
Using cached jupyter_client-8.4.0-py3-none-any.whl (103 kB)
Collecting traitlets>=5.4
Using cached traitlets-5.11.2-py3-none-any.whl (83 kB)
Collecting ipykernel>=6.14
Using cached ipykernel-6.25.2-py3-none-any.whl (154 kB)
Collecting ipython
Using cached ipython-8.12.3-py3-none-any.whl (798 kB)
Collecting platformdirs>=2.5
Using cached platformdirs-3.11.0-py3-none-any.whl (17 kB)
Collecting six
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Requirement already satisfied: setuptools in ./env/lib/python3.8/site-packages (from zope-interface>=5->twisted->ilua==0.2.2.dev18+g05eb181) (44.0.0)
Collecting idna>=2.5
Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting wcwidth
Using cached wcwidth-0.2.8-py2.py3-none-any.whl (31 kB)
Collecting python-dateutil>=2.8.2
Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting tornado>=6.2
Using cached tornado-6.3.3-cp38-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (427 kB)
Collecting importlib-metadata>=4.8.3; python_version < "3.10"
Using cached importlib_metadata-6.8.0-py3-none-any.whl (22 kB)
Collecting debugpy>=1.6.5
Using cached debugpy-1.8.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5 MB)
Collecting nest-asyncio
Using cached nest_asyncio-1.5.8-py3-none-any.whl (5.3 kB)
Collecting matplotlib-inline>=0.1
Using cached matplotlib_inline-0.1.6-py3-none-any.whl (9.4 kB)
Collecting psutil
Using cached psutil-5.9.6-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (283 kB)
Collecting comm>=0.1.1
Using cached comm-0.1.4-py3-none-any.whl (6.6 kB)
Collecting packaging
Using cached packaging-23.2-py3-none-any.whl (53 kB)
Collecting pexpect>4.3; sys_platform != "win32"
Using cached pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
Collecting stack-data
Using cached stack_data-0.6.3-py3-none-any.whl (24 kB)
Collecting jedi>=0.16
Using cached jedi-0.19.1-py2.py3-none-any.whl (1.6 MB)
Collecting pickleshare
Using cached pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB)
Collecting decorator
Using cached decorator-5.1.1-py3-none-any.whl (9.1 kB)
Collecting backcall
Using cached backcall-0.2.0-py2.py3-none-any.whl (11 kB)
Collecting zipp>=0.5
Using cached zipp-3.17.0-py3-none-any.whl (7.4 kB)
Collecting ptyprocess>=0.5
Using cached ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Collecting asttokens>=2.1.0
Using cached asttokens-2.4.0-py2.py3-none-any.whl (27 kB)
Collecting pure-eval
Using cached pure_eval-0.2.2-py3-none-any.whl (11 kB)
Collecting executing>=1.2.0
Using cached executing-2.0.0-py2.py3-none-any.whl (24 kB)
Collecting parso<0.9.0,>=0.8.3
Using cached parso-0.8.3-py2.py3-none-any.whl (100 kB)
Installing collected packages: attrs, six, automat, typing-extensions, zope-interface, incremental, constantly, idna, hyperlink, twisted, pygments, termcolor, pyzmq, txzmq, wcwidth, prompt-toolkit, python-dateutil, traitlets, tornado, platformdirs, jupyter-core, zipp, importlib-metadata, jupyter-client, debugpy, nest-asyncio, matplotlib-inline, ptyprocess, pexpect, asttokens, pure-eval, executing, stack-data, parso, jedi, pickleshare, decorator, backcall, ipython, psutil, comm, packaging, ipykernel, jupyter-console, ilua
Running setup.py develop for ilua
Successfully installed asttokens-2.4.0 attrs-23.1.0 automat-22.10.0 backcall-0.2.0 comm-0.1.4 constantly-15.1.0 debugpy-1.8.0 decorator-5.1.1 executing-2.0.0 hyperlink-21.0.0 idna-3.4 ilua importlib-metadata-6.8.0 incremental-22.10.0 ipykernel-6.25.2 ipython-8.12.3 jedi-0.19.1 jupyter-client-8.4.0 jupyter-console-6.6.3 jupyter-core-5.4.0 matplotlib-inline-0.1.6 nest-asyncio-1.5.8 packaging-23.2 parso-0.8.3 pexpect-4.8.0 pickleshare-0.7.5 platformdirs-3.11.0 prompt-toolkit-3.0.39 psutil-5.9.6 ptyprocess-0.7.0 pure-eval-0.2.2 pygments-2.16.1 python-dateutil-2.8.2 pyzmq-25.1.1 six-1.16.0 stack-data-0.6.3 termcolor-2.3.0 tornado-6.3.3 traitlets-5.11.2 twisted-23.8.0 txzmq-1.0.0 typing-extensions-4.8.0 wcwidth-0.2.8 zipp-3.17.0 zope-interface
(env) rhaberkorn@thinkpad-x270:~/working-copies/ilua$ python3 setup.py install_data
WARNING: The wheel package is not available.
WARNING: The wheel package is not available.
WARNING: The wheel package is not available.
/home/rhaberkorn/working-copies/ilua/.eggs/setuptools_scm-8.0.4-py3.8.egg/setuptools_scm/_integration/setuptools.py:30: RuntimeWarning:
ERROR: setuptools==44.0.0 is used in combination with setuptools_scm>=8.x
Your build configuration is incomplete and previously worked by accident!
setuptools_scm requires setuptools>=61
Suggested workaround if applicable:
- migrating from the deprecated setup_requires mechanism to pep517/518
and using a pyproject.toml to declare build dependencies
which are reliably pre-installed before running the build tools
warnings.warn(
running install_data
creating /home/rhaberkorn/working-copies/ilua/env/share/jupyter/kernels/lua
copying defaultspec/logo-64x64.png -> /home/rhaberkorn/working-copies/ilua/env/share/jupyter/kernels/lua
copying defaultspec/logo-license.txt -> /home/rhaberkorn/working-copies/ilua/env/share/jupyter/kernels/lua
copying defaultspec/logo-32x32.png -> /home/rhaberkorn/working-copies/ilua/env/share/jupyter/kernels/lua
copying defaultspec/kernel.json -> /home/rhaberkorn/working-copies/ilua/env/share/jupyter/kernels/lua
(env) rhaberkorn@thinkpad-x270:~/working-copies/ilua$ ilua
Jupyter console 6.6.3
ILua 0.2.2.dev18+g05eb181
In [1]: print2023-10-22T01:27:26+0300 [ilua.kernel.ILuaKernel#critical] Uncought exception in message handler
Traceback (most recent call last):
File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1997, in _inlineCallbacks
result = context.run(gen.send, result)
File "/home/rhaberkorn/working-copies/ilua/ilua/kernelbase.py", line 199, in handle_message
content = yield self.do_complete(**msg['content'])
File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 2245, in unwindGenerator
return _cancellableInlineCallbacks(gen)
File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 2157, in _cancellableInlineCallbacks
_inlineCallbacks(None, gen, status, _copy_context())
--- <exception caught here> ---
File "/home/rhaberkorn/working-copies/ilua/ilua/kernelbase.py", line 199, in handle_message
content = yield self.do_complete(**msg['content'])
File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1997, in _inlineCallbacks
result = context.run(gen.send, result)
File "/home/rhaberkorn/working-copies/ilua/ilua/kernel.py", line 197, in do_complete
result = yield self.proto.sendRequest({
builtins.AttributeError: 'ILuaKernel' object has no attribute 'proto'
Unhandled exception in event loop:
File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/prompt_toolkit/buffer.py", line 1919, in new_coroutine
await coroutine(*a, **kw)
File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/prompt_toolkit/buffer.py", line 1743, in async_completer
async for completion in async_generator:
File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/prompt_toolkit/completion/base.py", line 323, in get_completions_async
async for completion in completer.get_completions_async(
File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/prompt_toolkit/completion/base.py", line 199, in get_completions_async
for item in self.get_completions(document, complete_event):
File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/jupyter_console/ptshell.py", line 150, in get_completions
content = self.jup_completer.complete_request(
File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/jupyter_console/completer.py", line 37, in complete_request
msg = run_sync(self.client.shell_channel.get_msg)(timeout=self.timeout)
File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/jupyter_client/channels.py", line 233, in get_msg
raise Empty
Exception
Press ENTER to continue...
In [1]:
Do you really want to exit ([y]/n)? y
Shutting down kernel
(env) rhaberkorn@thinkpad-x270:~/working-copies/ilua$ python3 --version
Python 3.8.10
The Python is quite old as you see, but Ubuntu 20.04 is still LTS. It also makes no difference which Lua interpreter I specify via -i.
Trying to execute 23:
In [1]: 232023-10-22T01:51:42+0300 [ilua.kernel.ILuaKernel#critical] Uncought exception in message handler
Traceback (most recent call last):
File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1997, in _inlineCallbacks
result = context.run(gen.send, result)
File "/home/rhaberkorn/working-copies/ilua/ilua/kernelbase.py", line 196, in handle_message
content = yield self.do_is_complete(**msg['content'])
File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 2245, in unwindGenerator
return _cancellableInlineCallbacks(gen)
File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 2157, in _cancellableInlineCallbacks
_inlineCallbacks(None, gen, status, _copy_context())
--- <exception caught here> ---
File "/home/rhaberkorn/working-copies/ilua/ilua/kernelbase.py", line 196, in handle_message
content = yield self.do_is_complete(**msg['content'])
File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1997, in _inlineCallbacks
result = context.run(gen.send, result)
File "/home/rhaberkorn/working-copies/ilua/ilua/kernel.py", line 184, in do_is_complete
result = yield self.proto.sendRequest({"type": "is_complete",
builtins.AttributeError: 'ILuaKernel' object has no attribute 'proto'
/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/jupyter_console/ptshell.py:787: UserWarning: The kernel did not respond to an is_complete_request. Setting `use_kernel_is_complete` to False.
warn('The kernel did not respond to an is_complete_request. '
In [1]: 23
2023-10-22T01:51:43+0300 [ilua.kernel.ILuaKernel#critical] Uncought exception in message handler
Traceback (most recent call last):
File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1997, in _inlineCallbacks
result = context.run(gen.send, result)
File "/home/rhaberkorn/working-copies/ilua/ilua/kernelbase.py", line 193, in handle_message
content = yield self.do_execute(**msg['content'])
File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 2245, in unwindGenerator
return _cancellableInlineCallbacks(gen)
File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 2157, in _cancellableInlineCallbacks
_inlineCallbacks(None, gen, status, _copy_context())
--- <exception caught here> ---
File "/home/rhaberkorn/working-copies/ilua/ilua/kernelbase.py", line 193, in handle_message
content = yield self.do_execute(**msg['content'])
File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1997, in _inlineCallbacks
result = context.run(gen.send, result)
File "/home/rhaberkorn/working-copies/ilua/ilua/kernel.py", line 131, in do_execute
result = yield self.proto.sendRequest({"type": "execute",
builtins.AttributeError: 'ILuaKernel' object has no attribute 'proto'
@guysv What's the recommended environment to run this in? Perhaps I can try with Docker. However it won't be an option for me to run the Lua interpreter within Docker as well.
Also, will it be possible to use the web frontend as well?
Also, will it be possible to use the web frontend as well?
Just to explain this. I have a project based on a custom LuaJIT REPL loop and I am planning to add context sensitive auto-completions anyway. That's not really what I need ILua for. What really interests me is a way to cheaply add a Web interface. People nowadays are spoiled you know. In this context, it would also be interesting to find out whether it will be possible to render any markup or graphics.
EDIT: Should probably tag you, since this was initially addressing your comment @rhaberkorn
I'm running into the same problem, except I can't evaluate anything before it crashes.
I was/still am digging through the code, and traced the problem back to breaking changes in twisted, with the latest working version being v22.10.0, 23+ breaks ILua. I never used twisted, nor have I created a Jupyter kernel before, or interacted with its libraries, but I'm currently spending some time trying to make ILua work with twisted 23+
Until then, the only way to make ILua run is to use a version less than 23. Since Twisted may be used by various other Python packages on the system, globally downgrading it just for ILua is not a good idea.
Instead, if you just want to get it working for the moment, here are some overly methodical instructions for people who come across this issue:
# Pick somewhere to put it. I just placed it next to the repo for debugging purposes.
cd ~/Source/third-party/clones/ilua/
# Create and enter the virtual environment.
python -m venv vilua
cd vilua
# Activate the virtual environment by sourcing whichever activation script you nee to source.
source ./bin/activate.fish
# Make sure it's activated by checking if python and or pip points to the virtual environment, not the system.
which pip
# Specify any version < 23 for twisted. 22.10.0 is the latest compatible.
pip install twisted==22.10.0 ilua
# Success?
ilua
Maybe a script to run it from anywhere conveniently:
#!/bin/sh
source ~/Source/third-party/ilua/vilua/bin/activate
ilua
deactivate
I placed it next to ILua
mv vilua ~/.local/bin/vilua
sudo chmod ug+x ~/.local/bin/vilua
This isn't a permanent solution, nor a complete one.
I think the dependency stems from ILua using it to re-implement existing parts of the ipython / jupyter libraries to work with Twisted, in order to have Python 2 async support?
"""
Twisted-based Jupyter base kernel implementation
Pretty much a reimplementation of ipykernel.kernelbase,
but with Twisted, which allows async code in python 2,
and provides txzmq.
"""
Since Python2 is deprecated, and this re-write would need to be maintained long-term, I think the best solution would be to drop twisted entirely. It seems to be more trouble than it's worth, having broken ILua with v23 here, as well as unnecessarily broadening its attack surface (see CVE-2023-46137), emphasizing the importance of maintenance. In this case the CVE had nothing to do with the way ILua uses Twisted, but why even have an "in this case" and not just drop it entirely and not be forced to maintain it?
That's the ideal anyway, but the fastest fix would be to resolve the breaking changes from v22 to v23 of twisted, and write the v23 equivalent of the v22 code, which is what I'm investigating. That being said I don't have an awful lot of time, but I felt like I should at least share what I learned so far.
@PsychedelicShayna Will try that. Thanks! Using virtual environments is IMHO not hacky at all. Library conflicts are almost unavoidable in real life. Somebody somewhere is not declaring the Twister-dependency on versions < 23 properly.
I can confirm that everything works for me once forcing Twister to the right version.
I can even run my custom Lua interpreter in a console - although, that's not terribly useful as I wrote.
It is indeed possible to run ILua in a notebook (Web UI) as well and it will even work with custom Lua interpreters. First you will have to install jupyter-notebook (into the same environment as everything else of course):
pip3 install notebook
Now just running jupyter notebook will start a web server and when creating a notebook from within the website you can manually choose a Lua kernel. It's probably also possible to make this the default choice, but I haven't yet figured out how that works.
Unfortunately, there does not seem to be a way to pass arguments to selected kernels via the jupyter notebook command line. That would be useful in order to set ILua's --lua-interpreter. If you know how, just tell me. It might be possible to just shadow lua in PATH of course and that might even be the best option if you need to pass any custom arguments to your interpreter as ILua does not support passing anything down to the interpreter so passing custom arguments requires a wrapper script anyway. Otherwise, you might just create a new kernel by copying ilua/env/share/jupyter/kernels/lua/ to some new directory, let's say ilua/env/share/jupyter/kernels/luajit/ and edit its kernel.json file. For instance to create an ILua-based kernel using luajit as the interpreter:
{
"argv":
[
"python",
"-m",
"ilua.app",
"-c",
"{connection_file}",
"--lua-interpreter", "luajit"
],
"display_name": "LuaJIT",
"language": "lua",
"interrupt_mode": "message"
}
What I haven't found out yet is:
- How to interrupt a busy ILua kernel from a running notebook. At least, the interrupt button does not appear to send SIGINT to the corresponding Lua interpreter process.
- How to display some rich text or graphics in command output. From reading the IPython source code I get the impression that this must be supported by the kernel, ie. ILua in our case.
All of this should really be documented in some official place.
thanks for the heads up for the twisted version. working on a release pinning the version. I definitely can't keep up with latest twisted for this project so virtual envs are the way to go.
