cpython
cpython copied to clipboard
python
A new Python REPL
This issue will track all the different steps to bootstrap a new Python REPL with many new features. The target of this issue is that we can reach a point where new contributions can be made easily to the REPL once is written in pure Python. This issue only coveres the initial ground work to reach this status.
Tasks to be done:
- [x] Redirect the parser and the main module to a new Python module.
- [x] Fallback to the previous tokenizer-based REPL if the terminal is not a tty (and for backwards compat reasons)
- [x] Bootstrap a new REPL module (based on a trimmed version pypy's pyrepl module).
- [x] Ensure that the basic features work:
- [x] History
- [x] Completions
- [x] Ctrl-r and Ctrl-s
Linked PRs
- gh-111567
- gh-118616
- gh-118635
- gh-118638
- gh-118653
- gh-118700
- gh-118705
- gh-118712
- gh-119262
- gh-119274
If you want to help with this endeavour, please mention it here so we can coordinate everyone!
I have a initial version that we can start building upon, I will create a PR shortly.
I learned about this from the "core.py" podcast and it sounds like a great idea! I'd be happy help with this effort :)
Fallback to the previous tokenizer-based REPL if the terminal is not a tty (and for backwards compat reasons)
Do you mean not using _pyrepl? Can _pyrepl be used without readline nor tty?
Can _pyrepl be used without readline nor tty?
It needs at least a tty yes. Readline can be lifted but I am not sure yet
I would like to see a new REPL (read, evaluate, print, loop) for the Python interpreter if the interpreter goes by a different name. Two other major Python interpreters also exist, and they already use other names ("bpython" and "ipython"). I would recommend forking the current REPL and renaming the interpreter with the new REPL to "zpython" instead. Also, I would like to see a built-in "clear()" function in the REPL as I already encounter with the "exit()" function and other REPL-specific functions.
Most buildbots are broken with:
Traceback (most recent call last):
File ".../Lib/unittest/mock.py", line 1420, in patched
return func(*newargs, **newkeywargs)
File ".../Lib/test/test_pyrepl.py", line 643, in test_push_without_key_in_keymap
eq = EventQueue(sys.stdout.fileno(), "utf-8")
~~~~~~~~~~~~~~~~~^^
io.UnsupportedOperation: fileno
e.g.: https://buildbot.python.org/all/#/builders/725/builds/7939/steps/5/logs/stdio
Hi, I am seeing the following error:
======================================================================
FAIL: test_completion_with_many_options (test.test_pyrepl.TestPyReplCompleter.test_completion_with_many_options)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/..../cpython-main/Lib/test/test_pyrepl.py", line 586, in test_completion_with_many_options
self.assertEqual(output, "os.O_ASYNC")
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'os.O_AS' != 'os.O_ASYNC'
- os.O_AS
+ os.O_ASYNC
? +++
The reason is most likely that os.O_ASYNC is not necessarily available on all platforms (I am seeing this on Solaris where it's missing). I tried replacing it with O_AP and O_APPEND and then it works as expected.
(hmm there is only one constant starting with os.O_AS, so not sure where the "with_many_options" comes in. should it just be testing os.O_? if anything test_simple_completion tests multiple options because of getenv and getenvb)
(hmm there is only one constant starting with
os.O_AS, so not sure where the "with_many_options" comes in. should it just be testingos.O_? if anythingtest_simple_completiontests multiple options because of getenv and getenvb)
Yes, but the "multiple options" comes from the fact that the test first checks what happens if you type os. and you press tab twice (which displays all options) and then you complete from one. Notice the sequence of characters is:
os.\t\tO_AP\t\n
- I think history is only separate when using
helpstatement, not when callinghelp() - When you use the
helpstatement (but not when callinghelp()), there is an extra>>>when you exit - Previously it was common for users to encounter "Type help(object) for help about object" message. I wonder if we should now mention this in the main
helphelp - Paste mode is awesome. Would be even more awesome if we can have it mostly work all the time, like IPython
- When you use the
helpstatement (but not when callinghelp()), there is an extra>>>when you exit
Exiting history mode (hitting F2 and then q) also shows an extra >>>.
- Paste mode is awesome. Would be even more awesome if we can have it mostly work all the time, like IPython
IPython relies on prompt toolkit which relies on bracketed paste mode. Issue #84001 is about that.
Bracketed paste mode and Windows support (which is a much bigger ask than bracket paste mode I assume) would likely fully replace my tendency to explain pip and venv early for the sake of nudging brand new Python programmers toward IPython.
I am so excited to use this new REPL while teaching after 3.13's release. 👏
One behaviour I was pretty confused by right now is the "recursive" help mode. If you press F1 a bunch of times, it will start nested helps and each of them needs an exit/ctrl-d to leave. Is that intentional?
Bracketed paste coming right up: https://github.com/python/cpython/pull/118700
Pressing Ctrl-R and then an arrow key trips an assertion for me. Logged as #118682, mentioning here for visibility.
Another issue, unfortunately caused by bracketed paste mode: pasting text is now about 1,000 times slower (by my estimate). Pasting the 441,033 character text of Frankenstein takes a couple seconds in the old REPL, but pasting just the first 7,628 characters took 39 seconds in the new REPL.
Hitting Ctrl+C while pasting also doesn't stop the pasting, so accidentally pasting a very large block is an unfortunate mistake to make currently.
With the old REPL on macOS, sys.prefix.replace("\\", "\\\\") prints out a string. But the new one doesn't:
❯ PYTHON_BASIC_REPL=1 ./python.exe
Python 3.13.0a6+ (heads/main:fcf52d7cee, May 8 2024, 09:40:54) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/usr/local'
>>> sys.prefix.replace("\\", "\\\\")
'/usr/local'
>>>
main on main [$?⇡] via C v15.0.0-clang via 🐍 v3.12.3 via 💎 v3.1.3 took 36s
❯ ./python.exe
Python 3.13.0a6+ (heads/main:fcf52d7cee, May 8 2024, 09:40:54) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/usr/local'
>>> sys.prefix.replace("\\", "\\\\")
>>>
With the old REPL on macOS,
sys.prefix.replace("\\", "\\\\")prints out a string. But the new one doesn't:❯ PYTHON_BASIC_REPL=1 ./python.exe Python 3.13.0a6+ (heads/main:fcf52d7cee, May 8 2024, 09:40:54) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.prefix '/usr/local' >>> sys.prefix.replace("\\", "\\\\") '/usr/local' >>>main on main [$?⇡] via C v15.0.0-clang via 🐍 v3.12.3 via 💎 v3.1.3 took 36s ❯ ./python.exe Python 3.13.0a6+ (heads/main:fcf52d7cee, May 8 2024, 09:40:54) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.prefix '/usr/local' >>> sys.prefix.replace("\\", "\\\\") >>>
Hummm, are you using the latest main? It works for me:
main on main [$?⇕] took 49s
❯ ./python.exe
Python 3.13.0a6+ (heads/main:c4f9823be27, May 8 2024, 10:02:25) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/usr/local'
>>> sys.prefix.replace("\\", "\\\\")
'/usr/local'
Oh, by any chance have you pasted the code into the REPL? Paste mode uses exec in some occasions so it may have a bug when it does not print the result
Yes, I was on latest main and I did paste it in.
Yep, confirmed it reproduces when pasting, and works fine when typing or selecting from history.
Since _pyrepl.__main__ is top-level now, its imports/definitions are exposed to user. Should we hide them?
Python 3.13.0a6+ (heads/main:c4f9823be2, May 8 2024, 21:42:56) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> dir()
['CAN_USE_PYREPL', '__annotations__', '__builtins__', '__cached__', '__doc__', '__file__',
'__loader__', '__name__', '__package__', '__spec__', 'interactive_console', 'os', 'sys']
>>>
When editing the last line in a function in the new REPL, hitting Enter doesn't end the block if the last line has indentation on it.
Demo of the bug, demo of what IPython does
When editing the last line in a function in the new REPL, hitting Enter doesn't end the block if the last line has indentation on it.
We will look into it but that's by design I think. The block won't end unless you are in the last line at the start and you press enter. Otherwise it will think you are editing the block adding new lines
Now python and python -m asyncio works differently.
I want to give it a shot and fix it :)
Hm, right now I don't see any simple way of merging these two together: https://github.com/python/cpython/blob/c68acb1384a51eb745f572687eaf677371b9e765/Lib/asyncio/main.py#L15-L21
and
https://github.com/python/cpython/blob/c68acb1384a51eb745f572687eaf677371b9e765/Lib/_pyrepl/simple_interact.py#L81-L83
I think that this would require some API change. Not sure which one, though :( I would like pass it to someone else.
We've discovered that if you manually type out a condition that evaluates to False in the new REPL, False is printed as the result, as expected...
>>> (3, 13, 0, "final") < (3, 13, 0, "beta")
False
...but that if you copy and paste this condition into the REPL, nothing is printed (implying that the condition evaluates to None:
>>> (3, 13, 0, "final") < (3, 13, 0, "beta")
>>>
Thanks @Eclips4 for realising that the difference in behaviour here was due to copying-and-pasting the condition rather than typing it out!
