Homebrew no longer building Python with GNU Readline for versions >= 3.11

[drgillis]

Homebrew has decided to stop building Python with GNU Readline, starting with version 3.11. An update should be made to reflect this change in the section informing users about whether they need this package.

This thread on homebrew-core has a bit more information on their decision.

[ludwigschwardt]

Interesting… Thanks very much for the heads-up! I’ll have to do some testing soon.

[keeely]

My goodness the readline landscape has gotten significantly more complicated thanks to the Brew change.

In source we now need:

try:
    import gnureadline as readline
except ImportError:
    import readline

In setup.py going forward we're going to need something like:

    install_requires=[
        "pyreadline3; os_name=='nt'",
        "gnureadline; sys_platform=='darwin' and python_version > '3.10.0'",
    ],

I don't think Homebrew should care about including GNU code and don't understand the reason behind this change, but whatever.

[ludwigschwardt]

Ugh, thanks for the reminder 😂

Yes, Homebrew has a way of just forging ahead and breaking things. I think it's better than back in the days when they juggled Python 2 and 3 at the same time, but sometimes I'm not sure...

I'll hopefully get some time for this at the beginning of next month.

[keeely]

Just to add, my hacks above don't actually help much. You need to do more to 'fix' things properly and completely remove use of libedit from your Python instance. cmd.py imports readline inside methods, so it's not a simple monkey-patch. I've hooked module import to make cmd.py use gnureadline. I'm sure the code could be improved as I'm not an expert on import hooking, however this recipe is working for me:

import sys
import importlib

class MetaPathLoader:
    def load_module(self, fullname):
        if fullname in sys.modules:
            return sys.modules[fullname]

        gnu = importlib.import_module("gnureadline")
        sys.modules["readline"] = gnu
        return gnu

class MetaPathFinder:
    def find_module(self, fullname, path=None):
        if fullname is "readline":
            return MetaPathLoader()


sys.meta_path.insert(0, MetaPathFinder())

import cmd

class MyShell(cmd.Cmd):
    def do_command(self, _arg):
        print(f"running {self}")

MyShell().cmdloop()

I hope that people find this useful. I've also raised a Python ticket because I think cmd.py should work properly with either readline out of the box. https://github.com/python/cpython/issues/102130

[keeely]

Here's another way of doing it:

DARWIN_LIB_EDIT_VERSION = (3, 11, 0)
CURRENT_VERSION = (sys.version_info.major, sys.version_info.minor, sys.version_info.micro)

import importlib
if sys.platform == 'darwin':
    if CURRENT_VERSION >= DARWIN_LIB_EDIT_VERSION:
        sys.modules["readline"] = importlib.import_module("gnureadline")

[ludwigschwardt]

We tried tricks like that in the past (see 3c115ae) but this has its own can of worms...

The safest approach is to import gnureadline explicitly and use it directly.

[ludwigschwardt]

Hi Dan and @keeely, I've finally come round to this issue... I'm hoping to thrash out a good replacement for easy_install readline to go in the documentation.

Have you tried usercustomize.py as described in #62? This looks like a good place to put the sys.modules['readline'] switcheroo before rlcompleter and friends notice...

[onlynone]

Would you consider removing or modifying the line in the readme about not needing this library if

you use the Python provided by Homebrew or Fink on macOS (It has real readline already!)

I noticed my ctrl-r history search was no longer working in the python shell and then found that the python provided by homebrew had switched to libedit. I remembered the ability to install gnureadline from back when I would use the system python in macos, but the page on pypi confused me for a bit since it seemed to indicate that I shouldn't need it if I installed from homebrew.

[keeely]

I would scrap the entire section about why you wouldn't want to use it, and instead just point out that you're not going to need this library if you're already using gnu readline then explain how to find out if you are, e.g.

python3 -c "import readline;print(readline.__doc__)"
Importing this module enables command line editing using libedit readline.

[ludwigschwardt]

Good points, @onlynone and @keeely. Some folks were stuck on older Pythons and/or GNU readlines and wanted to replace their buggy gnureadline modules with an up-to-date one, but that's admittedly a small subset of a small subset... (and pyenv helps a lot these days).

I was spurred on to make a new release before GitHub's Python 2.7 support ran out on June 19 (actions/setup-python#672) and was making good progress until life intervened 😅 See the override-readline-via-site-customization branch for more details.

Thanks for the reminder - I'll incorporate your suggestions and wrap this up!

[ludwigschwardt]

I've finally released gnureadline 8.2.10 to PyPI. Thanks again for your patience 😅