Improve extension discovery using custom setuptools entry points

[arthurlm]

Problem description

It would be great if smart_open could be extended using setuptools / entry_points capabilities. For now, if user install a package that provide new smart_open transport mechanism, it cannot be automatically registered.

Example use case

Please note: schema is not relevant here.

In package smart_ext_transport I have following file:

# smart_ext_transport/custom_transport.py
SCHEMA = "custom"
...

Current situation

The only way to load extension for now are listed bellow:

Option 1: using register_transport everywhere and every time it might be required :cry:

register_transport("smart_ext_transport.custom_transport")

Option 2: using __init__.py and import statement:

Adding:

# smart_ext_transport/__init__.py
register_transport("smart_ext_transport.custom_transport")

Then every time custom schema might be needed:

import smart_ext_transport

Improved situation with setuptools

In setup.py from smart_ext_transport we may add:

from setuptools import setup

setup(
    name="smart_ext_transport",
    packages=["smart_ext_transport"],
    entry_points={"smart_open_transport": [
        "custom = smart_ext_transport.custom_transport"],
    },
)

Once package is installed, smart_open can automatically load every plugin that is registered using setuptools entry points. User will no longer have to import package to register extension manually.

This is how pytest / setuptools CLI / flake8 / jupyter nb convert works for example.

Possible implementation

Feature can be easily implemented.

It would require adding following lines between: https://github.com/RaRe-Technologies/smart_open/blob/84e9aac271653e029d68b090b02c545e7b43efcb/smart_open/transport.py#L103-L105

from importlib import metadata

for ep in metadata.entry_points()['smart_open_transport']:
    try:
        register_transport(ep.value)
    except Exception:
        logger.warning("Fail to load smart_open transport extension: %s", ep.name)

I can provide PR for this if needed :grin:.

Versions

Linux-4.15.0-175-generic-x86_64-with-glibc2.27
Python 3.8.0 (default, Dec  9 2021, 17:53:27) 
[GCC 8.4.0]
smart_open 5.2.1