Support external tools and flows

[olofk]

The new Flow API allows for more esoteric flows and tools. Some of these will not make sense to have in the main Edalize code base but might be distributed with other projects or sometimes proprietary. To avoid a ton of Edalize forks just to add new tools and flows it would be great to support externally defined modules.

Implicit Namespace Packages seems to be the easiest way to do this. The only problem is that it requires removing all __init__.py files and Edalize currently exposes the functions get_edatools, get_edatool and get_flow through the main __init__.py function. Can we a) do something clever that allows us to keep the main __init__.py and still support namespaces or b) Move the functions to some other file and still keep the API? If not, we need to form a migration strategy where I suggest we mark them as deprecated and provide alternative functions (in the next 0.3.4 release?) and eventually drop the old ones completely (in 0.4?).

[umarcor]

FTR, pytest uses a mechanism to allow "plugins to be installable by others" (https://docs.pytest.org/en/6.2.x/writing_plugins.html#making-your-plugin-installable-by-others) which is based on providing an specific entry_point in the setup:

If you want to make your plugin externally available, you may define a so-called entry point for your distribution so that pytest finds your plugin module. Entry points are a feature that is provided by setuptools. pytest looks up the pytest11 entrypoint to discover its plugins and you can thus make your plugin available by defining it in your setuptools-invocation:

# sample ./setup.py file
from setuptools import setup

setup(
    name="myproject",
    packages=["myproject"],
    # the following makes a plugin available to pytest
    entry_points={"pytest11": ["name_of_plugin = myproject.pluginmodule"]},
    # custom PyPI classifier for pytest plugins
    classifiers=["Framework :: Pytest"],
)

[olofk]

This was fixed in 0.5.0

[shareefj]

Edit: don't read this, see the next comment.

@olofk I've just been looking at how this works and thought I'd ask questions here so that at least it is semi-documented.

It looks like the way you've implemented it means that any "plugins" have to mirror the edalize/ directory structure and you need to add the parent directory to the PYTHONPATH in order for them to be picked up. At least that seems to work for my testing. Was that your intent?

So if I wanted to add custom plugins to my working tree under a util/ directory, I'd have the following tree:

$ tree util/edalize_plugins
util/edalize_plugins
└── edalize
    ├── flows
    ├── testplugin.py
    └── tools

3 directories, 1 file

This works for this sort of directory structure but doesn't work, for example, if users want to keep their plugins in a separate package and pip install them. Or at least I couldn't find a way of pip installing a package and then adding a subdirectory to PYTHONPATH. Is there a way to do this with the current implementation that you can think of?

Alternatively, can we add support for proper plugins by defining a naming convention to use? Either use entry points in the same way as pytest/pluggy, or do something simple like declare that plugins must be packages named edalize_* and then use pkgutil or whatever in a similar loop.

[shareefj]

Okay, I realise I've completely misunderstood how this works. In browsing the Git log I found your commit that mentions PEP420 which got me reading...

So the way to create plugins is via namespace packages and the only practical change for users is to not include an __init__.py file in their package.

So taking your own tests/test_plugin directory and putting it into its own package would look like:

$ tree
.
├── pyproject.toml
└── src
    └── edalize
        ├── flows
        │   └── customexternalflow.py
        ├── testplugin.py
        └── tools
            └── customtool.py

4 directories, 4 files

and the pyproject.toml would look something like:

$ cat pyproject.toml
[project]
name = "edalize_plugins"
description = "A selection of plugins for Edalize"
version = "0.1.0"

[build-system]
requires = [
    "setuptools",
    "wheel",
]
build-backend = "setuptools.build_meta"

[tool.setuptools.packages.find]
where = ["src"]

[olofk]

@shareefj That's a good example. I haven't gotten around to learn how the pyproject.yaml files work