[Proposal] Plugin mechanism for ELK

[heming6666]

Why we need it?

There are more and more proposals about adding support for some backends that are not covered by Grimoirelab:

• Add Gitee backend support to GrimoireLab ELK: #835
• Implement Zulip backend to Grimoirelab ELK: #842
• Add rocketchat backend support to ELK: #876

and so on.

For Perceval, some discussions were raised whether these backends should be integrated to the core of perceval:

• Add Airtable backend: https://github.com/chaoss/grimoirelab-perceval/pull/640
• Add support for Trello: https://github.com/chaoss/grimoirelab-perceval/issues/664

And currently we can have separate repositories for these Perceval-related backends, e.g. chaoss/grimoirelab-perceval-mozilla and WillemJiang/grimoirelab-perceval-gitee. And they can be also imported by ELK in this way:

from perceval.backends.core.git import Git, GitCommand
from perceval.backends.mozilla.crates import Crates, CratesCommand
from perceval.backends.opnfv.functest import Functest, FunctestCommand

However, for grimoirelab-elk, we currently don't provide such a mechanism to allow people to implement their beckends in a separate repository( grimoirelab-elk-xxx), since all of the connector meta data should be decleared in grimoire_elk/utils.py.

If we can support plugin mechanism for ELK, then people can integrate the third party backends with the latest core easily.

How it could be?

Here is the demo repository I write for testing plugin mechanism of python.

In that case, grimoirelab-elk-xxx would act as a plugin and can be loaded or used by the core of ELK:

• For the plugins, they need to export the connectors (e.g. Gitee, GiteeOcean, GiteeEnrich and GiteeCommand) and use entry_points to register themselves for discovery.
• For the core of ELK, it only needs to discover the registered plugins automatically and then use __import__() to load the specific modules to import the connectors. And it doesn't need to know the plugins in advance.

ELK can select a suitable solution to implement as its foundation for users to implement their own plugins conveniently.

Furthermore, we can link those repositories in our README file as third-party backends. Or create a individual repository for users to submit their plugin implementation which cannot be merged into the main project.

I am looking forward to hearing the opinions from all of you. Thanks!

[valeriocos]

Hi @heming6666 ,

Thank you for opening this issue and showcasing your proposal with the demo repository. I really like the idea and appreciate your effort to present it.

We can use the Gitee connectors as a pilot study, and shape the process to share the plugins, WDYT?

WRT the modification on ELK, why we cannot use namespace_packages as done in Perceval? I'm asking this because it would be convenient to have the same solution implemented for both components (if possible).

In the demo repo I see that you use the prefix plugin (https://github.com/heming6666/python-plugins-test/blob/master/plugin-gitee/setup.py#L6), which is then used in the packages. I understand if we use the settings below, it should be OK to discover the gitee packages, right?

• name="grimoirelab-elk-gitee"
• the same packages used in ELK

Furthermore, we can link those repositories in our README file as third-party backends. Or create a individual repository for users to submit their plugin implementation which cannot be merged into the main project.

We could start with linking the repos to the README (to be consistent with the discussions around the Perceval backends). Later, depending on the number of the third-party backends we can create an individual repository, WDTY?

Thank you

[heming6666]

Hi @valeriocos , sorry for the late reply.

We can use the Gitee connectors as a pilot study, and shape the process to share the plugins, WDYT?

Yes. I would be glad to implement the Gitee backend and will update the process in #835 .

WRT the modification on ELK, why we cannot use namespace_packages as done in Perceval?

That's a good idea and I have updated the demo repository, which shows namespace_packages work well in this case.

However, I have two questions here:

• The first one is that when users want to use third-party plugins, they need to modify grimoire_elk/utils.py to import the needed connectors. For example, if I want to use the Gitee plugin and the plugin packages are perceval-gitee and grimoire-elk-gitee, then I need to add the following code:

from perceval.backends.gitee.gitee import Gitee, GiteeCommand
from .raw.gitee import GiteeOcean
from .enriched.gitee import GiteeEnrich

def get_connectors():
    return {
            ...
            "gitee": [Gitee, GiteeOcean, GiteeEnrich, GiteeCommand],
            ...
            }

Also I need to add the requirements in requirements.txt.

I think it's acceptable for users to do this, right? I am asking this because if we use entry_points, the only thing users need to do is adding the requirements in requirements.txt.

• The second one is taht this guide says that pkg_resources-style namespace packages is no longer recommended, it is widely present in most existing namespace packages. I am not sure if it would be better to use Native namespace packages or pkgutil-style namespace packages.

[sduenas]

I would like to explore the idea of using entry points. There was a similar comment about using it in Perceval a few months ago. What I'm not sure is if this is the best way to use them.

[valeriocos]

Hi @heming6666 ,

Thank you for the details

Yes. I would be glad to implement the Gitee backend and will update the process in #835 .

Perfect!

I think it's acceptable for users to do this, right? I am asking this because if we use entry_points, the only thing users need to do is adding the requirements in requirements.txt.

The entry_points solution looks much better from the users perspective.

The second one is taht this guide says that pkg_resources-style namespace packages is no longer recommended, it is widely present in most existing namespace packages. I am not sure if it would be better to use Native namespace packages or pkgutil-style namespace packages.

Reading the pointers you shared, the Native namespace packages are easier to manage and more recent than pkgutil-style namespace packages.

As commented at https://github.com/chaoss/grimoirelab-elk/issues/880#issuecomment-633001916, let's try use the Gitee implementations to test the entry points approach.

Let me know how you want to proceed, thanks

[WillemJiang]

+1 for using entry_points, as we can add more metadata information on the entry_points from utils.py to load. For Perceval, we just load the Backend and BackendCommand classes in _import_backends(modules)

[valeriocos]

For Perceval, we just load the Backend and BackendCommand classes in _import_backends(modules)

I would move the discussion about how to do it in Perceval to the Perceval issue tracker

[heming6666]

Hi @valeriocos @sduenas , sorry for the delay in replying :)

I created a pull request #885 that uses entry points approach to support plugin mechanism. Please help to review it.

Besides, I have updated the proof-of-concept repository that used entry points approach.

As commented at #880 (comment), let's try use the Gitee implementations to test the entry points approach.

Here's the initial implementation of grimoirelab-elk-gitee. And disscusions about the implementation of grimoirelab-elk-gitee can be found in #835 .

Any feedback or suggestions are appreciated. Thanks~

[heming6666]

Hi @valeriocos @sduenas , please help look into this issue whenever you have time :)

[valeriocos]

Hi @heming6666 , I'm a bit busy these days, I'll try to review it ASAP. Sorry for the delay.

[heming6666]

Hi @heming6666 , I'm a bit busy these days, I'll try to review it ASAP. Sorry for the delay.

It's ok. Thank you @valeriocos very much!

[jjmerchante]

This was implemented in https://github.com/chaoss/grimoirelab-elk/pull/1062

Thank you.