Plugin installation mechanism

[prescod]

Snowfakery could have a small package manager based on pip.

For example:

snowfakery --install git+https://github.com/SkypLabs/faker-wifi-essid

Or:

snowfakery --upgrade git+https://github.com/SkypLabs/faker-wifi-essid

Or:

snowfakery --install faker-wifi-essid

Perhaps if a plugin is referenced like this:

• plugin: git+https://github.com/SkypLabs/faker-wifi-essid!wifi_essid_provider.Provider

Then Snowfakery could automatically and internally do something like:

$ pip install -r git+https://github.com/SkypLabs/faker-wifi-essid --target=~/.snowfakery/site-packages

That directory would also be in the PythonPath.

As usual, cache invalidation and write-through would need to be worked out.

Also, the location of the cache would need to be controllable (probably through an ENV_VAR) for unusual situations.

The main reason to consider doing plugin installation management this way is that it doesn't require "commands" and therefore works the same way whether snowfakery is being used by itself or in an embedded context.

@acrosman

[acrosman]

I have a preference for being able to add project-specific/local plugins over worrying about global packages run through Snowfakery and the concerns of dealing with larger system issues.

There are two arguments that come to mind:

1. Faker community providers already have a pattern for adding those to faker itself. So for people who want to have the same provider across multiple projects can use that approach today. And other plugins could operate on the same patterns already well documented in the Python community.
2. The barrier to entry for a new plugin is a good deal lower if it can be part of the local project and doesn't require a more complete and published Python package. Both in terms of helping someone new get started and being able to focus on the one task they want to complete, and in terms of someone in a hurry just trying to add a quick feature not needing to worry about larger project overhead.

[prescod]

So for your particular example of snowfakery_extras, you envision that people would copy specific files from your repo into theirs?

[acrosman]

I was wondering about a CCI Task-like approach where we could have a plugins directory that's project specific and copy in whatever was needed for that project. The snowfakery_extras example I was playing with I was largely about making it work and while getting to understand the process -- the details of what that provider does is probably a poor example of the need I am seeing. But it would certainly ease development of a larger package by being able to have it just hang around in a local project for development and testing until it reached a point of being worth reconfiguration for easy distribution.

For anything that would be really useful to re-use on many projects, I think it could be properly packaged for Python setup and installed via pip to add to the system Python module path. But it would certainly ease development of a larger package by being able to have it just hang around in a local project for development and testing until it reached a point of being worth reconfiguration for easy distribution.

[davisagli]

@prescod It's probably worth a larger discussion, but I have an immediate negative reaction to the idea of automatically installing plugins as part of processing a recipe. There's definitely potential for a malicious plugin to be inadvertently installed. Also, how should we deal with the potential for different plugins to depend on incompatible versions of a common dependency? Installing into an explicit target directory helps avoid polluting the Python environment where snowfakery is installed, but if snowfakery needs Faker 5 but a plugin needs Faker 4, it doesn't help with that.

[acrosman]

Proposed solution in #229

[prescod]

We'll call this "fixed in #229".