Support for custom sources

[JamesRamm]

Hi First up - great work on the library, looks like it is going to solve a bunch of our problems so we dont have to! :+1:

I'd like to make it easy to create source 'plugins' which can lie outside the the joerd code tree so that users can easily get it to work with their own sources. Given that a huge portion of the issues are requests for supporting other data sources, I think it could be good to make it easy to extend joerd at runtime.

Proposed changes:

• Allow plugin in plugin.py to accept a full path to a module to import, rather than assuming it lies in the joerd package. This would in turn require _sources in server.py to support a different way of specifying modules to allow external modules. Something like:

for source in cfg.sources:
           if 'module' in source:
               #import module from outside joerd package
               create_fn = external_plugin(source['module']) # could also adjust the signature of 'plugin' to support different arugments, rather than creating a new plugin function
            else:
                source_type = source['type']
                create_fn = plugin('source', source_type, 'create')

• Define an interface for source plugins to implement. As far as I can see, joerd implicitly defines an interface - from the _sources function it appears a source must support a create method. Other methods to support are listed here: https://github.com/tilezen/joerd/blob/master/CONTRIBUTING.md It could be good to explicitly define the interface - e.g. provide an abstract base class for plugin developers to implement. I'd also advocate moving the documentation on the interface from CONTRIBUTING to the README or some dedicated documentation. The line create_fn = plugin('source', source_type, 'create') seems slightly counter-productive as plugin supports a custom entry point function, but server then hardcodes it to create. I'd either take the name of the entrypoint from the configuration (e.g. sources['entrypoint']), or just define that all plugins must implement create and attempt to call it (throwing an appropriate error if it doesnt exist).

I think these are relatively small changes which would allow users to implement their own data sources without having to either a) fork the entire codebase or b) wait for their pull request to be merged and a new release to be created.

Popular sources could still be merged into the base library, or even a separate plugin library maintained? Same approach could be applied to stores and outputs

[nvkelso]

Thanks for your kind comments and proposal!

We've been working on v2 which addresses some of these concerns over at https://github.com/mojodna/marblecutter/. Please poke around over there to see if it suits your needs.