[DRAFT] RFC Plugin-system for Storage back-ends

[jjerphan]

In order to have ArcticDB installable, usable, and maintainable, modularization of the code with respect of the storage back-ends might be valuable.

Benefits for users:

• access most used back-end directly (S3, LMDB) directly
• be able use or create a plugin for what you need if not available independently
• make ArcticDB usable in more conda environment (since dependencies are removed)

Benefits for maintenance:

• separate of concerns between back-ends as much as possible, and reach SOLID principles
• decouple ArcticDB from any (proprietary) storage solution
• stop adding or even remove heavy dependencies on the main project for the need of only some users

UX:

• we could imagine having users install a specific storage backend with:

  mamba install -c conda-forge "arcticdb[storage_backend]"
  

  pip install "arcticdb[storage_backend]"
  

Actionable steps include but are not limited to:

• [x] https://github.com/man-group/ArcticDB/pull/625
• [x] https://github.com/man-group/ArcticDB/issues/633
• [ ] a plugin extension point
  • [ ] defining interfaces which can be extended by users

Resources:

• Making a Plugin System

[DerThorsten]

as a plugin system we can use https://github.com/QuantStack/xplugin which we just released