Plugin system discussion

[blitzmann]

I looked in the open issues and don't see any issue specifically dealing with a plugin system. As mentioned in #1555, calibre-web doesn't support plugins. All interaction with the application needs to be done through the application itself, and there's no real way to extend it without modifying the core codebase.

I'm currently developing my own customization on my own branch to handle the feature in #1555, the idea being I will just deploy it for my own use and offer the repo/branch as-is on github to anyone using it. But this makes me sad, so I did a bit of digging around and I found this: https://github.com/sh4nks/flask-plugins

It hasn't been updated in a few years, but honestly it looks simple enough to the point where it probably doesn't need to be updated unless something is broken (and the example/ works with with the current version of Flask and python 3.6 installed).

The gist of the package is that it offers injection points for both templates and code. So, for example, the navigation template can have {{ emit_event("tmpl_navigation_last") }}, and if a plugin is listening for that event it can provide it with additional navigations links specific to that plugin. The same idea can be extend for things like settings - each plugin can provide it's own settings section. The same goes for code as well - there can be events that happen before and after a book is uploaded, and the plugin can then us that data to make modifications, etc. Couple this with the fact that most other requirements for plugin things can be bundled as part of a Blueprint (which seems to be the case for most of the "extension" calibre-web has currently), and I think this could solve a good chunk of requirements for a "plugin system".

I'm sure it won't handle everything you might want to throw at it. For e xample, I know there have been large discussions surrounding the possibility of storage backend plugins and the like, and I'm not sure how exactly this proposed system would work with that (instead of events it's more like replacing functionality ¯_(ツ)_/¯), but I'm curious if it might be a good place to at least start and work from?

I am interested in hearing thoughts on this subject or other proposed solutions for a plugin system.😀

[gelsas]

https://pypi.org/project/Flask-PluginKit/

This seems to offer something similar like your linked project but has active development.

[blitzmann]

Ohh, nice. Don't have time to dig into that one right now, but interested in looking at it a bit more tonight :)

[chadfurman]

Would be interested in some follow-up conversations on this. Was there more work done in this direction, by chance? I wonder how something like https://github.com/janeczku/calibre-web/issues/451 could be implemented with this plugin system. Would need a way of extending the database to collect the number of existing ratings for averaging, I guess. Possibly a way to track ratings per user, and optionally a way for users to leave comments. All of that would have to feed back into the UI

[blitzmann]

I looked a little bit and think I enjoy the suggestion of https://pypi.org/project/Flask-PluginKit/

However, been pretty busy with other priorities, and I'm not sure if something like this would support database integration, though these things seem to use blueprints, and you generally have access to sql alchemy. Did a search and found this https://stackoverflow.com/a/47964689/788054

I would really like blessing / support / discussion / thoughts from @OzzieIsaacs before even going down this path. From my limited testing with Flask-PluginKit, it was pretty easy, and didn't interfere with anything else if all you do is simply include it as an option, which would allow third parties to develop against it. Of course, eventually you would want to update templates and other bits of code to be able to provide injection points.

[chadfurman]

It looks like @OzzieIsaacs pushed code 4 days ago so maybe we will hear something from him this weekend

[OzzieIsaacs]

Which features we need to handle with plugins

• Theming (see #928), like simple theme for ebook readers -> different topic, should be handled somehow else (Flask-Theme maybe)

• Adding different metadata providers ->I currently thinking about this and I think a system similar to the already present task system could handle this (having a collection of inherited classes, 2/3 routes to call and the route is calling a function in each class, combining the results and sending it to browser). No config except maybe turn metadata provider on and off (which also could be handled somehow generic)

• different storage backends (currently there is no real interface for handling the storage backend, and my experience with the gdrive backend teached me, this will be hard and maybe never happen)

• "pre download" hook, so users could rename there files according to their needs (calibre seems to have a similar plugin called "plugboard")

• customized "send to reader" function. Some users wanted to have different formats send to their readers, or converted several formats to each other before sending it, change metadata, ...

• "post upload" hook so users could add books to shelfs, add/remove tags, series, change cover pictures, and so on after uploading

• extend/change "views", to have series views with alternate cover pictures (combination of all books of a series), showing cover of next unread book of a series, authors page with authors pictures, a view which shows all books uploaded from a certain user or whatever,

• "post login" hook to display a welcome message or the message of the day

• "timed background tasks" like reconnect to database after a certain time, delete users which have'nd logged in for a long time

• "independent" extensions like display and download user download statistics, discussion board for users, wishlist for books, or the rating system from #451, Auto-download of Humble Bundle books from #1555, ...

Overall it's a wide range of needed features and things to consider (and I think this is the reason, why I never started working on it). I had a look into both recommended libs Flask-PluginKit has the advantage of being actively developed. But to me it looks like it's missing an important feature for creating the above plugins. It only provids hooks for before/after/before first request. The event system from flask-plugins looks to me more flexible, but has definitely the problem that it is not actively developed.

[tinywrkb]

@OzzieIsaacs

Here are a few other examples to keep in mind when deciding on the available API methods and features.

To the extend/change "views" category, I would add a filtering view, like the Find Duplicates Calibre Plugin.

Maybe add another category, metadata extractor/parser. The Extract ISBN Calibre plugin would fit this category.
It's different from metadata provider plugins as it actually needs access to the book file.

[Dunrar]

@OzzieIsaacs

Would there be any value in trying to implement a subset of the features you mentioned with one of the libraries? To have some code to look at? Or is further planing needed? If you think it's worth spending time on, which library would you prefer?

[OzzieIsaacs]

@Dunrar:

Adding different metadata providers

This is meanwhile done. I think a lot on the "different storage backends" this needs a lot of refactoring to make the storage backend exchangeable. Currently it looks more like a system of plugin and not THE plugin system.

So it makes sense to start somewhere, as the one and only solution will not show up. And as usual tests are needed (I spend more time on writing and running tests than on programming). PRs for big changes like this please towards the development channel