Make Vorta modular

[bastiencyr]

trafficstars

Hello @m3nu ,

The proposition I want to submit is just a proposition and I didnt started to code anything. Even if we started to code this, we need a proof of concept with a minimalist example.

I did some researches over the module pattern :) This is a way to organize a code and make it more flexible.

The idea is that you have a core application, ie a backup software who can launch backup and collect metrics (archives, logs...) which are accessible by modules via the api. And you have modules. But in this approach, all can be modules. Even the archive tab can be seen as a non optional module which access metrics in db and print it to the user. All core functions are accessible via the API and we can control access (Read, write...).

Pros:

• All is modules
• Code re-usability (there is a benefit only if the api is used many times).
• Better encapsulation and separation of tasks. More easy to find bugs.
• Make code easy to maintain because independent and extensible. You can remove an entire module folder without crash the ui. So we can add beta or experimental code without compromise the core.
• Make code not cross platform ( schedule with systemd for example ). Its also a way to provide cross platform code when cross platform python lib doesn't exist and give us the time to implement for all os (for example snapshots for btrfs/mac...)
• You can choose what you want in the ui and avoid having 10 tabs
• Remove utils.py in part
• Start to integrate experimental QML support (qt has recently add a function to integrate QML in classical qtWidget)

Cons:

• All is modules
• Some module can be bad integrated in the ui
• Some Code can be not cross platform
• Need a lot of changes on current code but we can start step by step. We can for the moment just implement a single very tiny module as a POC, and then a bigger module (dashboard) and maybe (or not) migrate some current code. We first need a proof of concept and solid api. Notice that modules implementation is independent from the current code.

How a modular project looks like : src/vorta: ------ assets ------ borg ------ i18n ------ keyring ------ network_status ------ views ------ application.py ------ ... ------ api ------------ db_ro (read only) ------------ db_wo ------------ signals (connect vorta signals to modules method) ------------ ui (add modules components to the ui) ------------ ... ------ modules ------------ dashboard ------------ btrfs_snapshot ------------ mac_snapshot ------------ send_mail_if_error ------------ archive_tab
------------ a_useless_ml_algo ------------ a_module_in_another_lang

Api is build at the top of current_code. And modules at the top of the api.

Thanks for reading :)

[m3nu]

I'm always open to structure the project better. We already have a decent structure (I like to think). Like Borg, keyring, views, etc.

So views would become modules and include some business logic?

Agreed that signals and maybe the DB could be structured better.

Can you map our existing modules and packages to the tree above? To get a better idea.

[bastiencyr]

Can you map our existing modules and packages to the tree above? To get a better idea.

I have updated my first comment.

We already have a decent structure (I like to think). Like Borg, keyring, views, etc.

The current structure is good ! Modules structure is just build at the top of the current code and only a part of the code could be moved in the new struct.

I didn't have pb with signals. But yes with db I had some ones with multiple repos, there wasnt any method to interact with db. It lacks a DAO but core api in modules pattern can include this DAO.

So views would become modules and include some business logic?

For the beginning, we dont move any code. We can just build an api over existent code to add features with modules. But even the core app (its to say the current code) can use the api and must use. And after we can move some code in modules. And maybe your right, it will be the code in views that can be moved.

[bastiencyr]

Finally, What is your point of view over this @m3nu ?

[m3nu]

I'm following but haven't fully wrapped my head around it. So modules would be between views and api? Wouldn't keyring and network_status be models too?

[bastiencyr]

So modules would be between views and api?

Modules use api to build a particular view

Wouldn't keyring and network_status be models too?

Im not sure to understand. Did you mean modules instead of models ? It depends on what we decide to put in modules. In an extreme module organization, all is modules so keyring and network_status too. But it can be in the core too.

I didnt introduce core in the tree above. The core is all what is not in modules or api folder.

[real-yfprojects]

I agree to @bastiencyr that a transition of the code base to a modular approach will be an improvement. I would even argue that it is an important and necessary step that has to be done at some point when the feature set of vorta keeps growing.

It might make sense to convert the GUI into a modular system but I think the main problem is that the GUI is not separated from the application logic. Instead we should use Model/View Programming and expand this idea to the vorta code base itself. There should be classes/subpackages for the GUI and classes/subpackages for interacting with borg and also for doing the borg jobs. There should be a background daemon that is independent from the GUI that runs the scheduled backups and such while the GUI processes user input and tells the daemon what the user wants it to do. All these greater modules (gui, daemon, borg-wrapper) should interact only through an API we define so that they don't have to know what happens behind the API.

This also means that we do not loose cross platform support but make it easier to increase it. This is because components other than the keyring component don't have to think about which platform they are on and keyring back-end specifics. Only the keyring component handles this. APIs could be defined using abstract classes that are than subclassed while other components treat the instances as instances of the abstract superclass.

I visualized my first concrete approach for vorta in the following diagram. It is not complete at all and it even lacks some main features of the current vorta because I don't have the overall picture of the code base and the details of the libraries used. Additionally I am not sure what the profiles are all about and which user stories (use cases) they are supposed to tackle.

[bastiencyr]

Its a good known architecture in client server model. It is used in other backup software and I also thought about the same organization. Apart from the separation of the ui and the backend, it can solve some issue regarding scheduling (Currently scheduling need a graphical session which is problematic for backup when user is not connected). Notice that Module and client-server can be implemented together.

I have discarded this architecture because this involved a lot of changes in one time. But the server (so VortaDeamon) can be implemented step by step and client can migrate to the new arch step by step. But I think it still implies a huge change.

In fact, there is a sort of "server" in the current architecture but it only includes borg (communication with JSON). Vorta adds a nice backend layer (keyring, profile among other things) and so it's not client-server orientated.

Note: Profiles provide a nice way to organize your backups. There are proven use cases, for example some users want to backup some folders in one location and some others in another location. VortaDeamon can handle profiles. The lines between keyringProvider/SettingsProvider and VortaDeamon were confusing for a real client-server arch until I saw "access through VortaDeamon".

[real-yfprojects]

Actually I envisioned a Python internal API using classes (therefore access through VortaDaemon), one single process with multiple threads and qt signals to keep the GUI integration simple. But using multiple processes and sockets or similar would be way better because it overcomes the need for a graphical session and the involvement of qt in the backup script making it more reliable/stable.

Is it intended to use multiple profiles for the same borg repository?

[bastiencyr]

Is it intended to use multiple profiles for the same borg repository?

I don't know, I don't use an other profile than the default one.

[real-yfprojects]

@m3nu You still backing modularization of vorta? I am encountering more and more bugs or FRs that are very hard to fix with the current code base and would be easier when we had a proper modularization.

[m3nu]

There are already a lot of packages and modules, no? What's missing?

Else my opinion is "whoever does the work decides"