Reorganize backends versus UI

[davinov]

Since the introduction of the server/ module, weaverbird is becoming more than just a front-end project to translate pipelines to Mongo queries.

We now have two different ways of implementing backends:

• translators: like the mongo one, which convert pipelines into corresponding queries
• executors: like the one made with pandas, which produces transformation functions

and two different ways to implement them:

• using TypeScript: to be executed in a browser or with node
• using Python: to be executed server-side.

They are similar in how they must handle steps, except the executors don't need an external DB to be run against.

The front-end part of weaverbird consist only of UI components to visualize and edit pipelines. It could be completely separated from the translators or executors in a different folder/module. The two only needs to share a list of supported steps for the translators and executors. This could be done using a simple JSON, (optionally generated from the translators/executors files), instead of typescript files without any implementation like https://github.com/ToucanToco/weaverbird/blob/master/src/lib/translators/pandas.ts A third part, shared between the UI module and the translators/executors, would be the definition of the pipeline steps.

The proposed architecture would look like:

lib/ # definitions of the pipeline steps
ui/ # front-end components
backends/ # or engines? where to implement translators and executors

This leads to a first action plan:

• [x] document the difference between translators and executors
• [ ] move the UI part in the ui/ folder
• [x] move the steps types to the lib/ folder
• [x] move the mongo translator to a backends/mongo subfolder
• [x] move the pandas executor to a backends/pandas subfolder
• [ ] rewrite the supported/unsupported steps logic in the UI part to remove the dependency to any translators
• [ ] document this architecture breakdown for contributors

Corollary: I'd like some suggestions for a common word to regroup translators and executors. I've thought of "backends" or "engines" but I'm not entirely convinced they are the right word.

Notes:

• the lib part should contain steps definitions in every language we'd like to allow backends implementation. In our case, it would be a collection of
• another issue/RFC will follow regarding the internal organization of the backends/ folder

[austil]

100% on board with this "structural" refactoring !

I'd like some suggestions for a common word to regroup translators and executors. I've thought of "backends" or "engines" but I'm not entirely convinced they are the right word.

Some food for though using the programming language taxonomy :

• translators "compiles" pipelines to another query language
• while executors "interpret" or "run" pipelines on the fly

and those are two different type of ... "implementations" ? Maybe "executors" should be the term for describing both in the end ? And the two type could be "translators" & "runtimes" ?
Isn't "backend" just fine ? What would be the full description of those ? "Weaverbird pipeline language backends" ?

Suggestion : do the schema that "document this architecture breakdown for contributors" first two see how everything fit together !

[davinov]

Done with #1581!