Parallelization

[lexman]

In order to run several process at the same time, a worker should be able to join the main runner.

Workers should not have to be on the same machine, which would enable multiplateform workflows

A simple way of dealing with authentication and API whould be... To use shared directories : if you have the right to access and write to the project directory, you have the right to run a process and to update the workflow state

Moving the workfow's state to sqlite could be a way of :

• defining an interface between workers
• dealing with concurency
• storing workflow's state

One fear is that sqlite recommands not sharing databases with Windows or OSX because some file lock implementations are broken in these operating systems : https://www.sqlite.org/atomiccommit.html#sect_9_0

This issue could be avoided with an extra lock

Implementing concurecy could be the occasion to have a thread read stdout and stderr from a process, and display it in realtime (for the moment, logs are displayed once the process is complete)

With this system, one could add processing power to an already running workflow on his machine, on the fly. And contribute processing power to his coworkers if they share their developpement directory