Discussion: handled async tasks

[vsoch]

Given that a snakemake workflow is launched from panoptes, how would it be best to handle waiting for it? E.g., you could add some kind of queue with redis or celery, but on the other hand, we don't want to make the tool too bulky to use. Are you planning to add another service to Docker compose for this, or some strategy to just make the user wait in the interface while it's running the job?

[fgypas]

I think that @dafentoulis and @gkostoulas are more experienced than me on this issue and might have a better idea/vision.

Personally, I would like to avoid relying only on a solution that requires an admin to install the service. The whole point is that simple users can deploy the service with ease and use it almost out of the box. This is why I would avoid using a queueing solution if possible. This is one of the reasons why we were thinking of switching to FastAPI (see #102 ), as it is supporting background tasks.

[fgypas]

Touching again this. After some years of experience it seems that a queue is the way to go as @vsoch suggested. One of the things I observed happening is that http requests get lost when the workflows are rather big and the service cannot cope with them.

[weber8thomas]

Hi @vsoch, @fgypas , thanks for all the developments on the project! what is the current roadmap regarding this ? I tested panoptes quite intensively over the last 6 months but I'm still facing some serious performance issues when dealing with large snakemake workflows. Panoptes is not responding suddenly and need to be restarted, thus the snakemake pipelines also need to be stopped and restarted, and so on. Do you still consider switching to FastAPI ?

[vsoch]

I can’t comment it’s not my project - I was only briefly helping out.

[fgypas]

Hi @weber8thomas I have also experienced similar issues when hundreds of requests are sent to the web server. I think that there are multiple issues:

• The database is sqlite, and it cannot handle a big number of http requests
• Another thing is that we do not use a ASGI (Asynchronous Server Gateway Interface) that might help when the number of requests is high. There is this recommendation: https://github.com/panoptes-organization/panoptes/issues/172 . In other (not open source) projects where FastAPI is used together with uvicorn (properly configured) we can handle thousands of requests.
• The app at the moment is monolithic. Yes, it is easy to install and run, but maybe we need to split the different parts of the application and make it more generic (DB, Front end, wms-monitor API, back end for the logic).

The problem is that I don't have much time to work on the project, but I think that the second bullet point is easy to implement and also maybe the first one. For the third one I think that we would need a bigger team, so we need to reach out to the community. I am also happy to give the project to someone like @vsoch or other members of the snakemake community to move it forward.