Add celery as task manager?

[moloney]

I am working on an issue right now, where certain PDF files can take 30+ minutes to be converted to text by pdfminer3. There are some improvements I am working on to help this issue, but for quality text extraction there will always be the potential it takes at least a couple minutes.

This got me thinking that the conversion should really be run as a background task, rather than just timing out on the http response. Using the celery package seems like a good fit for moin since it can use the filesystem as a message broker (doesn't require a redis instance).

Any thoughts?

[RogerHaase]

Current version does not support windows, but there may be a hack: https://www.distributedpython.com/2018/08/21/celery-4-windows/

Another use could be for email notifications when an item is saved.

[ThomasWaldmann]

When adding celery, do it in a way so it is optional to use it (use it only if there is configuration for it).

[moloney]

Sorry, I didn't think about windows support. An alternative could be something like huey, although it looks like the multiprocess worker option won't work in Windows. I guess being stuck with the multithreaded worker in Windows is still better than the current situation.

@ThomasWaldmann - if there is a pure python option that supports all platforms, would you be okay making it a hard dependency?