AMQP for working queues between SCAN, INDEX, NLP

[bamthomas]

Is your feature request related to a problem? Please describe.

We have task queue that is either:

• Memory
• Redis
• AMQP

The parameter is batchQueueType

Then we have queueType that are used to instantiate queues used between pipeline tasks (scan, index, deduplicate...).

Describe the solution you'd like

We could imagine:

• to have a single queueType parameter that is used by both
• to consider each extract of a file/document as a task and use task queues, avoiding these two mecanisms

[github-actions[bot]]

This issue is stale because it has been open for 40 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 20 days since being marked as stale.

[github-actions[bot]]

This issue was closed because it has been inactive for 20 days since being marked as stale.