haystack
haystack copied to clipboard
deepset-ai
Async Pipelines
We want to showcase async RAG pipelines in 2.0.
There is already async 2.0 code in: https://github.com/deepset-ai/haystack/commit/48534693a45072041309c3f9e3b572ef09f2ac9e
FYI: We currently don't know if asyncio actually is the best solution. Could be that threading might actually be better. @ArzelaAscoIi and I want to give this another stab this Friday
Depriotized for the beta release since we wont break existing functionality.
Can we bring this back? Would love to have async, so things are a bit easier for the backend - particularly if you call mostly REST endpoints in the pipeline.
honestly, if being super ideal .. then would use rust as backend to handle async/multithreading (no GIL in rust)/multiprocessing for efficiency and expose api in python with way to create components in rust or python ;)
Hi there, to anybody in this thread. We are going to host an async office hour next Tuesday in our discord server. Office hours are friendly informal sessions. We would love to see you there to find out more about your use case, and maybe share what you can already achieve today that might solve your problem.
Link for the event: https://discord.com/events/993534733298450452/1232379918822932530
We look forward to seeing you there!
Hi! Is there a clean outline somewhere about using async components with haystack?
If we cannot write all our components asynchronously (we are serving our chat endpoints via FastAPI) this would be a dealbreaker for us. It would also be nice if most components offered by haystack would be asynchronous.
I would be thankful for a quick update since we are doing a code refactor very soon. 🙂 Thank you!
Hi @tostenzel we're in the process of introducing an AsyncPipeline along with a subset of components for which it makes sense to support async execution. The AsyncPipeline will be able to run non-async components as well, so it will be a drop-in replacement for the normal Pipeline.
While this is still in development I would like to propose to also support Trio as an async framework. Trio is an async API for humans.
Writing sound code with Asyncio is notoriously difficult. I suggest reading some of these items to find out why. Trio makes it orders of magnitudes easier to write clean and tractable async code by following principles of "structured concurrency" . It's been a real life saver for me and I could imagine it would be for many others too.
Probably the easiest way to have Haystack support both Trio and Asyncio async frameworks simultaneously would be to use AnyIO.
Hi @tostenzel we're in the process of introducing an
AsyncPipelinealong with a subset of components for which it makes sense to support async execution. TheAsyncPipelinewill be able to run non-async components as well, so it will be a drop-in replacement for the normalPipeline.
@masci when do you anticipate this feature being released?
