distilabel
distilabel copied to clipboard
[FEATURE] `distilabel` server
Description
Currently distilabel
can be only executed locally executing either a python script with the pipeline or using the CLI.
For future integrations with Argilla UI or deploying distilabel
in services like HF Inference Endpoints, it would be nice to have a distilabel
server that it's waiting for requests to execute certain pipeline.
The server will have a queue of pipeline executions. The queue will be filled from a POST
requests to an HTTP server, that will contain the pipeline and the runtime parameters needed to execute the pipeline.
Adding a layer in distilabel
allowing to generate statistics about the pipeline execution could allow to send back the pipeline execution information to a UI using SSE.