distilabel icon indicating copy to clipboard operation
distilabel copied to clipboard

[FEATURE] Make LLM runtime parameters available in LLM instantiation

Open dvsrepo opened this issue 7 months ago • 2 comments

I remember we've briefly discussed this in the past and there were some actions, but I think this can have a positive impact on usability.

Go from:

with Pipeline(
    name="llama3-text-generation-pipeline",
) as pipeline:
    #...
    generation_with3 = TextGeneration(
      llm=InferenceEndpointsLLM(
        model_id="meta-llama/Meta-Llama-3-70B-Instruct",
      ),
    )

    load_dataset >> [generation_with3,generation_with3_1] >> CombineColumns(columns=["generation", "model_name"])

if __name__ == "__main__":
    distiset = pipeline.run(
        parameters={
            generation_with3.name: {
                "llm": {
                    "temperature": 1.0, 
                    "do_sample": True,
                    "frequency_penalty": 0.1
                }
            }
        },
        use_cache=False
    )  

to:

with Pipeline(
    name="llama3-text-generation-pipeline",
) as pipeline:

    #...

    generation_with3 = TextGeneration(
      llm=InferenceEndpointsLLM(
        model_id="meta-llama/Meta-Llama-3-8B-Instruct",
        temperature=1.0,
        do_sample=True,
        frequency_penalty=0.1
      ),
    )
    load_dataset >> [generation_with3,generation_with3_1] >> CombineColumns(columns=["generation", "model_name"])

if __name__ == "__main__":
    distiset = pipeline.run(use_cache=False)

This has several benefits:

  1. I don't need to configure my llm in two places and in two different ways (dict config vs init)
  2. Could this help to understand which params I can use for each LLM without needing to dig into the internals?
  3. A less fragmented API
  4. We wrap the generate/agenerate call anyway so why not letting users config everything into one place (LLM init).

I understand in this specific case (with the use_openai_client param) it might be challenging to expose all possible params but then maybe there are other decisions to make in terms of design.

Note that If someone runs the pipeline with parameters this will still override the values passed at init (I recall this was one of the arguments in favor of runtime params: reusability).

WDYT @gabrielmbmb @plaguss ? I'm especially interested in potential issues for (1) advanced users, and (2) maintenance.

dvsrepo avatar Jul 22 '24 13:07 dvsrepo