promptflow
promptflow copied to clipboard
microsoft
[Feature Request] Allow any combination of variants to be selected for a run
Is your feature request related to a problem? Please describe.
When a flow has more than one node with each node having more than one variant, only one variant can be specified per run. When running with the selected node variant, all other nodes with variants will run with their specified default. Because of this functionality, it is not possible to run all combinations of variants without having to update defaults.
For illustration consider the following flow: I have two LLM nodes each with 2 variants. In each node there is a variant that adds a statement to the prompt to limit the number of characters and the other variant does not. In this situation, I would like to experiment running all combinations of the prompts. For the illustration, I will refer to node one and two as N1 and N2, and Variant one and to as V1 and V2
| Total possible combinations |
|---|
| N1V1*, N2V1* |
| N1V1*, N2V2 |
| N1V2, N2V1* |
| N1V2, N2V2 |
- Indicates default variant
Given the constraint a node variant can only run against another node's default variant, in the case of multiple nodes with multiple variants, there will exist nodes that are unreachable. In the example above, if we specify V1 as the default for each node. In this case, the combination of non-default variant of Node 1 with non-default variant of Node 2 would not be possible to execute. Describe the solution you'd like I believe this is probably not something straightforward, but I would think maybe:
- Provide ability to have variant be defined at the flow level
- Provide ability to specify which variant will be used for which node (seems that this might get complicated)
Describe alternatives you've considered I have not been able to find a workaround here aside from changing the default prior to running the experiment.
Additional context An example of the flow above can be found here: https://github.com/pragmatical/llmops-promptflow-template/blob/main/web_classification/flows/experiment/flow.dag.yaml
Hello, is there any progress on this issue? I don't think this should be a long-term goal.
Lacking the ability of running a combination of all variants is an obvious need for anyone familiar with grid search in machine learning. @brynn-code @elliotzh
Reference from the MS learn: https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/how-to-tune-prompts-using-variants?view=azureml-api-2#run-all-variants-with-a-single-row-of-data-and-check-outputs
Hello @ZhiliangWu is it ok if I start prototyping around this feature ?
What are the potential changes involved in such a change besides simply adding a list in the argparse and then loading each node variant as specified in that list ?
Hello @ZhiliangWu is it ok if I start prototyping around this feature ?
What are the potential changes involved in such a change besides simply adding a list in the argparse and then loading each node variant as specified in that list ?
Hello @sashokbg, thanks for picking this up! That's up to your design. For me, it would be great if we can deliberately choose the variants for multiple nodes with variants (instead of only being able to overwrite one)
--variant ${node_name1.variant_name_1} --variant ${node_name2.variant_name_1}
What do you think?
Hello @ZhiliangWu is it ok if I start prototyping around this feature ? What are the potential changes involved in such a change besides simply adding a list in the argparse and then loading each node variant as specified in that list ?
Hello @sashokbg, thanks for picking this up! That's up to your design. For me, it would be great if we can deliberately choose the variants for multiple nodes with variants (instead of only being able to overwrite one)
--variant ${node_name1.variant_name_1} --variant ${node_name2.variant_name_1}What do you think?
Hello @ZhiliangWu yes the input seems clear enough. My worries are that within the code we reference a run instance with it's variant. And currently the variant is just one string. So to add multi-param variants we should change the "primary" key of a run's instance etc and this might have some implications that I am not aware. (I have only very limited knowledge of the inner parts of the project and I am not a python developer)
I will do some crunching and come back to you.
@crazygao, @YingChen1996, @wangchao1230 any thoughts tips on that topic ?
I would recommend not to take this now. As we haven't clearly defined the signature. The return type of pf.runs.create should be signle run. What's the sdk API poposal here?
Hello @wangchao1230 the api would still be the same. pf.runs.create will return a single run, but the args.variant will become a list.
The idea is that the a variant (let's called it Flow Variant) should be a list of node overrides.
We can model this as a new class ex:
class FlowVariant: _name: str # allow to give a meaningful name to a variant ex: mistral_with_vector_search _overrides: list[str] # ex: [${node_chat.mistral}, ${node_rag.vector}]
Hi, Any updates for this feature request ? It should not be long-term goal.
