[RFC] Integrate Hyperparameter Optimization into Llama Stack

[varshaprasad96]

🚀 Describe the new functionality needed

This issue proposes integrating Hyperparameter Optimization (HPO) into the Llama stack to enhance model performance tuning and improve efficiency in parameter selection.

💡 Why is this needed? What if we don't build it?

Many state-of-the-art ML workflows rely on efficient HPO techniques such as (inspired from KATIB):

1. Grid Search & Random Search (baseline search methods)
2. Bayesian Optimization & TPE (probabilistic methods)
3. Hyperband (adaptive early stopping)
4. Population-Based Training (PBT) (dynamic adaptation of hyperparameters)

Integrating HPO will significantly benefit users by automating hyperparameter tuning, leading to:

• Better model accuracy and efficiency
• Reduced manual tuning efforts

In addition to the above, HPO can be used to fine-tune parameters such as retrieval strategies, chunking methods, embedding dimensions, and prompt tuning to improve the quality and relevance of generated responses in RAG-based applications.

Other thoughts

This issue would need a RFC to evaluate if we can extend the existing score API to also run HPO, or build an isolated API to support it. It would be helpful to know if the community finds this feature helpful. If so, we can move to brainstorming the possible ways to achieve it.