[RFC] Integrate Hyperparameter Optimization into Llama Stack

[varshaprasad96]

🚀 Describe the new functionality needed

This issue proposes integrating Hyperparameter Optimization (HPO) into the Llama stack to enhance model performance tuning and improve efficiency in parameter selection.

💡 Why is this needed? What if we don't build it?

Many state-of-the-art ML workflows rely on efficient HPO techniques such as (inspired from KATIB):

1. Grid Search & Random Search (baseline search methods)
2. Bayesian Optimization & TPE (probabilistic methods)
3. Hyperband (adaptive early stopping)
4. Population-Based Training (PBT) (dynamic adaptation of hyperparameters)

Integrating HPO will significantly benefit users by automating hyperparameter tuning, leading to:

• Better model accuracy and efficiency
• Reduced manual tuning efforts

In addition to the above, HPO can be used to fine-tune parameters such as retrieval strategies, chunking methods, embedding dimensions, and prompt tuning to improve the quality and relevance of generated responses in RAG-based applications.

Other thoughts

This issue would need a RFC to evaluate if we can extend the existing score API to also run HPO, or build an isolated API to support it. It would be helpful to know if the community finds this feature helpful. If so, we can move to brainstorming the possible ways to achieve it.

[franciscojavierarceo]

This is great @varshaprasad96 do you want to link your blog post when it's ready? (or maybe PR for now)

[varshaprasad96]

Sure, here is the PR: https://github.com/kubeflow/blog/pull/161

[SLR722]

Thanks for the propose! Which APIs do you plan to integrate HPO to? post_training?

[franciscojavierarceo]

I think that makes sense! We'd probably wrap it in a Kubeflow Pipeline.

CC @mprahl @HumairAK @anishasthana @astefanutti @szaher @JamesKunstle

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity within 60 days. It will be automatically closed if no further activity occurs within 30 days.

[github-actions[bot]]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant!