axolotl
axolotl copied to clipboard
OpenAccess-AI-Collective
Hyperparameter optimization CLI
⚠️ Please check that this feature request hasn't been suggested before.
- [X] I searched previous Ideas in Discussions didn't find any similar feature requests.
- [X] I searched previous Issues didn't find any similar feature requests.
🔖 Feature description
Create a new CLI that allows users to perform hyperparameter optimization to get the most optimized training runs.
✔️ Solution
Using Optuna could be a solution. https://github.com/optuna/optuna
❓ Alternatives
No response
📝 Additional Context
No response
Acknowledgements
- [X] My issue title is concise, descriptive, and in title casing.
- [X] I have searched the existing issues to make sure this feature has not been requested yet.
- [X] I have provided enough information for the maintainers to understand and evaluate this request.
what would this look like in axolotl, specifically? new module? arguments? what would the output look like?
@ehartford you can sort of optimize three things in general:
- learning rate
- batch size
- scheduler
You should specify an axolotl config. Additionally, you need extra arguments to specify the search space you want to optimize within. For example, you could specify a range 0.0005 - 0.01 to optimize the learning rate.
A good reference: https://shengdinghu.notion.site/MiniCPM-Unveiling-the-Potential-of-End-side-Large-Language-Models-d4d3a8c426424654a4e80e42a711cb20
Yes - I was asking how you imagine the feature fitting into axolotl's architecture and workflow
Let's imagine that Axolotl had the feature.
What would the command / arguments be to invoke it? Would it happen in the course of training or as a separate manually launched event? Would it use arguments or a config file? What changes to the config file schema?
Hi everybody, I guess there should be a balise 'use_optuna' If it is set to True, you could put a range for learning rate (and not int) and a list for learning rate scheduler. So everything is in just in the config file.
