transformers
transformers copied to clipboard
huggingface
Introduce configured_state arg for accelerator_config
What does this PR do?
This PR starts to allow users to bring in their own Accelerator. To start, we are simply allowing users to define a PartialState or AcceleratorState outside the TrainingArguments, and then enable the user to use its state using a new use_configured_state arg.
For instance, a user can now do:
from accelerate import PartialState
from transformers import TrainingArguments
state = PartialState()
args = TrainingArguments(accelerator_config={"use_configured_state":True})
And this will use the defined state already made.
These states are Singeltons, so defining it once and calling it will maintain the same state on subsequent calls.
This may lead to issues with hyperparameter tuning, which requires the state to be reset each time, as noted in the doc description
Fixes related accelerate issue which occurs when defining an Accelerator before the TrainingArguments: https://github.com/huggingface/accelerate/issues/2564
Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the contributor guideline, Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
- [x] Did you write any new necessary tests?
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.
@amyeroberts @pacman100
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
@amyeroberts requesting a rereview after having to gut much of it on further testing 😅
