ipex-llm
ipex-llm copied to clipboard
Update nano training ipex bf16
Description
1. Why the change?
related to https://github.com/intel-analytics/BigDL/issues/5288
2. User API changes
Setting use_ipex=True and precision='bf16' will cast model parameters to torch.bfloat16
and use fused optimizer step for training.
trainer = Trainer(max_epochs=max_epochs,
use_ipex=True,
precision='bf16')
Only setting=16/32/64/'bf16' will use the PyTorch-Lightning precision plugin
trainer = Trainer(max_epochs=max_epochs,
precision=64)
3. Summary of the change
Subclass the PrecisionPlugin
(pytorch_lightning.plugins.precision.PrecisionPlugin
) and enable torch.cpu.amp.autocast that autocast input to torch.bfloat16
. And call closure
for optimizers not supported by ipex.
4. How to test?
- [x] Unit test
- [x] Jenkins http://10.112.231.51:18889/job/BigDL-Chronos-PR-Validation/674/