ipex-llm icon indicating copy to clipboard operation
ipex-llm copied to clipboard

Update nano training ipex bf16

Open y199387 opened this issue 1 year ago • 0 comments

Description

1. Why the change?

related to https://github.com/intel-analytics/BigDL/issues/5288

2. User API changes

Setting use_ipex=True and precision='bf16' will cast model parameters to torch.bfloat16 and use fused optimizer step for training.

trainer = Trainer(max_epochs=max_epochs,
                           use_ipex=True,
                           precision='bf16')

Only setting=16/32/64/'bf16' will use the PyTorch-Lightning precision plugin

trainer = Trainer(max_epochs=max_epochs,
                           precision=64)

3. Summary of the change

Subclass the PrecisionPlugin (pytorch_lightning.plugins.precision.PrecisionPlugin) and enable torch.cpu.amp.autocast that autocast input to torch.bfloat16. And call closure for optimizers not supported by ipex.

4. How to test?

  • [x] Unit test
  • [x] Jenkins http://10.112.231.51:18889/job/BigDL-Chronos-PR-Validation/674/

y199387 avatar Aug 08 '22 00:08 y199387