jeffhataws
jeffhataws
Hi @miladm @zpcore , you can remove XLA_USE_FP16 and XLA_DOWNCAST_FP16 in 2.7 (I though these are already removed). It would be best to remove XLA_USE_BF16 and XLA_DOWNCAST_BF16 in 2.8 instead....
Sure. XLA_USE_FP16 and XLA_DOWNCAST_FP16 are already deleted. I will update those lines to be BF16 and add the release for full removal.
xrt_world_size and get_ordinal changes needed in HF transformers/accelerate: https://github.com/huggingface/transformers/issues/37301
@bfolie any idea what the error above could be? " The composite op functionalization fallback expects its inputs all not to be functional tensors".
@ysiraichi is this still a problem? Is this why we don't have nightly torch-xla builds since 8/28?
Some naive questions: 1) The doc describe going from PyTorch native to XLA equivalence. Should there be a path going the other way? 2) Can the propagator be the main...
Mark this as performance issue since affects lazy mode training performance for multiple models.
Strange. This issue is not reproducible on GPU. It is confusing.
It turns out disabling functionalization (XLA_DISABLE_FUNCTIONALIZATION=1) is causing the error. It is XLA_DISABLE_FUNCTIONALIZATION=1 by default in Neuron environment. To resolve this, please set XLA_DISABLE_FUNCTIONALIZATION=0. ``` cpu_venv_py310) ubuntu@ip-10-3-190-82:~$ XLA_DISABLE_FUNCTIONALIZATION=0 PJRT_DEVICE=CPU python...
Reopen since XLA_DISABLE_FUNCTIONALIZATION=1 is still used by Neuron.