PiPPy icon indicating copy to clipboard operation
PiPPy copied to clipboard

[DTensor] missing rule for aten.fill.Scalar causing unit tests to fail for SPMD

Open lessw2020 opened this issue 2 years ago • 0 comments

PR's for tau are failing due to an unrelated missing rule for DTensor: "Operator aten.fill.Scalar does not have a DistributedTensor rule registered."

Details: Traceback (most recent call last): File "/__w/tau/tau/test/spmd/tensor/test_dtensor_ops.py", line 606, in run_dtensor_crossref dtensor_rs = func(*dtensor_args, **dtensor_kwargs) File "/usr/local/lib/python3.9/site-packages/torch/_tensor.py", line 1287, in torch_function ret = func(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/torch/distributed/_tensor/api.py", line 232, in torch_dispatch return op_dispatch.operator_dispatch( File "/usr/local/lib/python3.9/site-packages/torch/distributed/_tensor/dispatch.py", line 244, in operator_dispatch target_schema, redistribute, output_sharding = propagate_input_sharding( File "/usr/local/lib/python3.9/site-packages/torch/distributed/_tensor/dispatch.py", line 183, in propagate_input_sharding raise NotImplementedError( NotImplementedError: Operator aten.fill.Scalar does not have a DistributedTensor rule registered.

Example of issue in unit tests: https://github.com/pytorch/tau/actions/runs/3921608865/jobs/6704028631

lessw2020 avatar Jan 15 '23 18:01 lessw2020