PiPPy
PiPPy copied to clipboard
[DTensor] missing rule for aten.fill.Scalar causing unit tests to fail for SPMD
PR's for tau are failing due to an unrelated missing rule for DTensor: "Operator aten.fill.Scalar does not have a DistributedTensor rule registered."
Details: Traceback (most recent call last): File "/__w/tau/tau/test/spmd/tensor/test_dtensor_ops.py", line 606, in run_dtensor_crossref dtensor_rs = func(*dtensor_args, **dtensor_kwargs) File "/usr/local/lib/python3.9/site-packages/torch/_tensor.py", line 1287, in torch_function ret = func(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/torch/distributed/_tensor/api.py", line 232, in torch_dispatch return op_dispatch.operator_dispatch( File "/usr/local/lib/python3.9/site-packages/torch/distributed/_tensor/dispatch.py", line 244, in operator_dispatch target_schema, redistribute, output_sharding = propagate_input_sharding( File "/usr/local/lib/python3.9/site-packages/torch/distributed/_tensor/dispatch.py", line 183, in propagate_input_sharding raise NotImplementedError( NotImplementedError: Operator aten.fill.Scalar does not have a DistributedTensor rule registered.
Example of issue in unit tests: https://github.com/pytorch/tau/actions/runs/3921608865/jobs/6704028631