pytorch_dlprim icon indicating copy to clipboard operation
pytorch_dlprim copied to clipboard

crash on rusticl drivers: Missing aten::aten::isin.Tensor_Tensor_out and other operators

Open acbits opened this issue 9 months ago • 7 comments

Package:pytorch_ocl-0.2.0+torch2.4-cp312-none-linux_x86_64.whl

/home/dev/.local/lib/python3.12/site-packages/transformers/pytorch_utils.py:343: UserWarning: The operator 'aten::isin.Tensor_Tensor_out' is not currently supported on the ocl backend. Please open an issue at for requesting support https://github.com/artyom-beilis/pytorch_dlprim/issues (Triggered internally at /home/artik/Projects/dlprim_build/pytorch_dlprim/src/tensor_ops.cpp:415.)
  return torch.isin(elements, test_elements)
/home/dev/.local/lib/python3.12/site-packages/transformers/generation/utils.py:1882: UserWarning: The operator 'aten::any.all_out' is not currently supported on the ocl backend. Please open an issue at for requesting support https://github.com/artyom-beilis/pytorch_dlprim/issues (Triggered internally at /home/artik/Projects/dlprim_build/pytorch_dlprim/src/tensor_ops.cpp:415.)
  and isin_mps_friendly(elements=eos_token_tensor, test_elements=pad_token_tensor).any()
/home/dev/.local/lib/python3.12/site-packages/transformers/generation/utils.py:1891: UserWarning: The operator 'aten::any.all_out' is not currently supported on the ocl backend. Please open an issue at for requesting support https://github.com/artyom-beilis/pytorch_dlprim/issues (Triggered internally at /home/artik/Projects/dlprim_build/pytorch_dlprim/src/tensor_ops.cpp:415.)
  torch.is_floating_point(eos_token_tensor) or (eos_token_tensor < 0).any()
/home/dev/.local/lib/python3.12/site-packages/torch/nn/functional.py:2267: UserWarning: The operator 'aten::index_select' is not currently supported on the ocl backend. Please open an issue at for requesting support https://github.com/artyom-beilis/pytorch_dlprim/issues (Triggered internally at /home/artik/Projects/dlprim_build/pytorch_dlprim/src/tensor_ops.cpp:415.)
  return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
/home/dev/.local/lib/python3.12/site-packages/transformers/models/t5/modeling_t5.py:249: UserWarning: The operator 'aten::pow.Tensor_Scalar_out' is not currently supported on the ocl backend. Please open an issue at for requesting support https://github.com/artyom-beilis/pytorch_dlprim/issues (Triggered internally at /home/artik/Projects/dlprim_build/pytorch_dlprim/src/tensor_ops.cpp:415.)
  variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
/home/dev/.local/lib/python3.12/site-packages/transformers/models/t5/modeling_t5.py:250: UserWarning: The operator 'aten::rsqrt.out' is not currently supported on the ocl backend. Please open an issue at for requesting support https://github.com/artyom-beilis/pytorch_dlprim/issues (Triggered internally at /home/artik/Projects/dlprim_build/pytorch_dlprim/src/tensor_ops.cpp:415.)
  hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
/home/dev/.local/lib/python3.12/site-packages/transformers/models/t5/modeling_t5.py:441: UserWarning: The operator 'aten::where.self' is not currently supported on the ocl backend. Please open an issue at for requesting support https://github.com/artyom-beilis/pytorch_dlprim/issues (Triggered internally at /home/artik/Projects/dlprim_build/pytorch_dlprim/src/tensor_ops.cpp:415.)
  relative_buckets += torch.where(is_small, relative_position, relative_position_if_large)
/home/dev/.local/lib/python3.12/site-packages/transformers/generation/utils.py:1571: UserWarning: The operator 'aten::cumsum.out' is not currently supported on the ocl backend. Please open an issue at for requesting support https://github.com/artyom-beilis/pytorch_dlprim/issues (Triggered internally at /home/artik/Projects/dlprim_build/pytorch_dlprim/src/tensor_ops.cpp:415.)
  cache_position = torch.ones_like(input_ids[0, :], dtype=torch.int64).cumsum(0) - 1
/home/dev/.local/lib/python3.12/site-packages/transformers/models/t5/modeling_t5.py:1315: UserWarning: The operator 'aten::masked_fill_.Scalar' is not currently supported on the ocl backend. Please open an issue at for requesting support https://github.com/artyom-beilis/pytorch_dlprim/issues (Triggered internally at /home/artik/Projects/dlprim_build/pytorch_dlprim/src/tensor_ops.cpp:415.)
  causal_mask[:, :, :, :mask_length] = causal_mask[:, :, :, :mask_length].masked_fill(
LLVM ERROR: Cannot select: 0x7f6e84976030: f32 = and 0x7f6e8498f420, Constant:i32<2147483647>
  0x7f6e8498f420: f32 = bitcast 0x7f6e8498ea10
    0x7f6e8498ea10: i32 = llvm.amdgcn.wwm TargetConstant:i64<2832>, 0x7f6e8498ecb0
      0x7f6e8498db50: i64 = TargetConstant<2832>
      0x7f6e8498ecb0: i32 = llvm.amdgcn.readlane TargetConstant:i64<2650>, 0x7f6e8498d320, Constant:i32<63>
        0x7f6e8498ed90: i64 = TargetConstant<2650>
        0x7f6e8498d320: i32,ch,glue = CopyFromReg # D:1 0x7f6e8498e020, Register:i32 %2976, 0x7f6e8498e020:1
          0x7f6e849763b0: i32 = Register %2976
          0x7f6e8498e020: ch,glue = inlineasm # D:1 0x7f6e8498d240, TargetExternalSymbol:i64'; 6', MDNode:ch<null>, TargetConstant:i64<1>, TargetConstant:i32<2228234>, Register:i32 %2976, TargetConstant:i32<-2147483639>, Register:i32 %2977, 0x7f6e8498d240:1
            0x7f6e8498d8d0: i64 = TargetExternalSymbol'; 6'
            0x7f6e8498f030: i64 = TargetConstant<1>
            0x7f6e8498da70: i32 = TargetConstant<2228234>
            0x7f6e849763b0: i32 = Register %2976
            0x7f6e8498d5c0: i32 = TargetConstant<-2147483639>
            0x7f6e8498f730: i32 = Register %2977
            0x7f6e8498d240: ch,glue = CopyToReg # D:1 0x7f6e84975a10:1, Register:i32 %2977, 0x7f6e8498cb40
              0x7f6e8498f730: i32 = Register %2977
              0x7f6e8498cb40: i32 = bitcast # D:1 0x7f6e8498d400
                0x7f6e8498d400: f32 = fadd # D:1 0x7f6e8498dca0, 0x7f6e8498d1d0
                  0x7f6e8498dca0: f32 = fadd # D:1 0x7f6e8498d390, 0x7f6e8498cde0


                  0x7f6e8498d1d0: f32 = bitcast 0x7f6e8498d990

        0x7f6e8498d710: i32 = Constant<63>
  0x7f6e8498c980: i32 = Constant<2147483647>
In function: main
Aborted (core dumped)

acbits avatar Mar 08 '25 17:03 acbits