torch-mlir icon indicating copy to clipboard operation
torch-mlir copied to clipboard
verified publisher icon llvm

[Tracker] Fix tests added to xfail_set due to Roll PyTorch and LLVM bump update

Open vivekkhandelwal1 opened this issue 1 year ago • 7 comments

This issue is for tracking the failing tests due to the roll PyTorch update done here: https://github.com/llvm/torch-mlir/pull/3727. Some of the tests are common for the FX_IMPORTER_XFAIL_SET and FX_IMPORTER_STABLEHLO_XFAIL_SET.

FX_IMPORTER_XFAIL_SET

  • [ ] "CrossEntropyLossModule_basic",
  • [ ] "CrossEntropyLossNoReductionModule_basic",
  • [x] "ElementwiseExpm1IntModule_basic",
  • [x] "ElementwiseExpm1Module_basic",
  • [x] "IndexPutImpl1DFloatAccumulateModule_basic",
  • [x] "IndexPutImpl1DFloatNonAccumulateModule_basic",
  • [x] "IndexPutImpl1DIntAccumulateModule_basic",
  • [x] "IndexPutImpl1DIntNonAccumulateModule_basic",
  • [x] "IndexPutImpl2DFloatNonAccumulateModule_basic",
  • [x] "IndexPutImpl2DImplicitModule_basic",
  • [x] "IndexPutImpl2DIndexModule_basic",
  • [x] "IndexPutImpl2DNoneIndexStaticModule_basic",
  • [x] "IndexPutImpl3DFloatNonAccumulateModule_basic",
  • [x] "IndexPutImplIndexWithNoneModule_basic",
  • [ ] "InterpolateDynamicModule_sizes_nearest",
  • [ ] "IouOfModule_basic",
  • [ ] "MeshgridIndexingIJ_basic",
  • [ ] "MeshgridIndexingXY_basic",
  • [ ] "Meshgrid_basic",
  • [x] "OneHotModule_basic",
  • [ ] "BernoulliFloatModule_basic",
  • [ ] "BernoulliTensorModule_basic",
  • [ ] "UniformModule_basic",
  • [ ] "UniformStaticShapeModule_basic",
  • [ ] "RollModule_basic"

FX_IMPORTER_CRASHING_SET

  • [ ] "AdaptiveAvgPool1dGeneralDynamicNoBatches_basic",

FX_IMPORTER_STABLEHLO_XFAIL_SET

  • [ ] "AdaptiveAvgPool1dGeneralDynamic_basic",
  • [ ] "AdaptiveAvgPool1dGeneralDynamicNoBatches_basic",
  • [ ] "AdaptiveAvgPool1dStaticEvenMultiple_basic",
  • [ ] "AdaptiveAvgPool1dStaticLargerOutput_basic",
  • [ ] "AdaptiveAvgPool2dDynamicNoBatch_basic",
  • [ ] "AdaptiveAvgPool2dDynamic_basic",
  • [ ] "AddIntModule_basic",
  • [ ] "AtenIntTensorByteDtypeModule_basic",
  • [ ] "AtenIntTensorCharDtypeModule_basic",
  • [ ] "AtenItemIntOpModule_basic",
  • [ ] "CrossEntropyLossModule_basic",
  • [ ] "CrossEntropyLossNoReductionModule_basic",
  • [ ] "EinsumStaticContractRhsModule_basic",
  • [ ] "EinsumStaticFourDimensionModule_basic",
  • [ ] "EinsumStaticModule_basic",
  • [ ] "EinsumStaticWithEllipsisSlicingAndBroadcastModule_basic",
  • [ ] "EinsumStaticWithEllipsisSlicingModule_basic",
  • [ ] "ElementwiseExpm1IntModule_basic",
  • [ ] "ElementwiseExpm1Module_basic",
  • [ ] "InterpolateDynamicModule_sizes_nearest",
  • [ ] "IouOfModule_basic",
  • [ ] "IscloseStaticModuleTrue_basic",
  • [ ] "IscloseStaticModule_basic",
  • [ ] "MeshgridIndexingIJ_basic",
  • [ ] "MeshgridIndexingXY_basic",
  • [ ] "Meshgrid_basic",
  • [ ] "MulIntModule_basic",
  • [x] "OneHotModule_basic",
  • [ ] "ReduceFrobeniusNormComplexModule_basic",
  • [ ] "ScalarImplicitIntModule_basic",
  • [ ] "ScaledDotProductAttentionBoolMaskModule_basic",
  • [ ] "ScaledDotProductAttentionDifferentCausalModule_basic",
  • [ ] "ScaledDotProductAttentionDifferentDynamicCausalModule_basic",
  • [ ] "ScaledDotProductAttentionDifferentModule_basic",
  • [ ] "ScaledDotProductAttentionMaskModule_basic",
  • [ ] "ScaledDotProductAttentionSameCausalModule_basic",
  • [ ] "ScaledDotProductAttentionSameDynamicModule_basic",
  • [ ] "ScaledDotProductAttentionSameModule_basic",
  • [ ] "SubIntModule_basic",
  • [ ] "TensorToIntZeroRank_basic",
  • [ ] "UpSampleNearest2dDynamicFactor_basic",
  • [ ] "UpSampleNearest2dDynamicSize_basic",
  • [ ] "UpSampleNearest2dStaticFactor_basic",
  • [ ] "UpSampleNearest2dStaticSize_basic",
  • [ ] "UpSampleNearest2d_basic",
  • [ ] "BernoulliFloatModule_basic",
  • [ ] "UniformModule_basic",
  • [ ] "UniformStaticShapeModule_basic",

FX_IMPORTER_STABLEHLO_CRASHING_SET

  • [ ] "CrossEntropyLossModule_basic",
  • [ ] "CrossEntropyLossNoReductionModule_basic",

FX_IMPORTER_TOSA_XFAIL_SET

  • [ ] "AdaptiveAvgPool1dNonUnitOutputSizeDynamicModule_basic"
  • [ ] "AdaptiveAvgPool1dNonUnitOutputSizeStaticModule_basic"
  • [ ] "AdaptiveAvgPool1dUnitOutputSizeDynamicModule_basic"
  • [ ] "AdaptiveAvgPool1dUnitOutputSizeStaticModule_basic"
  • [ ] "ElementwiseRreluTrainModule_basic"
  • [ ] "ElementwiseRreluTrainStaticModule_basic"
  • [ ] "MaxPool1dEmptyStrideStaticModule_basic"
  • [ ] "MaxPool1dStaticCeilModeTrueModule_basic"
  • [ ] "MaxPool1dStaticModule_basic"
  • [ ] "ReduceFrobeniusNormKeepDimModule_basic"
  • [ ] "RepeatInterleaveSelfIntModule_basic"
  • [ ] "ElementwiseSpecialExpm1IntModule_basic",
  • [ ] "ElementwiseSpecialExpm1Module_basic",

vivekkhandelwal1 avatar Oct 15 '24 15:10 vivekkhandelwal1

Hi @vivekkhandelwal1, @sjarus and I are also tracking fx_importer_tosa regressions from this update. Here are some additional tests that failed on fx_importer_tosa that haven't been included in your list. I also have the full list if you want:

  • [ ] "AdaptiveAvgPool1dNonUnitOutputSizeDynamicModule_basic"
  • [ ] "AdaptiveAvgPool1dNonUnitOutputSizeStaticModule_basic"
  • [ ] "AdaptiveAvgPool1dUnitOutputSizeDynamicModule_basic"
  • [ ] "AdaptiveAvgPool1dUnitOutputSizeStaticModule_basic"
  • [ ] "ElementwiseRreluTrainModule_basic"
  • [ ] "ElementwiseRreluTrainStaticModule_basic"
  • [ ] "MaxPool1dEmptyStrideStaticModule_basic"
  • [ ] "MaxPool1dStaticCeilModeTrueModule_basic"
  • [ ] "MaxPool1dStaticModule_basic"
  • [ ] "ReduceFrobeniusNormKeepDimModule_basic"
  • [ ] "RepeatInterleaveSelfIntModule_basic"

justin-ngo-arm avatar Nov 01 '24 21:11 justin-ngo-arm

Hi @vivekkhandelwal1, @sjarus and I are also tracking fx_importer_tosa regressions from this update. Here are some additional tests that failed on fx_importer_tosa that haven't been included in your list. I also have the full list if you want:

Hi @justin-ngo-arm, thanks for the update. Actually, the CI does not run the fx_importer_tosa tests hence I missed them. I have updated the above list.

vivekkhandelwal1 avatar Nov 05 '24 11:11 vivekkhandelwal1

@vivekkhandelwal1 which targets does CI run now ? We'd previously discussed potentially swapping one or more existing TOSA targets for fx_importer_tosa.

sjarus avatar Nov 06 '24 00:11 sjarus

@vivekkhandelwal1 which targets does CI run now ? We'd previously discussed potentially swapping one or more existing TOSA targets for fx_importer_tosa.

Hi @sjarus, we run the following script https://github.com/llvm/torch-mlir/blob/main/build_tools/ci/test_posix.sh to run the tests in CI. The configs for which tests are run are: onnx, fx_importer, fx_importer_stablehlo.

vivekkhandelwal1 avatar Nov 12 '24 11:11 vivekkhandelwal1

@vivekkhandelwal1 which targets does CI run now ? We'd previously discussed potentially swapping one or more existing TOSA targets for fx_importer_tosa.

Hi @sjarus, we run the following script https://github.com/llvm/torch-mlir/blob/main/build_tools/ci/test_posix.sh to run the tests in CI. The configs for which tests are run are: onnx, fx_importer, fx_importer_stablehlo.

Thanks! What would be the logistical impact of adding fx_importer_tosa to that list ? There are multiple contributors to the TorchToTosa path who would benefit from this.

sjarus avatar Nov 12 '24 17:11 sjarus

@vivekkhandelwal1 which targets does CI run now ? We'd previously discussed potentially swapping one or more existing TOSA targets for fx_importer_tosa.

Hi @sjarus, we run the following script https://github.com/llvm/torch-mlir/blob/main/build_tools/ci/test_posix.sh to run the tests in CI. The configs for which tests are run are: onnx, fx_importer, fx_importer_stablehlo.

Thanks! What would be the logistical impact of adding fx_importer_tosa to that list ? There are multiple contributors to the TorchToTosa path who would benefit from this.

I don't exactly remember why did we remove this but AFAICR the reason might have been irregular maintenance of this code path. If you think we should have this in CI again then I think @stellaraccident is the best person to take a call on this.

vivekkhandelwal1 avatar Nov 14 '24 06:11 vivekkhandelwal1

If it is maintained and of reasonable duration, I think we can handle that. Certainly open to give it a try and see how it goes.

stellaraccident avatar Nov 14 '24 07:11 stellaraccident