iree icon indicating copy to clipboard operation
iree copied to clipboard
verified publisher icon iree-org

Compiler hitting assert for several "shape" ONNX tests

Open ScottTodd opened this issue 1 year ago • 1 comments

Misc background:

Noticed these failures while looking at https://github.com/openxla/iree/pull/16811#issuecomment-2004317128. Haven't been able to determine if this is a regression / find a culprit. Probably an issue in upstream torch-mlir, but might already be fixed. IREE CI is running these tests without LLVM asserts, so something is slipping through the test suite.

General info

Building at https://github.com/openxla/iree/commit/ee32fc7b74b4aad49fc46d67ea6c1f617416c4d4

Seeing test failures from compiler crashes here:

FAILED onnx/node/generated/test_shape_end_1/model.mlir::cpu_llvm_sync
FAILED onnx/node/generated/test_shape_start_1_end_negative_1/model.mlir::cpu_llvm_sync
FAILED onnx/node/generated/test_shape_start_1/model.mlir::cpu_llvm_sync
FAILED onnx/node/generated/test_shape_start_negative_1/model.mlir::cpu_llvm_sync
FAILED onnx/node/generated/test_shape_start_1_end_2/model.mlir::cpu_llvm_sync
FAILED onnx/node/generated/test_shape_end_negative_1/model.mlir::cpu_llvm_sync

Sample test case source: https://github.com/nod-ai/SHARK-TestSuite/blob/main/iree_tests/onnx/node/generated/test_shape_start_1/model.mlir

Repro command: iree-compile model.mlir --iree-hal-target-backends=llvm-cpu -o /dev/null

Logs

Callstack from Windows (click to expand) 


iree-compile.exe!HandleAbort(int Sig) Line 424 (d:\dev\projects\iree\third_party\llvm-project\llvm\lib\Support\Windows\Signals.inc:424)
ucrtbase.dll!00007ffe72681881() (Unknown Source:0)
ucrtbase.dll!00007ffe72682851() (Unknown Source:0)
ucrtbase.dll!00007ffe726841b5() (Unknown Source:0)
ucrtbase.dll!00007ffe726844f1() (Unknown Source:0)
iree-compile.exe!mlir::DenseElementsAttr::get(mlir::ShapedType type, llvm::ArrayRef<:attribute> values) Line 898 (d:\dev\projects\iree\third_party\llvm-project\mlir\lib\IR\BuiltinAttributes.cpp:898)
iree-compile.exe!mlir::torch::Torch::Aten_ShapeAsTensorOp::fold(mlir::torch::Torch::Aten_ShapeAsTensorOpGenericAdaptor<:arrayref>> adaptor) Line 3381 (d:\dev\projects\iree\third_party\torch-mlir\lib\Dialect\Torch\IR\TorchOps.cpp:3381)
[Inline Frame] iree-compile.exe!mlir::Op<:torch::torch::aten_shapeastensorop>::Impl,mlir::OpTrait::ZeroSuccessors,mlir::OpTrait::OneOperand,mlir::OpTrait::OpInvariants,mlir::torch::Torch::OpTrait::AllowsTypeRefinement,mlir::torch::Torch::OpTrait::HasValueSemantics,mlir::torch::Torch::OpTrait::ReadOnly>::foldSingleResultHook(mlir::Operation *) Line 1903 (d:\dev\projects\iree\third_party\llvm-project\mlir\include\mlir\IR\OpDefinition.h:1903)
iree-compile.exe!mlir::Op<:torch::torch::aten_shapeastensorop>::Impl,mlir::OpTrait::ZeroSuccessors,mlir::OpTrait::OneOperand,mlir::OpTrait::OpInvariants,mlir::torch::Torch::OpTrait::AllowsTypeRefinement,mlir::torch::Torch::OpTrait::HasValueSemantics,mlir::torch::Torch::OpTrait::ReadOnly>::getFoldHookFn::__l5::(mlir::Operation * op, llvm::ArrayRef<:attribute> operands, llvm::SmallVectorImpl<:opfoldresult> & results) Line 1879 (d:\dev\projects\iree\third_party\llvm-project\mlir\include\mlir\IR\OpDefinition.h:1879)
iree-compile.exe!llvm::detail::UniqueFunctionBase<:logicalresult>,llvm::SmallVectorImpl<:opfoldresult> &>::CallImpl<:logicalresult>(mlir::Operation *, llvm::ArrayRef<:attribute>, llvm::SmallVectorImpl<:opfoldresult> &) const>(void * CallableAddr, mlir::Operation * , llvm::ArrayRef<:attribute> , llvm::SmallVectorImpl<:opfoldresult> & ) Line 221 (d:\dev\projects\iree\third_party\llvm-project\llvm\include\llvm\ADT\FunctionExtras.h:221)
iree-compile.exe!llvm::unique_function<:logicalresult __cdecl>,llvm::SmallVectorImpl<:opfoldresult> &)const>::operator()(mlir::Operation * , llvm::ArrayRef<:attribute> , llvm::SmallVectorImpl<:opfoldresult> & ) Line 411 (d:\dev\projects\iree\third_party\llvm-project\llvm\include\llvm\ADT\FunctionExtras.h:411)
iree-compile.exe!mlir::RegisteredOperationName::Model<:torch::torch::aten_shapeastensorop>::foldHook(mlir::Operation * op, llvm::ArrayRef<:attribute> attrs, llvm::SmallVectorImpl<:opfoldresult> & results) Line 539 (d:\dev\projects\iree\third_party\llvm-project\mlir\include\mlir\IR\OperationSupport.h:539)
[Inline Frame] iree-compile.exe!mlir::OperationName::foldHook(mlir::Operation *) Line 266 (d:\dev\projects\iree\third_party\llvm-project\mlir\include\mlir\IR\OperationSupport.h:266)
iree-compile.exe!mlir::Operation::fold(llvm::ArrayRef<:attribute> operands, llvm::SmallVectorImpl<:opfoldresult> & results) Line 636 (d:\dev\projects\iree\third_party\llvm-project\mlir\lib\IR\Operation.cpp:636)
iree-compile.exe!mlir::Operation::fold(llvm::SmallVectorImpl<:opfoldresult> & results) Line 666 (d:\dev\projects\iree\third_party\llvm-project\mlir\lib\IR\Operation.cpp:666)
iree-compile.exe!`anonymous namespace'::GreedyPatternRewriteDriver::processWorklist() Line 490 (d:\dev\projects\iree\third_party\llvm-project\mlir\lib\Transforms\Utils\GreedyPatternRewriteDriver.cpp:490)
[Inline Frame] iree-compile.exe!`anonymous-namespace'::RegionPatternRewriteDriver::simplify::__l4::::operator()() Line 839 (d:\dev\projects\iree\third_party\llvm-project\mlir\lib\Transforms\Utils\GreedyPatternRewriteDriver.cpp:839)
iree-compile.exe!llvm::function_ref::callback_fn>(__int64 callable) Line 45 (d:\dev\projects\iree\third_party\llvm-project\llvm\include\llvm\ADT\STLFunctionalExtras.h:45)
[Inline Frame] iree-compile.exe!llvm::function_ref::operator()() Line 68 (d:\dev\projects\iree\third_party\llvm-project\llvm\include\llvm\ADT\STLFunctionalExtras.h:68)
[Inline Frame] iree-compile.exe!mlir::MLIRContext::executeAction(llvm::function_ref) Line 275 (d:\dev\projects\iree\third_party\llvm-project\mlir\include\mlir\IR\MLIRContext.h:275)
iree-compile.exe!`anonymous namespace'::RegionPatternRewriteDriver::simplify(bool * changed) Line 837 (d:\dev\projects\iree\third_party\llvm-project\mlir\lib\Transforms\Utils\GreedyPatternRewriteDriver.cpp:837)
iree-compile.exe!mlir::applyPatternsAndFoldGreedily(mlir::Region & region, const mlir::FrozenRewritePatternSet & patterns, mlir::GreedyRewriteConfig config, bool * changed) Line 880 (d:\dev\projects\iree\third_party\llvm-project\mlir\lib\Transforms\Utils\GreedyPatternRewriteDriver.cpp:880)
[Inline Frame] iree-compile.exe!mlir::applyPatternsAndFoldGreedily(mlir::Operation *) Line 147 (d:\dev\projects\iree\third_party\llvm-project\mlir\include\mlir\Transforms\GreedyPatternRewriteDriver.h:147)
iree-compile.exe!`anonymous namespace'::Canonicalizer::runOnOperation() Line 63 (d:\dev\projects\iree\third_party\llvm-project\mlir\lib\Transforms\Canonicalizer.cpp:63)
[Inline Frame] iree-compile.exe!mlir::detail::OpToOpPassAdaptor::run::__l2::::operator()() Line 519 (d:\dev\projects\iree\third_party\llvm-project\mlir\lib\Pass\Pass.cpp:519)
iree-compile.exe!llvm::function_ref::callback_fn>(__int64 callable) Line 45 (d:\dev\projects\iree\third_party\llvm-project\llvm\include\llvm\ADT\STLFunctionalExtras.h:45)
[Inline Frame] iree-compile.exe!llvm::function_ref::operator()() Line 68 (d:\dev\projects\iree\third_party\llvm-project\llvm\include\llvm\ADT\STLFunctionalExtras.h:68)
[Inline Frame] iree-compile.exe!mlir::MLIRContext::executeAction(llvm::function_ref) Line 275 (d:\dev\projects\iree\third_party\llvm-project\mlir\include\mlir\IR\MLIRContext.h:275)
iree-compile.exe!mlir::detail::OpToOpPassAdaptor::run(mlir::Pass * pass, mlir::Operation * op, mlir::AnalysisManager am, bool verifyPasses, unsigned int parentInitGeneration) Line 525 (d:\dev\projects\iree\third_party\llvm-project\mlir\lib\Pass\Pass.cpp:525)
iree-compile.exe!mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager & pm, mlir::Operation * op, mlir::AnalysisManager am, bool verifyPasses, unsigned int parentInitGeneration, mlir::PassInstrumentor * instrumentor, const mlir::PassInstrumentation::PipelineParentInfo * parentInfo) Line 585 (d:\dev\projects\iree\third_party\llvm-project\mlir\lib\Pass\Pass.cpp:585)
[Inline Frame] iree-compile.exe!mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl::__l2::::operator()(mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl::__l2::OpPMInfo & opInfo) Line 805 (d:\dev\projects\iree\third_party\llvm-project\mlir\lib\Pass\Pass.cpp:805)
iree-compile.exe!mlir::failableParallelForEach<:_vector_iterator>>>, &>(mlir::MLIRContext * context, std::_Vector_iterator<:_vector_val>>> begin, std::_Vector_iterator<:_vector_val>>> end, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl::__l2:: & func) Line 46 (d:\dev\projects\iree\third_party\llvm-project\mlir\include\mlir\IR\Threading.h:46)
iree-compile.exe!mlir::failableParallelForEach<:vector>> &, &>(mlir::MLIRContext * context, std::vector> & range, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl::__l2:: & func) Line 92 (d:\dev\projects\iree\third_party\llvm-project\mlir\include\mlir\IR\Threading.h:92)
iree-compile.exe!mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool verifyPasses) Line 815 (d:\dev\projects\iree\third_party\llvm-project\mlir\lib\Pass\Pass.cpp:815)
[Inline Frame] iree-compile.exe!mlir::detail::OpToOpPassAdaptor::run::__l2::::operator()() Line 517 (d:\dev\projects\iree\third_party\llvm-project\mlir\lib\Pass\Pass.cpp:517)
iree-compile.exe!llvm::function_ref::callback_fn>(__int64 callable) Line 45 (d:\dev\projects\iree\third_party\llvm-project\llvm\include\llvm\ADT\STLFunctionalExtras.h:45)
[Inline Frame] iree-compile.exe!llvm::function_ref::operator()() Line 68 (d:\dev\projects\iree\third_party\llvm-project\llvm\include\llvm\ADT\STLFunctionalExtras.h:68)
[Inline Frame] iree-compile.exe!mlir::MLIRContext::executeAction(llvm::function_ref) Line 275 (d:\dev\projects\iree\third_party\llvm-project\mlir\include\mlir\IR\MLIRContext.h:275)
iree-compile.exe!mlir::detail::OpToOpPassAdaptor::run(mlir::Pass * pass, mlir::Operation * op, mlir::AnalysisManager am, bool verifyPasses, unsigned int parentInitGeneration) Line 525 (d:\dev\projects\iree\third_party\llvm-project\mlir\lib\Pass\Pass.cpp:525)
iree-compile.exe!mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager & pm, mlir::Operation * op, mlir::AnalysisManager am, bool verifyPasses, unsigned int parentInitGeneration, mlir::PassInstrumentor * instrumentor, const mlir::PassInstrumentation::PipelineParentInfo * parentInfo) Line 585 (d:\dev\projects\iree\third_party\llvm-project\mlir\lib\Pass\Pass.cpp:585)
iree-compile.exe!mlir::detail::OpToOpPassAdaptor::run::__l2::(mlir::OpPassManager & pipeline, mlir::Operation * root) Line 502 (d:\dev\projects\iree\third_party\llvm-project\mlir\lib\Pass\Pass.cpp:502)
iree-compile.exe!llvm::function_ref<:logicalresult __cdecl>::callback_fn<:logicalresult>(mlir::OpPassManager &, mlir::Operation *)>(__int64 callable, mlir::OpPassManager & , mlir::Operation * ) Line 45 (d:\dev\projects\iree\third_party\llvm-project\llvm\include\llvm\ADT\STLFunctionalExtras.h:45)
[Inline Frame] iree-compile.exe!llvm::function_ref<:logicalresult __cdecl>::operator()(mlir::OpPassManager &) Line 68 (d:\dev\projects\iree\third_party\llvm-project\llvm\include\llvm\ADT\STLFunctionalExtras.h:68)
[Inline Frame] iree-compile.exe!mlir::Pass::runPipeline(mlir::OpPassManager &) Line 199 (d:\dev\projects\iree\third_party\llvm-project\mlir\include\mlir\Pass\Pass.h:199)
iree-compile.exe!mlir::iree_compiler::`anonymous namespace'::AutoInputConversionPipelinePass::runOnOperation() Line 64 (d:\dev\projects\iree\compiler\src\iree\compiler\InputConversion\Common\AutoInputConversionPipeline.cpp:64)
[Inline Frame] iree-compile.exe!mlir::detail::OpToOpPassAdaptor::run::__l2::::operator()() Line 519 (d:\dev\projects\iree\third_party\llvm-project\mlir\lib\Pass\Pass.cpp:519)
iree-compile.exe!llvm::function_ref::callback_fn>(__int64 callable) Line 45 (d:\dev\projects\iree\third_party\llvm-project\llvm\include\llvm\ADT\STLFunctionalExtras.h:45)
[Inline Frame] iree-compile.exe!llvm::function_ref::operator()() Line 68 (d:\dev\projects\iree\third_party\llvm-project\llvm\include\llvm\ADT\STLFunctionalExtras.h:68)
[Inline Frame] iree-compile.exe!mlir::MLIRContext::executeAction(llvm::function_ref) Line 275 (d:\dev\projects\iree\third_party\llvm-project\mlir\include\mlir\IR\MLIRContext.h:275)
iree-compile.exe!mlir::detail::OpToOpPassAdaptor::run(mlir::Pass * pass, mlir::Operation * op, mlir::AnalysisManager am, bool verifyPasses, unsigned int parentInitGeneration) Line 525 (d:\dev\projects\iree\third_party\llvm-project\mlir\lib\Pass\Pass.cpp:525)
iree-compile.exe!mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager & pm, mlir::Operation * op, mlir::AnalysisManager am, bool verifyPasses, unsigned int parentInitGeneration, mlir::PassInstrumentor * instrumentor, const mlir::PassInstrumentation::PipelineParentInfo * parentInfo) Line 585 (d:\dev\projects\iree\third_party\llvm-project\mlir\lib\Pass\Pass.cpp:585)
iree-compile.exe!mlir::PassManager::runPasses(mlir::Operation * op, mlir::AnalysisManager am) Line 896 (d:\dev\projects\iree\third_party\llvm-project\mlir\lib\Pass\Pass.cpp:896)
iree-compile.exe!mlir::PassManager::run(mlir::Operation * op) Line 875 (d:\dev\projects\iree\third_party\llvm-project\mlir\lib\Pass\Pass.cpp:875)
iree-compile.exe!mlir::iree_compiler::embed::`anonymous namespace'::Invocation::runPipeline(iree_compiler_pipeline_t pipeline) Line 961 (d:\dev\projects\iree\compiler\src\iree\compiler\API\Internal\CompilerDriver.cpp:961)
iree-compile.exe!mlir::iree_compiler::runIreecMain::__l2::(iree_compiler_source_t * source) Line 247 (d:\dev\projects\iree\compiler\src\iree\compiler\Tools\iree_compile_lib.cc:247)
iree-compile.exe!mlir::iree_compiler::runIreecMain(int argc, char * * argv) Line 348 (d:\dev\projects\iree\compiler\src\iree\compiler\Tools\iree_compile_lib.cc:348)
[Inline Frame] iree-compile.exe!invoke_main() Line 78 (d:\a01\_work\43\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78)
iree-compile.exe!__scrt_common_main_seh() Line 288 (d:\a01\_work\43\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)
kernel32.dll!00007ffe742e7344() (Unknown Source:0)
ntdll.dll!00007ffe74a026b1() (Unknown Source:0)

Tail of --mlir-print-ir-before-all --debug:

// -----// IR Dump Before Canonicalizer (canonicalize) //----- //
mlir-asm-printer: Verifying operation: func.func
func.func @test_shape_start_1(%arg0: !torch.vtensor<[3,4,5],f32>) -> !torch.vtensor<[2],si64> attributes {torch.assume_strict_symbolic_shapes, torch.onnx_meta.ir_version = 10 : si64, torch.onnx_meta.opset_version = 21 : si64, torch.onnx_meta.producer_name = "backend-test", torch.onnx_meta.producer_version = ""} {
  %none = torch.constant.none
  %0 = torch.aten._shape_as_tensor %arg0 : !torch.vtensor<[3,4,5],f32> -> !torch.vtensor<[2],si64>
  return %0 : !torch.vtensor<[2],si64>
}

ImplicitTypeIDRegistry::lookupOrInsert(mlir::OpTrait::OneTypedResult<class mlir::Type>::Impl<struct `public: static class mlir::TypeID __cdecl mlir::TypeID::get<class mlir::OpTrait::OneTypedResult<class mlir::Type>::Impl>(void)'::`2'::Empty>)
ImplicitTypeIDRegistry::lookupOrInsert(mlir::OpTrait::OneOperand<struct `public: static class mlir::TypeID __cdecl mlir::TypeID::get<class mlir::OpTrait::OneOperand>(void)'::`2'::Empty>)
ImplicitTypeIDRegistry::lookupOrInsert(mlir::torch::Torch::OpTrait::HasValueSemantics<struct `public: static class mlir::TypeID __cdecl mlir::TypeID::get<class mlir::torch::Torch::OpTrait::HasValueSemantics>(void)'::`2'::Empty>)
ImplicitTypeIDRegistry::lookupOrInsert(mlir::torch::Torch::OpTrait::ReadOnly<struct `public: static class mlir::TypeID __cdecl mlir::TypeID::get<class mlir::torch::Torch::OpTrait::ReadOnly>(void)'::`2'::Empty>)

//===-------------------------------------------===//
Processing operation : 'torch.constant.none'(0x2296da19400) {
  %0 = "torch.constant.none"() : () -> !torch.none

ImplicitTypeIDRegistry::lookupOrInsert(mlir::OpTrait::HasRecursiveMemoryEffects<struct `public: static class mlir::TypeID __cdecl mlir::TypeID::get<class mlir::OpTrait::HasRecursiveMemoryEffects>(void)'::`2'::Empty>)
  ** Erase   : 'torch.constant.none'(0x2296da19400)
} -> success : operation is trivially dead
//===-------------------------------------------===//

//===-------------------------------------------===//
Processing operation : 'torch.aten._shape_as_tensor'(0x2296e262750) {
  %0 = "torch.aten._shape_as_tensor"(%arg0) : (!torch.vtensor<[3,4,5],f32>) -> !torch.vtensor<[2],si64>

Assertion failed: hasSameElementsOrSplat(type, values), file D:\dev\projects\iree\third_party\llvm-project\mlir\lib\IR\BuiltinAttributes.cpp, line 896

full output: https://gist.github.com/ScottTodd/6710e616b80385e6836811324dfbe560

ScottTodd avatar Mar 18 '24 16:03 ScottTodd

Still seeing these, and it's making updating XFAIL lists between local builds and CI builds slightly tricky. Should

  • Fix the crash
  • Enable asserts on pkgci

ScottTodd avatar Apr 25 '24 17:04 ScottTodd