BladeDISC
BladeDISC copied to clipboard
coredump when set tf_enable_tao be true
tensorflow version is 1.15.5, cuda version cuda-10.1
when set TF_ENABLE_TAO true, coredump as following:
2023-07-17 21:34:47.554155: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:288] Restoring SavedModel bundle. *** Aborted at 1689600887 (unix time) try "date -d @1689600887" if you are using GNU date *** PC: @ 0x7f62463f39d4 tensorflow::tao::GetGlobalJitLevelForGraph() *** SIGSEGV (@0x400000011) received by PID 288 (TID 0x7f6031358700) from PID 17; stack trace: *** @ 0x7f629ffe0929 google::(anonymous namespace)::FailureSignalHandler() @ 0x7f62a06a25f0 (unknown) @ 0x7f62463f39d4 tensorflow::tao::GetGlobalJitLevelForGraph() @ 0x7f624638fafa tensorflow::tao::(anonymous namespace)::MarkForCompilation() @ 0x7f6246392fb0 tensorflow::tao::TaoMarkForCompilationPass::Run() @ 0x7f624639403c tensorflow::tao::TaoOptimizationPass::Run() @ 0x7f62b0ffb196 tensorflow::OptimizationPassRegistry::RunGrouping() @ 0x7f62aeae25e0 tensorflow::GraphExecutionState::BuildGraph() @ 0x7f62aea245da tensorflow::DirectSession::CreateGraphs() @ 0x7f62aea25fc7 tensorflow::DirectSession::CreateExecutors() @ 0x7f62aea29689 tensorflow::DirectSession::MakeCallable() @ 0x7f62a7a3a95d tensorflow::(anonymous namespace)::RunOnce() @ 0x7f62a7a3b56d tensorflow::(anonymous namespace)::RunRestore() @ 0x7f62a7a3cdce tensorflow::(anonymous namespace)::LoadSavedModelInternal() @ 0x7f62a7a3d8e4 tensorflow::LoadSavedModel() @ 0x7f62a79eeffb tensorflow::serving::LoadSessionBundleOrSavedModelBundle() @ 0x7f62a78d5bf5 tensorflow::serving::SavedModelBundleV2Factory::InternalCreateSavedModelBundleV2() @ 0x7f62a78d6781 tensorflow::serving::SavedModelBundleV2Factory::CreateSavedModelBundleV2() @ 0x7f62a78d1887 ZNSt17_Function_handlerIFN10tensorflow6StatusEPSt10unique_ptrINS0_18SavedModelBundleV2ESt14default_deleteIS3_EEEZNKS0_7serving31SavedModelBundleV2SourceAdapter18GetServableCreatorESt10shared_ptrINS9_25SavedModelBundleV2FactoryEERKSsEUlS7_E_E9_M_invokeERKSt9_Any_dataOS7 @ 0x7f62a78cf65e tensorflow::serving::SimpleLoader<>::Load() @ 0x7f62a78b67b9 _ZNSt17_Function_handlerIFN10tensorflow6StatusEvEZNS0_7serving13LoaderHarness4LoadEvEUlvE_E9_M_invokeERKSt9_Any_data @ 0x7f62a78b90ea tensorflow::serving::Retry() @ 0x7f62a78b7ec6 tensorflow::serving::LoaderHarness::Load() @ 0x7f62a78b3603 tensorflow::serving::BasicManager::ExecuteLoad() @ 0x7f62a78b3b8c tensorflow::serving::BasicManager::ExecuteLoadOrUnload() @ 0x7f62a78b56a2 tensorflow::serving::BasicManager::HandleLoadOrUnloadRequest() @ 0x7f62a78b5892 _ZNSt17_Function_handlerIFvvEZN10tensorflow7serving12BasicManager20LoadOrUnloadServableERKNS3_19LoadOrUnloadRequestESt8functionIFvRKNS1_6StatusEEEEUlvE0_E9_M_invokeERKSt9_Any_data @ 0x7f62b177be0c Eigen::ThreadPoolTempl<>::WorkerLoop() @ 0x7f62b1778d98 _ZNSt17_Function_handlerIFvvEZN10tensorflow6thread16EigenEnvironment12CreateThreadESt8functionIS0_EEUlvE_E9_M_invokeERKSt9_Any_data @ 0x7f629e1c7b9f execute_native_thread_routine @ 0x7f62a069ae65 start_thread @ 0x7f629dc2988d __clone