🐛 [mediapipe_task_genai] On Flutter/Android llm engine crashes : failed to validate graph

[tempo-riz]

OS Platform and Distribution

tried on Windows 11 and mac M3

Mobile device

Android emulator and real device api 34

Programming Language and version

flutter 3.22 and dart 3.4

Describe the actual behavior

crashes with "LLM Engine failed in ValidatedGraphConfig Initialization step."

Describe the expected behaviour

expect the model to give actual output !

Standalone code/steps you may have used to try to get what you need

Describe the bug LLM Engine failed in ValidatedGraphConfig Initialization step.

Steps to reproduce Steps to reproduce the behavior:

Download gemma-2b-it-gpu-int8.bin weight from kaggle Upload weight to Android: adb push gemma-2b-it-gpu-int8.bin /data/local/tmp Clone the repo, go to folder flutter-mediapipe/packages/mediapipe-task-genai/example and run flutter run -d 9TAUH6MRNZJ7KN6H --dart-define=GEMMA_8B_GPU_URI=/data/local/tmp/gemma-2b-it-gpu-int8.bin, in which 9TAUH6MRNZJ7KN6H is my Android ID. On Android screen, click Gemma 8b GPU and then input Hello, world! and click send button;

error : E/native ( 9752): E0000 00:00:1716588635.537158 9824 llm_engine.cc:68] NOT_FOUND: ValidatedGraphConfig Initialization failed. See detailed output below

Other info / Complete Logs

I/example.example( 9752): This is non sticky GC, maxfree is 33554432 minfree is 8388608 I/native ( 9752): I0000 00:00:1716588635.534848 9824 llm_inference_engine.cc:186] Session config: backend: 2, threads: 4, num_output_candidates: 1, topk: 40, temperature: 0.8, max_tokens: 1024, num_decode_steps_per_sync: 3, enable_fast_tuning: 1, enable_winograd_opt: 0, sequence_batch_size: 20, activation_data_type: 1, fake_weights_mode: 0, benchmark_info.input_token_limit: 0, benchmark_info.wait_for_input_processing: 0, Model type: 6 E/native ( 9752): E0000 00:00:1716588635.537158 9824 llm_engine.cc:68] NOT_FOUND: ValidatedGraphConfig Initialization failed. E/native ( 9752): No registered object with name: TokenizerCalculator; Unable to find Calculator "TokenizerCalculator" E/native ( 9752): No registered object with name: DetokenizerCalculator; Unable to find Calculator "DetokenizerCalculator" E/native ( 9752): No registered object with name: LlmGpuCalculator; Unable to find Calculator "LlmGpuCalculator" E/native ( 9752): No registered object with name: TokenCostCalculator; Unable to find Calculator "TokenCostCalculator" E/native ( 9752): No registered object with name: ModelDataCalculator; Unable to find Calculator "ModelDataCalculator" F/native ( 9752): F0000 00:00:1716588635.537238 9824 llm_engine.cc:98] Check failed: graph_->ObserveOutputStream( "result", [&, init_start_time, first_token_observed = false]( const mediapipe::Packet& p) mutable { const auto& tokens = p.Get<std::vectorstd::string>(); if (!tokens.empty() && !first_token_observed) { first_token_observed = true; benchmark_result_.time_to_first_token = absl::Now() - init_start_time; } timestamp_++; for (int i = 0; i < tokens.size(); ++i) { outputs_[i].response_tokens.push_back(tokens[i]); } if (async_callback_) { auto step_outputs = std::vector<Response>(tokens.size()); for (int i = 0; i < tokens.size(); ++i) { step_outputs[i].response_tokens.push_back(tokens[i]); } async_callback_(step_outputs); } return absl::OkStatus(); }) is OK (INTERNAL: ) F/native ( 9752): terminating. F/native ( 9752): F0000 00:00:1716588635.537238 9824 llm_engine.cc:98] Check failed: graph_->ObserveOutputStream( "result", [&, init_start_time, first_token_observed = false]( const mediapipe::Packet& p) mutable { const auto& tokens = p.Get<std::vectorstd::string>(); if (!tokens.empty() && !first_token_observed) { first_token_observed = true; benchmark_result_.time_to_first_token = absl::Now() - init_start_time; } timestamp_++; for (int i = 0; i < tokens.size(); ++i) { outputs_[i].response_tokens.push_back(tokens[i]); } if (async_callback_) { auto step_outputs = std::vector<Response>(tokens.size()); for (int i = 0; i < tokens.size(); ++i) { step_outputs[i].response_tokens.push_back(tokens[i]); } async_callback_(step_outputs); } return absl::OkStatus(); }) is OK (INTERNAL: ) F/native ( 9752): terminating. F/libc ( 9752): Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 9824 (DartWorker), pid 9752 (example.example)

---

Build fingerprint: 'Redmi/corot/corot:13/TP1A.220624.014/V14.0.14.0.TMLCNXM:user/release-keys' Revision: '0' ABI: 'arm64' Timestamp: 2024-05-25 00:10:35.700959586+0200 Process uptime: 57s ZygotePid: 1662649373 Cmdline: com.example.example pid: 9752, tid: 9824, name: DartWorker >>> com.example.example <<< uid: 10497 tagged_addr_ctrl: 0000000000000001 (PR_TAGGED_ADDR_ENABLE) pac_enabled_keys: 000000000000000f (PR_PAC_APIAKEY, PR_PAC_APIBKEY, PR_PAC_APDAKEY, PR_PAC_APDBKEY) signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr -------- Abort message: 'F0000 00:00:1716588635.537238 9824 llm_engine.cc:98] Check failed: graph_->ObserveOutputStream( "result", [&, init_start_time, first_token_observed = false]( const mediapipe::Packet& p) mutable { const auto& tokens = p.Get<std::vectorstd::string>(); if (!tokens.empty() && !first_token_observed) { first_token_observed = true; benchmark_result_.time_to_first_token = absl::Now() - init_start_time; } timestamp_++; for (int i = 0; i < tokens.size(); ++i) { outputs_[i].response_tokens.push_back(tokens[i]); } if (async_callback_) { auto step_outputs = std::vector<Response>(tokens.size()); for (int i = 0; i < tokens.size(); ++i) { step_outputs[i].response_tokens.push_back(tokens[i]); } async_callback_(step_outputs); } return absl::OkStatus(); }) is OK (INTERNAL: ) ' x0 0000000000000000 x1 0000000000002660 x2 0000000000000006 x3 00000078afb73610 x4 0000000100000006 x5 0000000100000006 x6 0000000100000006 x7 0000000100000006 x8 00000000000000f0 x9 000000794a82ebf0 x10 0000000000000001 x11 000000794a8a3bd0 x12 0000000000000001 x13 00000000ffffffff x14 0000000000000001 x15 0000000000000001 x16 000000794a913a68 x17 000000794a8ec710 x18 0000007811eba000 x19 0000000000002618 x20 0000000000002660 x21 00000000ffffffff x22 0000007700008081 x23 0000007708986409 x24 0000007700008081 x25 00000078afa93000 x26 b4000078af380e00 x27 0000007708a9e3d0 x28 0000000800000077 x29 00000078afb73690 lr 000000794a893e08 sp 00000078afb735f0 pc 000000794a893e34 pst 0000000000001000 backtrace: #00 pc 000000000008ce34 /apex/com.android.runtime/lib64/bionic/libc.so (abort+164) (BuildId: 95768e541c682c20709fc11df6431f16) #01 pc 0000000000f636c8 /data/app/~~65CDM0Omz3LuIQ1W7f9Z9w==/com.example.example-TM_1NsWO-1ke7EEEzBqEig==/lib/arm64/arm64_libllm_inference_engine.so (absl::log_internal::LogMessage::FailWithoutStackTrace()+100) #02 pc 0000000000f64028 /data/app/~~65CDM0Omz3LuIQ1W7f9Z9w==/com.example.example-TM_1NsWO-1ke7EEEzBqEig==/lib/arm64/arm64_libllm_inference_engine.so (absl::log_internal::LogMessage::Die()+60) #03 pc 0000000000f639a4 /data/app/~~65CDM0Omz3LuIQ1W7f9Z9w==/com.example.example-TM_1NsWO-1ke7EEEzBqEig==/lib/arm64/arm64_libllm_inference_engine.so (absl::log_internal::LogMessage::SendToLog()+176) #04 pc 0000000000f62edc /data/app/~~65CDM0Omz3LuIQ1W7f9Z9w==/com.example.example-TM_1NsWO-1ke7EEEzBqEig==/lib/arm64/arm64_libllm_inference_engine.so (absl::log_internal::LogMessage::Flush()+540) #05 pc 0000000000f64174 /data/app/~~65CDM0Omz3LuIQ1W7f9Z9w==/com.example.example-TM_1NsWO-1ke7EEEzBqEig==/lib/arm64/arm64_libllm_inference_engine.so (absl::log_internal::LogMessageFatal::~LogMessageFatal()+28) #06 pc 000000000080823c /data/app/~~65CDM0Omz3LuIQ1W7f9Z9w==/com.example.example-TM_1NsWO-1ke7EEEzBqEig==/lib/arm64/arm64_libllm_inference_engine.so (odml::infra::LlmInferenceEngine::Session::Session(odml::infra::proto::SessionConfig, mediapipe::CalculatorGraphConfig*)+1112) #07 pc 0000000000814438 /data/app/~~65CDM0Omz3LuIQ1W7f9Z9w==/com.example.example-TM_1NsWO-1ke7EEEzBqEig==/lib/arm64/arm64_libllm_inference_engine.so (std::__ndk1::__unique_ifodml::infra::LlmInferenceEngine::Session::__unique_single std::__ndk1::make_unique<odml::infra::LlmInferenceEngine::Session, odml::infra::proto::SessionConfig const&, mediapipe::CalculatorGraphConfig*>(odml::infra::proto::SessionConfig const&, mediapipe::CalculatorGraphConfig*&&)+128) #08 pc 000000000080ad98 /data/app/~~65CDM0Omz3LuIQ1W7f9Z9w==/com.example.example-TM_1NsWO-1ke7EEEzBqEig==/lib/arm64/arm64_libllm_inference_engine.so (odml::infra::LlmInferenceEngine::CreateSession(odml::infra::proto::SessionConfig const&, std::__ndk1::optionalodml::infra::proto::LlmModelType)+348) #09 pc 00000000007f51ec /data/app/~~65CDM0Omz3LuIQ1W7f9Z9w==/com.example.example-TM_1NsWO-1ke7EEEzBqEig==/lib/arm64/arm64_libllm_inference_engine.so (LlmInferenceEngine_CreateSession+3824) #10 pc 0000000000007f84 [anon:dart-code] Lost connection to device.

https://github.com/google/flutter-mediapipe/issues/56

[kuaashish]

Hi @tempo-riz,

Based on this comment, it appears the issue is on our end. A relevant fix has been provided and is awaiting approval. I will highlight this issue internally and keep you updated on any progress.

Thank you!!

[kuaashish]

Hi @tempo-riz,

Currently, we do not have information about this fix. However, we have marked it as a bug and sharing it with our team, so it is on our radar. We will fix it as soon as possible, but we cannot provide a timeline at this moment.

Thank you!!

[tempo-riz]

hey @kuaashish any update on this ?

[schmidt-sebastian]

Hello, this issue may have been caused by one of our older build configurations. Can you let us know if this is still a problem? If so, we will have to look at your setup. The missing components should be part of the .so file that is loaded during initialzation.

Please reply to this issue or open a new issue if you are still facing problems.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No

[ndnam198]

It's still an issue today and it's happening to me, I have the same setup as the thread owner. Please help

[jawad111]

workaround over here

https://github.com/google/flutter-mediapipe/issues/56#issuecomment-2603938767