google-ai-edge
[FEATURE] Add Alternative GPU Delegate Support for ChromeOS (e.g., Vulkan, WebGPU)
Issue:
After installing the ai-edge-gallery.apk (Version: 1.0.3) on my Chromebook via ADB Sideload and after installing any suggested model like gemma and trying to use it I get this error:
Failed to initialize session: %sINTERNAL: RET_CHECK failure (third_party/odml/infra/genai/inference/calculators/llm_vision_inference_runner_cl.cc:62) interpreter_->Modify Graph With Delegate (std::move (delegate)) == kTfLiteOk (2 vs. 0)
If I hit "close" on the dialog and still try to prompt the prompt is stuck at processing (I doubt it happens but the animation is there).
My Hardware:
Lenovo Chromebook Plus gen 10 14 inch Mediatek:
-Model: Lenovo Chromebook 14 (Ultra 910-16GB-256GB)
-Specific Modelversion: 83MY000HGE
-Specific Processor: Mediatek Kompanio 910 Ultra (ARM)
My Environment (german) :
Google Chrome: 135.0.7049.104 (Offizieller Build) (64-Bit) Revision: a93cb0ba09e15e80adb484aa937ec7c622e363b3-refs/branch-heads/7049@{#1898} Plattform: 16209.50.0 (Official Build) stable-channel rauru
Firmwareversion: Google_Navi.16174.69.0 Personalisierungs-ID: navi ARC 13328662 SDK-Version: 33 JavaScript: V8 13.5.212.10 User-Agent: Mozilla/5.0 (X11; CrOS x86_64 14541.0.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36
Active CMD settings are found at the bottom because they are too long.
Active ChromeOS Commandline parameters:
Befehlszeile: /opt/google/chrome/chrome --enable-native-gpu-memory-buffers --enable-webgl-image-chromium --enable-features=Pepper3DImageChromium,Vulkan,DefaultANGLEVulkan,VulkanFromANGLE,AvoidDuplicateDelayBeginFrame,UmaStorageDimensions,QuickUnlockFingerprint,AllowAmbientEQ,OledScaleFactorEnabled,ArcAdbSideloading,ArcEnableVirtioBlkForData,Crostini,SmartDim,OnDeviceSpeechRecognition,CameraEffectsSupportedByHardware,CameraSuperResSupported,ICASupportedByHardware,FeatureManagement16Desks,FeatureManagementBorealis,FeatureManagementConchGenAi,FeatureManagementCrosSodaConchLanguages,FeatureManagementDriveFsBulkPinning,FeatureManagementFeatureAwareDeviceDemoMode,FeatureManagementGameDashboardRecordGame,FeatureManagementGeminiAppPreinstall,FeatureManagementGrowthFramework,FeatureManagementHistoryEmbedding,FeatureManagementLiveTranslateCrOS,FeatureManagementLobster,FeatureManagementLocalImageSearch,FeatureManagementMahi,FeatureManagementMarkupPod,FeatureManagementOobeAiIntro,FeatureManagementOobeGeminiIntro,FeatureManagementOobeSimon,FeatureManagementOrca,FeatureManagementRoundedWindows,FeatureManagementSeaPen,FeatureManagementShouldExcludeFromSysUiHoldback,FeatureManagementShowoff,FeatureManagementSystemLiveCaption,FeatureManagementTimeOfDayScreenSaver,FeatureManagementTimeOfDayWallpaper,FeatureManagementVideoConference,MediaAppImageMantis,CrosSafetyService,ConchLargeModel,CoralFeature --gpu-sandbox-failures-fatal=yes --gpu-sandbox-start-early --video-capture-use-gpu-memory-buffer --enable-logging --log-level=1 --use-cras --enable-wayland-server --user-data-dir=/home/chronos --arc-build-properties={'device': 'rauru_cheets', 'firstApiLevel': '28'} --child-wallpaper-large=/usr/share/chromeos-assets/wallpaper/child_large.jpg --child-wallpaper-small=/usr/share/chromeos-assets/wallpaper/child_small.jpg --default-wallpaper-large=/usr/share/chromeos-assets/wallpaper/default_large.jpg --default-wallpaper-small=/usr/share/chromeos-assets/wallpaper/default_small.jpg --display-properties=[{"connector-type": 14, "rounded-corners": {"bottom-left": 13, "bottom-right": 13, "top-left": 13, "top-right": 13}}] --fingerprint-sensor-location=keyboard-bottom-right --guest-wallpaper-large=/usr/share/chromeos-assets/wallpaper/guest_large.jpg --guest-wallpaper-small=/usr/share/chromeos-assets/wallpaper/guest_small.jpg --regulatory-label-dir=navi --disable-mojo-broker --ash-use-cros-mojo-service-manager --cros-healthd-uses-service-manager --scheduler-boost-urgent=70 --login-profile=user --has-chromeos-keyboard --form-factor=CLAMSHELL --guest-wallpaper-large=/usr/share/chromeos-assets/wallpaper/guest_large.jpg --guest-wallpaper-small=/usr/share/chromeos-assets/wallpaper/guest_small.jpg --child-wallpaper-large=/usr/share/chromeos-assets/wallpaper/child_large.jpg --child-wallpaper-small=/usr/share/chromeos-assets/wallpaper/child_small.jpg --default-wallpaper-large=/usr/share/chromeos-assets/wallpaper/default_large.jpg --default-wallpaper-small=/usr/share/chromeos-assets/wallpaper/default_small.jpg --device-help-content-id=navi --regulatory-label-dir=navi --fingerprint-sensor-location=keyboard-bottom-right --ozone-nnpalm-properties={} --arc-availability=officially-supported --enable-arcvm --arc-erofs --enterprise-enrollment-initial-modulus=15 --enterprise-enrollment-modulus-limit=19 --ml_service=enabled --ondevice_handwriting=use_rootfs --ondevice_document_scanner=use_rootfs --camera-effects-supported-by-hardware --feature-management-level=2 --feature-management-max-level=2 --feature-management-scope=1 --login-manager --feature-flags=["crostini-gpu-support@1","crostini-multi-container@1"] --first-exec-after-boot --enable-features=AllowAmbientEQ,ArcAdbSideloading,ArcEnableVirtioBlkForData,AvoidDuplicateDelayBeginFrame,CameraEffectsSupportedByHardware,CameraSuperResSupported,ConchLargeModel,CoralFeature,CrosSafetyService,Crostini,DefaultANGLEVulkan,FeatureManagement16Desks,FeatureManagementBorealis,FeatureManagementConchGenAi,FeatureManagementCrosSodaConchLanguages,FeatureManagementDriveFsBulkPinning,FeatureManagementFeatureAwareDeviceDemoMode,FeatureManagementGameDashboardRecordGame,FeatureManagementGeminiAppPreinstall,FeatureManagementGrowthFramework,FeatureManagementHistoryEmbedding,FeatureManagementLiveTranslateCrOS,FeatureManagementLobster,FeatureManagementLocalImageSearch,FeatureManagementMahi,FeatureManagementMarkupPod,FeatureManagementOobeAiIntro,FeatureManagementOobeGeminiIntro,FeatureManagementOobeSimon,FeatureManagementOrca,FeatureManagementRoundedWindows,FeatureManagementSeaPen,FeatureManagementShouldExcludeFromSysUiHoldback,FeatureManagementShowoff,FeatureManagementSystemLiveCaption,FeatureManagementTimeOfDayScreenSaver,FeatureManagementTimeOfDayWallpaper,FeatureManagementVideoConference,ICASupportedByHardware,MediaAppImageMantis,OledScaleFactorEnabled,OnDeviceSpeechRecognition,Pepper3DImageChromium,QuickUnlockFingerprint,SmartDim,UmaStorageDimensions,Vulkan,VulkanFromANGLE --flag-switches-begin --enable-features=AllowAmbientEQ,ArcAdbSideloading,ArcEnableVirtioBlkForData,AvoidDuplicateDelayBeginFrame,CameraEffectsSupportedByHardware,CameraSuperResSupported,ConchLargeModel,CoralFeature,CrosSafetyService,Crostini,DefaultANGLEVulkan,FeatureManagement16Desks,FeatureManagementBorealis,FeatureManagementConchGenAi,FeatureManagementCrosSodaConchLanguages,FeatureManagementDriveFsBulkPinning,FeatureManagementFeatureAwareDeviceDemoMode,FeatureManagementGameDashboardRecordGame,FeatureManagementGeminiAppPreinstall,FeatureManagementGrowthFramework,FeatureManagementHistoryEmbedding,FeatureManagementLiveTranslateCrOS,FeatureManagementLobster,FeatureManagementLocalImageSearch,FeatureManagementMahi,FeatureManagementMarkupPod,FeatureManagementOobeAiIntro,FeatureManagementOobeGeminiIntro,FeatureManagementOobeSimon,FeatureManagementOrca,FeatureManagementRoundedWindows,FeatureManagementSeaPen,FeatureManagementShouldExcludeFromSysUiHoldback,FeatureManagementShowoff,FeatureManagementSystemLiveCaption,FeatureManagementTimeOfDayScreenSaver,FeatureManagementTimeOfDayWallpaper,FeatureManagementVideoConference,ICASupportedByHardware,MediaAppImageMantis,OledScaleFactorEnabled,OnDeviceSpeechRecognition,Pepper3DImageChromium,QuickUnlockFingerprint,SmartDim,UmaStorageDimensions,Vulkan,VulkanFromANGLE,CrostiniGpuSupport,CrostiniMultiContainer --flag-switches-end
AI generated breakdown: Let's break down the error message:
-
Failed to initialize session: This is the top-level error, meaning the application couldn't successfully set up the necessary environment to run the AI model.
-
INTERNAL: RET_CHECK failure: This is a low-level, internal check within the application's code that failed. RET_CHECK is a macro often used in Google's codebases to ensure that a condition is true. If it's not, the program halts with an error.
*third_party/odml/infra/genai/inference/calculators/llm_vision_inference_runner_cl.cc:62: This is the specific location in the source code where the error occurred. It points to a file related to running "LLM Vision Inference" (Large Language Model for vision tasks) and suggests it's part of the "ODML" (On-Device Machine Learning) infrastructure. The cl.cc suggests it might be related to C++ code for a "calculator," a component often found in ML pipelines.
-
interpreter_->Modify Graph With Delegate (std::move (delegate)) == kTfLiteOk (2 vs. 0): This is the most crucial part of the error.
-
interpreter_->Modify Graph With Delegate: This is a function call to modify the AI model's graph (its internal structure) using a "delegate." A delegate is a component that offloads parts of the model's computation to a specific hardware accelerator. In this case, the delegate is likely an optimization for the ARM processor's neural processing unit (NPU) or GPU.
-
== kTfLiteOk: This is the expected result of the function call. kTfLiteOk means "TensorFlow Lite OK," which is the success code (often represented as 0).
-
(2 vs. 0): This shows the actual result of the function call. The function returned 2, but it was expected to return 0. In TensorFlow Lite, a return value of 2 often corresponds to kTfLiteError, indicating a generic error. In simple terms, the error means: The application tried to use a special hardware acceleration component (a "delegate") to run the AI model more efficiently on your ARM Chromebook. However, this process failed. The reason for the failure is likely one of the following:
-
-
Incompatible Delegate: The delegate might not be compatible with your specific Chromebook's hardware or its drivers. The hardware accelerator might not be properly supported or its drivers might be outdated or missing.
-
Model/Delegate Mismatch: There could be a mismatch between the AI model and the delegate. The model might have been built to work with a different type of hardware accelerator, or it might not be compatible with the delegate's features.
-
Software Bug: There could be a bug in the AI Edge Gallery APK itself that is causing this failure on ARM Chromebooks.
Hi @adamhussein, Thank you for this incredibly detailed bug report and for providing the thorough error analysis. This information is extremely helpful for our team. We have logged this ChromeOS/ARM issue and we appreciate you taking the time to report this. We will definitely reach out if we need more information. Thanks!
@dpknag thank you. I bought this chromebook as my second development machine, and specifically for using the NPU for small on-device models.
So if there is anything I can do to speed-up the process please do reach out.
My vague memory tells me that ARCVM (ChromeOS Android) does not support OpenCL. Is it possible for ai-edge-gallery to support other backends, e.g. OpenGL, when OpenCL is unavailable?
@oToToT thanks that speeds it up indeed. I was not aware of this, so its highly unlikely that this will work if the ai edge gallery doesnt add support for different API.(It is unlikely that google will add openCL support to ARCVM I think).
@dpknag so it changed more to a feature request for different API support? Can I keep the issue open or should I open a new issue?
The following is an unchecked stupid AI breakdown of possible ways:
The TensorFlow Lite Delegates The AI Edge Gallery app uses TensorFlow Lite (TFLite) to run its LLMs.
TFLite is designed with a modular architecture that allows it to use "delegates" to offload computations to different hardware.
OpenCL Delegate: The current version of the app is attempting to use a TFLite GPU delegate that relies on the OpenCL API.
This is why you see ...llm_vision_inference_runner_cl.cc... in the error message.
Vulkan Delegate: For the app to work on your Chromebook's ARM architecture, the developers would need to modify the code to use a TFLite GPU delegate that relies on the Vulkan API.
This is not a simple find-and-replace. OpenCL and Vulkan are two different APIs with different programming models and concepts. The code that's written to talk to OpenCL would have to be re-written to talk to Vulkan.
3. The NPU and the Vulkan Delegate
You mentioned wanting to use the local NPU (Neural Processing Unit) on your ARM Chromebook. This adds another layer of complexity.
Vulkan for GPUs: The Vulkan API is primarily a graphics and compute API for GPUs. While you can use it for general-purpose computing, it doesn't directly interface with a dedicated NPU.
NNAPI for NPUs: On Android, the standard way for apps to access a dedicated NPU is through the Android Neural Networks API (NNAPI).
This is a higher-level API that allows developers to run their models on the best available hardware, including GPUs and NPUs, without having to write device-specific code.
So, for the developers of the AI Edge Gallery app to fully leverage your hardware, they would ideally need to:
a. Modify their app to use the TFLite NNAPI delegate. b. Build the app with Vulkan support for the GPU delegate as a fallback, as ARCVM explicitly supports Vulkan.
Hi @adamhussein and @oToToT,
Thank you for the detailed analysis and for identifying the root cause. This is very helpful for the team.
You are correct, this has evolved into a feature request. In that case, we will keep this issue open to preserve the context. I have retitled it to [FEATURE] Add Vulkan/NNAPI Delegate Support for ChromeOS and will update the labels accordingly.
We have passed this feature request on to the team. Thanks again for your contribution.
I think the Vulkan/NNAPI title is inaccurate.
NNAPI is for NPU and is already depreciated.
Here the FR is for GPU, and the viable options are OpenCL/WebGPU to my knowledge.
Additionally, this is not running directly on ChromeOS. It's running on an Android VM in ChromeOS, which is still Android. Thus, this is an issue about running ai-edge-gallery in an Android environment that does not have (full)OpenCL support.
@oToToT thanks again I checked your remark and NNAPI is indeed marked as depreciated
-However for my specific error: This issue aims at making the delegate compatible with Chromebooks ARCVM and utilizing possible available NPUs and GPUs.
So according to my (not so knowledgeable opinion)
Options on Chromeos are :
-Vulkan
- (AI recommendation needs to be checked): TensorFlow Lite in Google Play Services, and for hardware acceleration, the TFLite GPU delegate
Again AI Quicksearch (needs to be properly verified):
Is NNAPI deprecated? Yes, the NDK API for NNAPI has been deprecated as of Android 15. Google's documentation explicitly states, "Starting in Android 15, the NNAPI (NDK API) is deprecated."
What does this mean for developers? For developers using TensorFlow Lite, the recommended path is now to use TensorFlow Lite in Google Play Services and, for hardware acceleration, the TFLite GPU delegate.
The deprecation of the NDK API for NNAPI is meant to provide a clearer and more standardized path for machine learning on Android.
Is NNAPI gone entirely? No. The underlying Neural Networks HAL (Hardware Abstraction Layer) interface is still supported.
This is the low-level part that hardware manufacturers (like MediaTek or Qualcomm) use to provide drivers for their NPUs, DSPs, and other accelerators.
The change is about the API that app developers use to interact with it.
Instead of directly calling the NNAPI NDK, they are now encouraged to use the higher-level TFLite in Google Play Services.
@dpknag Here are some none AI Ressources I found for the NNAPI replacement:
-Announcement/Migrationguide: developer.android.com/ndk/guides/neuralnetworks/migration-guide
Resources from the guide:
- ai.google.dev/edge/litert/android/gpu
- ai.google.dev/edge/litert/android/play_services
@oToToT thanks for your continious input. I encourage you to keep correcting me, I really appreciate your comments they are very helpful.
From other people's error log https://github.com/google-ai-edge/gallery/issues/171, it is clear that ai-edge-gallery is using MLDrift for its high-level GPU implementation. And MLDrift only has WebGPU and OpenCL backend. https://arxiv.org/pdf/2505.00232 (Maybe they have more, but it's not public, so not sure)
Hi @adamhussein and @oToToT,
Apologies for the delay in response.
Thank you for the incredibly valuable follow-up discussion. Your research into the NNAPI NDK deprecation, the recommended migration to TensorFlow Lite in Google Play Services, and the potential underlying libraries has provided critical clarity for our team.
This is an exemplary case of community collaboration, and we genuinely appreciate the effort you've put into diagnosing the issue and mapping out the solution space. We have added this new information to the internal feature request to ensure our engineering team has the full context.
To accurately reflect this new information, I am updating the title of this issue. We will keep this thread open for any future updates.
Thank you again for your patience and your fantastic contributions.