tflite-support
tflite-support copied to clipboard
tensorflow
Android: RGB bitmap from YUV image
v0.2.0 introduced support for YUV android.media.Image as part of the tensorflow-lite-task-vision artifact. Seems like it uses libyuv under the hood. As far as I understand TensorImage delegates to MediaImageContainer in this case. At the moment of writing it doesn’t support the getBitmap method, throwing an exception.
We have such use case:
- take a YUV image from camera;
- convert and rotate the image to a RGB
Bitmapvia RenderScript; - pass the
Bitmapto TF; - save the
Bitmapas a JPEG file alongside detections done by TF.
We don’t grab an RGB bitmap from camera directly since it takes much more time than grabbing a YUV image and processing it via RenderScript. However, the Android 12 tooling deprecates RenderScript.
It would be great if the MediaImageContainer.getBitmap method returned a valid Bitmap. I imagine in this scenario our use case might avoid using RenderScript. As far as I understood, the byte buffer for rotated and converted image exists during the TF processing already, it just needs to be shifted to a Bitmap on demand.
If this is not what TF wants to support — can I ask for pointers on best practices regarding something like this? Meaning reading the camera frame, processing it via TF and then saving the frame.
Converting to RGB and image rotation are supported in Task library through ImageProcessingOptions, such as ImageClassifier#classify(TensorImage, ImageProcessingOptions). You don't need to convert it to Bitmap. We're working on publish examples and tutorials to introduce consuming YUV images directly.
Surfacing YUV image conversion to RGB in TensorImage is not prioritized at this moment. We have the C++ implementation, but haven't wrapped it in Java yet. In C++, image processing is done through FrameBuffer. See the code here for an example of how to.
@lu-wang-g, thanks for details!
I understand that the Bitmap becomes kind of redundant but we still need it (for our use case) to save the image as a JPEG file. Maybe it can be done on the native level as well though. We don’t save all images however since not everything is useful. Not sure how popular this scenario is to do the investment but I imagine that saving model-enriched images after their processing is used by someone besides us 😉
@arturdryomov I agree with that saving images could be very useful, at least for debugging purposes I can imagine.
If adding implementation to getBitmap on MediaImageContainer is all your need, and not very concern about performance, we can add an implementation there (as far as I know, our C++ YUV library is really fast, Java implementation might not be that efficient), but we can not promise a date due to workload.
If you're interested, any contribution is really welcomed!
@xunkai55, thanks, this might be useful indeed. In the meanwhile I guess we might want to explore using FrameBuffer API instead 🤔
@lu-wang-g, hey, is there a reason for this issue to be closed? Seems like a valid use case (not of a top priority of course but still).
Ha, I thought the issue was walked around by using FrameBuffer. Let me reopen it to track.
@lu-wang-g, unfortunately we weren’t able to use FrameBuffer for this since we didn’t want to pull this repo to get access to header files. We did something with libyuv which kinda works but it would be useful to have this built-in to avoid dealing with NDK / JNI.
BTW, as a side-note — have you tried using Vulkan for the YUV → RGB conversion? Would be interesting to know if there are benefits.
Got it. Supporting YUV conversion utils is in our roadmap. I'll give updates here once we have any progresses.
So far, we only touched image processing on CPU. We have some undergoing projects related to GPU computation, but they are still in an early stage.
We had to shift our roadmap on image utils soon after I left my previous comments last year. We won't be able to deliver this feature in the near feature. Sorry about it.
@lu-wang-g, no worries, thanks for the update!
Perhaps I can ask for a consult. Do you have a suggestion regarding the best approach to pass images from camera to a model with a possible persistence to a JPEG / WebP file? Maybe some team at Google had a similar workflow. Our model consumes RGB bitmaps, we’ve used RenderScript for the YUV → RGB conversion so far. Unfortunately, RenderScript is deprecated (as a tool) — we’ve tried libyuv as a replacement but it’s a bit less performant. The trick is — we have to persist resulting images in the original resolution (4K) so we cannot request small-ish images from the camera. I guess one approach might be to switch a YUV-consuming model but it will require a retraining and it still might be less performant IRL.
Libyuv should be very performant as far as I know. We use it to perform image processing in Task Library as well. Alternatively, you can request RGBA output from CameraX directly. See this blogpost: https://medium.com/androiddevelopers/convert-yuv-to-rgb-for-camerax-imageanalysis-6c627f3a0292.