Processing of bounding boxes by image_processing

[robmarkcole]

Currently we don't have a nice way to handle bounding box information returned by image_processing entities. My suggestion is that a nice solution would be to have a destination output for processed images, so that a second camera is created, showing the processed image with bounding boxes drawn on it. Alternatively we standardise the way that bounding boxes are returned (e.g. tensorflow and facebox have differing approaches currently) then use some other method to process those. Be good to get input of @hunterjm @dgomes

Duplicates this issue? https://github.com/home-assistant/architecture/issues/41

[arsaboo]

image_processing components should allow saving processed images with bounding boxes. Users are then free to use them with local_file camera or in an automation. In that sense, I am closer to the current implementation in Tensorflow component. It would be great if some of the features in the TF component can be abstracted so that all image_processing components can use them.

[robmarkcole]

@arsaboo I am using that basic approach here, with folder_watcher component to detect new saved files and display them on a local_file camera. It works but is a bit hacky.

I think the most straightforward approach is to refactor the tensorflow save image function to the platform level, and fire an image_saved event which can be used in automations

[arsaboo]

Indeed...save_image should be refactored to the platform level and an event (that can be used in automations) should be fired so that all platforms can use it.

[dgomes]

save_image is a terrible IPC. We should workout a better mechanism, else we are just killing SD cards in RPi's

I've previsouly proposed something https://github.com/home-assistant/architecture/issues/41

[robmarkcole]

Well we already let people save images, it is opt-in. I actually like your (scratched) suggestion in the other thread that the processed image (with bounding box) be displayed on a second 'sink' camera. The image then isn't saved but can be used in automations

[hunterjm]

That could be done without creating a camera dependency on the image processing component, but we could have an image processing camera component which takes the entity_id of the image_processing instance, and refactor the base image_processing component to store last_image in memory for each instance which would be the last processed image, with or without bounding boxes drawn.

save_image is still a handy feature IMO. Not everyone runs on a Pi, and the ability to save off snapshots is a handy feature. I agree that it should also be elevated, and potentially also become a service that utilizes last_image.

I'm also all for standardizing attributes and raising that up a level as well, but not every platform provides "bounding boxes", so there will need to be some thought put into it.

[arsaboo]

Indeed, Pi is not the "optimal" device for image_processing. So, we should not design with Pi as our focal device. In any case, save_image is optional and can stay that way.

Bounding boxes can be optional; components that don't provide bounding boxes can just save images.

[robmarkcole]

OK so we have a range of opinions, how to progress forward to action here?

[robmarkcole]

To update, I moved the bounding box save method to the image processing integration. I adopted the approach used by tensorflow. Now is a good time to make a decision about the sink camera idea, which I am in favour of as it will greatly simplify usage

[robmarkcole]

Update: saving of images to disk is just creating so many tedious issues, I am now leaning away from this approach. Much better to have another camera that displays the annotated image

[frenck]

This architecture issue is old, stale, and possibly obsolete. Things changed a lot over the years. Additionally, we have been moving to discussions for these architectural discussions.

For that reason, I'm going to close this issue.

../Frenck