Get image coordinates of mouse click on Gradio image.

[vvvm23]

• [X] I have searched to see if a similar issue already exists.

Is your feature request related to a problem? Please describe.
I'd like to be able to select different segmentation masks by clicking on objects in an image. Currently I use CheckboxGroup which works, but is a little awkward to use.

Describe the solution you'd like I would like to be able to get image coordinates and store it in a python variable (or have it as an auxiliary input to a callback function) when an image is clicked, so I can select objects in an image. I am unsure what the best API is for this as it doesn't really fit the standard format for a callback function, but I am open to helping with discussion.

Additional context

This request came from discussion from my discord message:

I'd like when I click on a Gradio image to be able to obtain the image coordinates of where I clicked, and have this accessible from the python code. Ideally, this would also call a callback function after passing the coordinates to the python program. Something like:

def click_callback(coords: Tuple[int, int]):
    ...

click_coords = gr.State()
image = gr.Image()

image.click(click_callback, inputs=click_coords, outputs=...)

Is this possible in any way?

Thanks!

In this demo I am working on, I want to be able to toggle masks by clicking an object in the image, rather than using the CheckboxGroup. Adding this feature would allow me to do this. The demo is shown below.

[pngwn]

This doesn't make sense with the current API, we need some way of passing meta information about the event to the callback function in addition to the component inputs.

[vvvm23]

That's a shame, is there any way I could hack this in outside the intended API?

[jtsoi]

@pngwn How about storing the click_coords as properties on the image passed to the callback function?

This would solve our issue of "click the correct bounding-box" for classification of bounding boxes.

[TommyTunaToro]

+1 Really needed this feature to work with segment_anything from Meta

[aliabid94]

on it!

[aliabid94]

Btw, we do support meta information about events by passing an EventData object type hinted into the function args, see https://gradio.app/docs/#eventdata. Had been planning on adding this to images, will work on it today.

[vvvm23]

Nice work! 🔥