DataPack function to Retrieve overlapped ImageAnnotation using data structure grid

[hepengfe]

Is your feature request related to a problem? Please describe. One of the main reasons for creating grid is to searching overlapped ImageAnnotation efficiently. However, the current code supports compute iou but not search overlapped ImageAnnotation.

Describe the solution you'd like

• store a grid in the DataPack
• store two dictionaries in the DataPack based on the grid
  • dictionary 1: key is grid cell position and value is a collection of image annotation presented at the grid cell
  • dictionary 2: key is the image annotation and value is a list of grid cell positions it's on
• implement a function that search overlapped ImageAnnnotation given a ImageAnnnotation using the two dictionaries

grid_cells = dict2[query_ia]
image_anns = set()
for gc in grid_cells:
     for ia in dict1[gc]:
        image_anns.add(ia)

out = []
for ia in  image_anns:
     if ia.overlap(query_ia):
        out.append(ia)

return ia

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context This is a follow-up issue on https://github.com/asyml/forte/pull/876#discussion_r925802414

[hunterhector]

Actually, this is not what we discussed, we don't want users to manually compute any IOU, they should be able to do something like "a_box.iou(a_circle)". So what we need here is not a tutorial.

1. We want to have a system interanlly to manage stuff like overlap/covering, we don't ask the users to write an algorithm for it. Please make a comparison to the text datapack system, where we can do something like sentence.get(Token). Most of the time the user don't have to interact with the Grid object, just like they don't have to interact with the Span object.
2. Please be careful and precise of the wording, we don't compute "overlapped images", we are computing overalp between ImageAnnotation

@mylibrar

[mylibrar]

Actually, this is not what we discussed, we don't want users to manually compute any IOU, they should be able to do something like "a_box.iou(a_circle)". So what we need here is not a tutorial.

1. We want to have a system interanlly to manage stuff like overlap/covering, we don't ask the users to write an algorithm for it. Please make a comparison to the text datapack system, where we can do something like sentence.get(Token). Most of the time the user don't have to interact with the Grid object, just like they don't have to interact with the Span object.
2. Please be careful and precise of the wording, we don't compute "overlapped images", we are computing overalp between ImageAnnotation

@mylibrar

Then maybe we should update the title and content of this issue. It should be targeted at implementing some internal algorithms for calculating overlap/covering between ImageAnnotations instead of creating a new tutorial/example.

[hunterhector]

Actually, this is not what we discussed, we don't want users to manually compute any IOU, they should be able to do something like "a_box.iou(a_circle)". So what we need here is not a tutorial.

1. We want to have a system interanlly to manage stuff like overlap/covering, we don't ask the users to write an algorithm for it. Please make a comparison to the text datapack system, where we can do something like sentence.get(Token). Most of the time the user don't have to interact with the Grid object, just like they don't have to interact with the Span object.
2. Please be careful and precise of the wording, we don't compute "overlapped images", we are computing overalp between ImageAnnotation

@mylibrar

Then maybe we should update the title and content of this issue. It should be targeted at implementing some internal algorithms for calculating overlap/covering between ImageAnnotations instead of creating a new tutorial/example.

The prioritization of this item can be determined according to other items. This item is a stretch goal for the last quarter. Our final goal is to provide computation among arbitrary image annotations, and the defined goal for last quarter is for regular shapes only, like https://github.com/asyml/forte/issues/802

[hepengfe]

I think currently our implementation can already compute overlapping between ImageAnnotation here https://github.com/asyml/forte/blob/8a61272fe2dd7d3f266f4d66a2f0d3e8f6fc3389/forte/data/ontology/top.py#L936.

For support like region.get(Box), we can build a grid in the DataPack and a dictionary stores image annotation presenting at each grid cell.

[hunterhector]

• Please be careful and precise of the wording, we don't compute "overlapped images", we are computing overalp between ImageAnnota

Yeah, that means we don't need the tutorial of how to do these for scratch, but it is valuable to show how to use these functions.

For simple get with regular shapes, we can do it easily. For irregular shapes, we then need to use Grid.

However, most of the supports for irregular shapes are not in our current scope, let's do them later. As for now, if operations involve those, we should throw some NotImplementedError.