coralnet
coralnet copied to clipboard
beijbom
Feature Request: Meta's SAM
As a baseline example, you could use SAM to generate masks using the segment everything script, and given the existing ground-truth annotations, provide the most frequently occurring class label of points that land within a given mask.
Alternatively, you could also try to create masks for each individual point, but I've found that the model is better at segmenting things rather than stuff. Therefore, you could also try to approach the task as instance segmentation instead of semantic segmentation.
I did just hear about SAM as well; seems to have gotten some traction!
Since we don't support segmentation workflows / data formats in any form yet (see issue #193), there'd be some baseline work to do before we could utilize such an engine well. But this could be a motivating springboard into doing that baseline work sooner rather than later.
Also, I don't know how novel or accurate the SAM algorithms themselves are relative to other segmentation engines, particularly against engines which specialize in benthic imagery (TagLab is one), so ideally we'd test at least two such engines as we start to explore segmentation.
I also mentioned this feature request to TagLab as an addition or replacement to their DeepCut algorithm (though I'll admit I'm not sure if their implementation was fine-tuned on coral data, which might have an advantage over SAM). But, preliminary results on coral reef orthomosaics made in Metashape were fairly good. I'm currently looking at using it as a precursor to TagLab to help create polygons around coral that can be imported to TagLab as a shapefile to reduce manual annotation for our team.