[Feature] How to Apply SAHI to training?

[thiagoribeirodamotta]

What is the feature?

In order to effectively use SAHI today, I have to train a new model based on the patches that SAHI creates and the only way I've found out to accomplish this is by running SAHI as an offline pre-processing step, which will proceed to create several new copies of each annotation file or image. Since SAHI can be configured to use different patch sizes or patch intersections, this can easily occupy several GB for each configuration tried, which doesn't look optimal.

To deal with the aforementioned issue, I'd like to automatically apply SAHI on the fly during the training, validation and test phases, thus not needing to save to disk any new image or annotation.

Including SAHI as a pre-processing/post-processing step could greatly aid training!

Any other context? As far as I could figure it out, there are two inserctions where SAHI is needed: For pre-processing before training and for post-processing, either right before loss calculation and/or right after the inference on Validation and Test Steps.

So I've been trying to dig where should those inserctions happen.

For the post process, is it at ValLoop and TestLoop run_iter method? Is it right after the after_val_iter hook? Is it at run right after the run_iter loop ends? How would I recover which images belong together so they can be stiched up back again? In any case, would I need a custom Runner to run SAHI?

For the pre-process, is it at data_preprocessor? If so, would I need to reimplement it for every single architecture (currently focusing on YoloV7/YoloV8) or is there a more generic way to apply it? How about right before run_iter on the IterBasedTrainLoop.run method?

[mchaniotakis]

Wouldn't just a crop augmentation achieve what you need for the training step?

[thiagoribeirodamotta]

To be honest, that's what I ended up going for, using random crop as a pre-transform, right after loading images and annotations.

But it feels a bit off to just use random crop for that, as it will only return a single crop from a full image where using SAHI could potentially result in dozens of crops from a full image, meaning it would "increase" the dataset size drastically instead of just transforming it.

[whittenator]

To be honest, that's what I ended up going for, using random crop as a pre-transform, right after loading images and annotations.

But it feels a bit off to just use random crop for that, as it will only return a single crop from a full image where using SAHI could potentially result in dozens of crops from a full image, meaning it would "increase" the dataset size drastically instead of just transforming it.

Did you ever figure this out? When I use SAHI with my model I get a lot of false positives. Which leads me to believe that I need to train my model to expect SAHI slices?

[thiagoribeirodamotta]

At the end of the day I didn't figure out how to perform SAHI on an online manner.. instead I created the SAHI slices offline and used that to train my model.