Integrating Kornia batch augmentation pipeline.

[shijianjian]

Hi mmdet maintainers,

Thanks for the repo. We are the project lead of Kornia, and thinking of the possibilities of integrating Kornia augmentation pipeline into mmdet.

From the feature-level, we can provide:

• GPU/CPU/TPU-based batch data augmentation.
• More augmentation options, like PatchSequential.
• Pytorch-native deployable image processing using torch.jit / onnx.
• Differentiability, which has already been adopted in applications like in GANs.

Motivation: mmdet data pipeline happened outside the PyTorch computation graph and computed on CPUs in a image-by-image manner, which might increase the deployment difficulties and time legacy whilst processing large batches. With Kornia, it is much easier to include test time image processing pipelines into the model whilst export and deployment. Also, the image preprocessing could enjoy the GPU accerlations provided by PyTorch.

Our proposal:

• As demonstrated above, our whole pipeline happens differently than common CPU-based augmentations. Thus, we probably need to introduce a batch_pipeline module for such operations. Commonly:

kornia_aug = ...
for image, label in dataloaders:
    image = kornia_aug(image)
    logits = model(image)

The user interface shall be like:

model = dict(
    preprocessing=dict(
        dict(type="RandomHorizontalFlip", p=0.5),
        dict(type="ColorJitter", hue=0.1, saturation=0.3),
    ),
    backbone=dict(
        depth=101,
        init_cfg=dict(
            type='Pretrained',
            checkpoint='open-mmlab://detectron2/resnet101_caffe')))

• we might include the test-time data processing into the exported model as well.