DETR - Training with different channel sizes (e.g. 1 channel or 4 channels)

[bastlen]

Hello,

is it possible to train the model with only 1 or 4 channels?

Regards B

[nasheedyasin]

There are some tricks you can do in order to literally use an n-dimensional image with Detr or any other model pretrained with 3-channel images.

Before feeding your image into the model, have a 1x1 conv2D layer (Refer InceptionNet paper), for this layer, you can set the number of channels as 4. Keep this layer trainable. I hypothesize that you should be able to generalize well with this approach.

Basically Input -> 2dConv(1*1) with 3 channels -> pretrained backbone

@NielsRogge What do you think?