Implement EfficientNet backbone

[daniel-j-h]

This recently came out

https://arxiv.org/abs/1905.11946

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

with incredible practical implications: we can either get a much better backbone than our current resnet50 for the same computational budget or get a highly computationally efficient backbone and roughly keep the resnet50's prediction performance.

Here's why we should look into this

• it's simple and easy to implement
• it provides huge practical engineering gains we care about (size, runtime)
• the accuracy is same or higher (depending on EfficientNet model)

Check out the paper's table 2 (below) for comparisons; we are currently using a resnet50.

I highlighted

• the resnet50 we are currently using
• the EfficientNet-B0 which reaches roughly the same accuracy as resnet50 but is x4.9 smaller and requires x11 less flops
• the EfficientNet-B4 which is a bit smaller than the resnet50, requires roughly the same amount of flops than the resnet50, but reaches a much higher accuracy

[daniel-j-h]

Took a couple more evenings than I hoped it would but now there is

https://github.com/daniel-j-h/efficientnet

Next actions:

• [x] finish impl (se blocks)
• [x] pre-train on imagenet (at least b0, b4)
• [ ] see how it behaves as a backbone for a segmentation model in robosat
• [ ] switch to a pre-trained efficientnet as a backbone

The EfficientNets generally use an order of magnitude fewer parameters and floating point operations per second compared to existing models with similar accuracy. If we can make this work for robosat we can expect either much better segmentation results or much smaller models (or both if we e.g. provide a B0 and a B4 backbone for our users).

cc @ocourtin might be interesting to you :)

[daniel-j-h]

Updating here. Making progress in https://github.com/daniel-j-h/efficientnet There are some open questions re. the implementation and tricks we can use on top of the paper, you can follow e.g.

• https://forums.fast.ai/t/efficientnet/46978/76
• https://forums.fast.ai/t/efficientnet/46978/86