I think you missed skip connection

[jugol]

May be I might be wrong, however I found that your MBConv block dosen't have skip connection.

In your model it's

MBConv

compare to

https://github.com/lukemelas/EfficientNet-PyTorch/blob/master/efficientnet_pytorch/model.py

MBConvBlock class

Or maybe not :(