Resnet overfitting

[shuuchen]

Hi there,

I tried to implement resnet and put it into Jupyter Notebook. https://github.com/shuuchen/pytorch_tutorials/blob/master/Deep%20Residual%20Network/drn.nbconvert.ipynb

However in the final gragh, the model got overfitting terribly, where the training loss getting nearly 0 but the testing loss getting higher and higher.

Anyone can help?

[fawazsammani]

As I can see from your code, you haven't normalized your images in the transforms. It happened once with me, then when I normalized them, overfitting was reduced. You need to find the mean and std deviation of your images, and then provide them to the data transforms. Hope that helps

[shuuchen]

@fawazsammani Thanks very much ! I used BatchNorm2d layers for normalization. Don' t they do what you talked about ?

[shuuchen]

@fawazsammani Hi, I tried the following code by add Normalize function to the transform composition. But it is still bad in evaluation loss. https://github.com/shuuchen/pytorch_tutorials/blob/master/Deep%20Residual%20Network/drn.nbconvert.ipynb

train_transform = transforms.Compose([transforms.Pad(4), 
                                transforms.RandomHorizontalFlip(), 
                                transforms.RandomCrop(32),
                                transforms.ToTensor(),
                                transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225))])

test_transform = transforms.Compose([transforms.ToTensor(), 
                               transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225))])

Can I see your code ?

[tooHotSpot]

@fawazsammani Thanks very much ! I used BatchNorm2d layers for normalization. Don' t they do what you talked about ?

Check BatchNorm parameter track_running_stats, it has to be True, by default it is False

[shuuchen]

@tooHotSpot

by default it is False

Actually, it is True by default. https://pytorch.org/docs/stable/nn.html#torch.nn.BatchNorm2d