Train a baseline classifier

[marco-c]

We need to find some good options for the classifier to reach a baseline acceptable accuracy:

• [x] Try vgg16, pretrained on imagenet
• [x] Try vgg16, training from scratch
• [ ] Try vgg16, pretrained with pretrain.py

We can start with the RMSProp optimizer.

[sdv4]

Class balance: Y: 53% D+N : 47%

Network - vgg16 Pretrained - none Image size: (48,48) Optimiser - sgd Epochs - 50 Test Set Accuracy - 84.54% N+D Prediction Precision: 83.33% Confusion matrix: [[215 50] [43 300]]

122658933ee9_19_38_2018_12_17.txt

Network - vgg16 Pretrained - none Image size: (48,48) Optimiser - rmsprop Epochs - 50 Test Set Accuracy - 82.57% N+D Prediction Precision: 84.78% Confusion matrix: [[195 70] [35 308]]

2678b9ed874b_05_31_2018_12_17.txt

[marco-c]

What exactly did you do in the pretrain.py case?

[marco-c]

Also, could you try using rmsprop instead of sgd?

[marco-c]

Also, could you try using rmsprop instead of sgd?

Actually, both, to compare exactly their results.

[sdv4]

What exactly did you do in the pretrain.py case?

I set target_size=(48,48) in utils.py/load_image, then ran my notebook with !python3 pretrain.py -n=vgg16 -o=sgd

Actually, both, to compare exactly their results.

Will do.

[marco-c]

I set target_size=(48,48) in utils.py/load_image, then ran my notebook with !python3 pretrain.py -n=vgg16 -o=sgd

OK. So, pretrain.py is meant to only pretrain the network. In theory, we should first pretrain with pretrain.py, then use the pretrained model to run train.py.

[sdv4]

Ahh, I was thinking that this might be the case. Now it makes sense why there was no .txt file, etc. Do you have info on what is going on in pretrain.py; in particular the "slightly different problem (for which we know the solution)"?

[marco-c]

This explains it all: https://github.com/marco-c/autowebcompat/blob/master/pretrain.py#L58

The goal of the classifier in the pretrain case is to detect when two screenshots belong to the same website.

[sdv4]

This explains it all: https://github.com/marco-c/autowebcompat/blob/master/pretrain.py#L58

Indeed it does, thanks!

[sdv4]

Class balance: Y: 53% D+N : 47%

Network - vgg16 Pretrained - imagenet Image size: (224,224) Optimiser - sgd Epochs - 50 Test Set Accuracy - 84.9% N+D Prediction Precision: 85.5% Confusion matrix: [[254 47] [43 264]]

tensorflow-1-vm_21_49_2018_12_20.txt

Network - vgg16 Pretrained - imagenet Image size: (224,224) Optimiser - rmsprop Epochs - 50 Test Set Accuracy - 80.9% N+D Prediction Precision: 81.6% Confusion matrix: [[222 73] [ 50 263]]

da78b06ff04a_00_41_2019_01_27.txt

[sdv4]

@marco-c I also performed hyper-parameter optimization with random search, and Hyperband. The best configuration found was via random search:

Network - vgg16 Pretrained - no Test Set Accuracy - 80.1% N+D Prediction Precision: 88.1% Image size: (48,48) Optimizer: Adam Learning rate: 1.7e-5
Decay: 1e-6 Momentum: 0.74
Epsilon: 8.4e-8
fc1 L2 regularization strength: 2.32e-2
fc2 L2 regularization strength: 5.19e-3
fc1 dropout: 3.37e-7
fc2 dropout: 1.71e-7

[marco-c]

Not bad! We should improve the labeling to have better/more precise results