Val_mssim very low

[nguyenanhtuan1008]

@AdneneBoumessouer Thank for your hard work. I try to run your code but I saw that accuracy very low. I don't know what happen but I try with 3 ways and all given not good result.

python3 train.py -d mvtec/pill -a baselineCAE -b 32 -l mssim -c rgb (I also tested with ssim, l2, grayscale) python3 train.py -d mvtec/pill -a mvtecCAE -b 32 -l mssim -c rgb python3 train.py -d mvtec/pill -a inceptionCAE -b 32 -l mssim -c rgb

-> Epoch 00013: Reducing Max LR on Plateau: new max lr will be 0.08725637942552567 (if not early_stopping). Restoring model weights from the end of the best epoch. 8/8 [==============================] - 10s 1s/step - loss: 0.4127 - mssim: 0.8223 - val_loss: 0.8789 - val_mssim: 0.3548 Epoch 00013: early stopping Weights from best epoch have been loaded into model.

INFO:autoencoder.autoencoder:loss_plot.png successfully saved.

INFO:autoencoder.autoencoder:lr_schedule_plot.png successfully saved. INFO:autoencoder.autoencoder:training history has been successfully saved as csv file. INFO:autoencoder.autoencoder:training files have been successfully saved at: /content/drive/Shared drives/1_New/Acuity/MVTec-Anomaly-Detection/saved_models/mvtec/pill/baselineCAE/mssim/13-10-2020_11-35-32 INFO:__main__:done.

And then, using finetune: python3 finetune.py -p "saved_models/mvtec/pill/baselineCAE/mssim/13-10-2020_11-35-32/baselineCAE_b32_e0.hdf5" -m ssim -t float64

Last, using test.py: python3 test.py -p "saved_models/mvtec/pill/baselineCAE/mssim/13-10-2020_11-35-32/baselineCAE_b32_e0.hdf5"

              filenames  predictions  truth  accurate_predictions

0 color/000.png 0 1 False 1 color/001.png 0 1 False 2 color/002.png 0 1 False 3 color/003.png 1 1 True 4 color/004.png 0 1 False 5 color/005.png 0 1 False 6 color/006.png 0 1 False 7 color/007.png 1 1 True 8 color/008.png 1 1 True 9 color/009.png 1 1 True 10 color/010.png 0 1 False 11 color/011.png 0 1 False 12 color/012.png 0 1 False 13 color/013.png 0 1 False 14 color/014.png 0 1 False 15 color/015.png 0 1 False 16 color/016.png 0 1 False 17 color/017.png 0 1 False 18 color/018.png 1 1 True 19 color/019.png 1 1 True 20 color/020.png 0 1 False 21 color/021.png 0 1 False 22 color/022.png 0 1 False 23 color/023.png 0 1 False 24 color/024.png 0 1 False 25 combined/000.png 0 1 False 26 combined/001.png 1 1 True 27 combined/002.png 0 1 False 28 combined/003.png 0 1 False 29 combined/004.png 0 1 False 30 combined/005.png 0 1 False 31 combined/006.png 1 1 True 32 combined/007.png 1 1 True 33 combined/008.png 1 1 True 34 combined/009.png 1 1 True 35 combined/010.png 1 1 True 36 combined/011.png 1 1 True 37 combined/012.png 1 1 True 38 combined/013.png 0 1 False 39 combined/014.png 0 1 False 40 combined/015.png 1 1 True 41 combined/016.png 0 1 False 42 contamination/000.png 0 1 False 43 contamination/001.png 0 1 False 44 contamination/002.png 0 1 False 45 contamination/003.png 0 1 False 46 contamination/004.png 0 1 False 47 contamination/005.png 0 1 False 48 contamination/006.png 0 1 False 49 contamination/007.png 0 1 False 50 contamination/008.png 0 1 False 51 contamination/009.png 0 1 False 52 contamination/010.png 0 1 False 53 contamination/011.png 1 1 True 54 contamination/012.png 1 1 True 55 contamination/013.png 0 1 False 56 contamination/014.png 0 1 False 57 contamination/015.png 0 1 False 58 contamination/016.png 0 1 False 59 contamination/017.png 0 1 False 60 contamination/018.png 1 1 True 61 contamination/019.png 0 1 False 62 contamination/020.png 1 1 True 63 crack/000.png 0 1 False 64 crack/001.png 0 1 False 65 crack/002.png 1 1 True 66 crack/003.png 0 1 False 67 crack/004.png 0 1 False 68 crack/005.png 0 1 False 69 crack/006.png 1 1 True 70 crack/007.png 0 1 False 71 crack/008.png 0 1 False 72 crack/009.png 0 1 False 73 crack/010.png 1 1 True 74 crack/011.png 1 1 True 75 crack/012.png 0 1 False 76 crack/013.png 1 1 True 77 crack/014.png 0 1 False 78 crack/015.png 0 1 False 79 crack/016.png 0 1 False 80 crack/017.png 0 1 False 81 crack/018.png 0 1 False 82 crack/019.png 1 1 True 83 crack/020.png 0 1 False 84 crack/021.png 0 1 False 85 crack/022.png 1 1 True 86 crack/023.png 1 1 True 87 crack/024.png 0 1 False 88 crack/025.png 0 1 False 89 faulty_imprint/000.png 0 1 False 90 faulty_imprint/001.png 0 1 False 91 faulty_imprint/002.png 1 1 True 92 faulty_imprint/003.png 0 1 False 93 faulty_imprint/004.png 0 1 False 94 faulty_imprint/005.png 0 1 False 95 faulty_imprint/006.png 1 1 True 96 faulty_imprint/007.png 1 1 True 97 faulty_imprint/008.png 0 1 False 98 faulty_imprint/009.png 0 1 False 99 faulty_imprint/010.png 0 1 False 100 faulty_imprint/011.png 0 1 False 101 faulty_imprint/012.png 1 1 True 102 faulty_imprint/013.png 0 1 False 103 faulty_imprint/014.png 0 1 False 104 faulty_imprint/015.png 0 1 False 105 faulty_imprint/016.png 1 1 True 106 faulty_imprint/017.png 0 1 False 107 faulty_imprint/018.png 1 1 True 108 good/000.png 0 0 True 109 good/001.png 0 0 True 110 good/002.png 0 0 True 111 good/003.png 0 0 True 112 good/004.png 0 0 True 113 good/005.png 0 0 True 114 good/006.png 0 0 True 115 good/007.png 0 0 True 116 good/008.png 0 0 True 117 good/009.png 0 0 True 118 good/010.png 0 0 True 119 good/011.png 0 0 True 120 good/012.png 1 0 False 121 good/013.png 0 0 True 122 good/014.png 1 0 False 123 good/015.png 0 0 True 124 good/016.png 0 0 True 125 good/017.png 1 0 False 126 good/018.png 0 0 True 127 good/019.png 0 0 True 128 good/020.png 0 0 True 129 good/021.png 0 0 True 130 good/022.png 0 0 True 131 good/023.png 0 0 True 132 good/024.png 0 0 True 133 good/025.png 0 0 True 134 pill_type/000.png 0 1 False 135 pill_type/001.png 0 1 False 136 pill_type/002.png 0 1 False 137 pill_type/003.png 0 1 False 138 pill_type/004.png 0 1 False 139 pill_type/005.png 0 1 False 140 pill_type/006.png 0 1 False 141 pill_type/007.png 1 1 True 142 pill_type/008.png 1 1 True 143 scratch/000.png 1 1 True 144 scratch/001.png 0 1 False 145 scratch/002.png 0 1 False 146 scratch/003.png 0 1 False 147 scratch/004.png 1 1 True 148 scratch/005.png 0 1 False 149 scratch/006.png 0 1 False 150 scratch/007.png 0 1 False 151 scratch/008.png 0 1 False 152 scratch/009.png 0 1 False 153 scratch/010.png 0 1 False 154 scratch/011.png 0 1 False 155 scratch/012.png 0 1 False 156 scratch/013.png 0 1 False 157 scratch/014.png 0 1 False 158 scratch/015.png 1 1 True 159 scratch/016.png 1 1 True 160 scratch/017.png 0 1 False 161 scratch/018.png 0 1 False 162 scratch/019.png 0 1 False 163 scratch/020.png 0 1 False 164 scratch/021.png 1 1 True 165 scratch/022.png 1 1 True 166 scratch/023.png 0 1 False test results: {'min_area': 770, 'threshold': 0.27000000000000013, 'TPR': 0.2907801418439716, 'TNR': 0.8846153846153846, 'score': 0.587697763229678, 'method': 'ssim', 'dtype': 'float64'}

The result is not good at all. Can you give me an advice?

[AdneneBoumessouer]

Rerun your experiments with mvtecCAE ,baselineCAE and inceptionCAE but this time by adding some variability to your dataset by increasing the data augmentation parameters in config.py, for example with the following values: ROT_ANGLE = 30 W_SHIFT_RANGE = 0.1 H_SHIFT_RANGE = 0.1 FILL_MODE = "nearest" BRIGHTNESS_RANGE = [0.90, 1.1] VAL_SPLIT = 0.1 Use the flag --inspect in train.py to generate inspection plots in order to assess the quality of the reconstructions.

If the results are not satisfactory, try using resnetCAE with a batch size of 8, the ssim loss and grayscale and set the flag --inspect: python3 train.py -d mvtec/pill -a resnetCAE -b 8 -l ssim -c grayscale --inspect

[nguyenanhtuan1008]

I try all mvtec, baseline, inception with your config, the val_mssim increased in training but the testing result not good.

• test results: {'min_area': 920, 'threshold': 0.11400000000000002, 'TPR': 0.3120567375886525, 'TNR': 0.8846153846153846, 'score': 0.5983360611020185, 'method': 'ssim', 'dtype': 'float64'} --> mvtecCAE

• test results: {'min_area': 785, 'threshold': 0.15400000000000005, 'TPR': 0.41134751773049644, 'TNR': 1.0, 'score': 0.7056737588652482, 'method': 'ssim', 'dtype': 'float64'} --> baselineCAE

• inceptionCAE got error: Finetune.py:

Traceback (most recent call last): File "finetune.py", line 360, in main(args) File "finetune.py", line 78, in main preprocessing_function = get_preprocessing_function(architecture) File "MVTec-Anomaly-Detection/processing/preprocessing.py", line 149, in get_preprocessing_function return preprocessing_function UnboundLocalError: local variable 'preprocessing_function' referenced before assignment

• resnetCAE got error when training:

Epoch 9/10 30/30 [==============================] - 8s 260ms/step - loss: nan - ssim: nan

done. Visually inspect loss plot and select learning rate associated with falling loss

INFO:autoencoder.autoencoder:lr with minimum loss divided by 10: 6.31E-02 INFO:autoencoder.autoencoder:lr with minimum numerical gradient: 1.45E-04 MVTec-Anomaly-Detection/autoencoder/autoencoder.py:231: RuntimeWarning: invalid value encountered in less self.lr_opt_i = np.argwhere(segment main(args) File "train.py", line 81, in main autoencoder.find_lr_opt(train_generator, validation_generator) File "MVTec-Anomaly-Detection/autoencoder/autoencoder.py", line 194, in find_lr_opt self.custom_lr_estimate() File "MVTec-Anomaly-Detection/autoencoder/autoencoder.py", line 231, in custom_lr_estimate self.lr_opt_i = np.argwhere(segment

[AdneneBoumessouer]

Did you encounter the same problems for the other categories of the MVTec Dataset ?

[nguyenanhtuan1008]

Did you encounter the same problems for the other categories of the MVTec Dataset ?

yes. got the same problem!

[chenghao618]

对于 MVTec 数据集的其他类别，您是否遇到过同样的问题？

是的。遇到了同样的问题！ Did you solve this problem？ I saw that accuracy very low too.