Problems reproducing performance

[Kwon-Min-Jeong]

Hello. Thank you for your excellent paper, which I read with great interest.

I conducted experiments to reproduce the performance described in the paper.

As a result, I found that the performance difference was significantly beyond the range of variation. (GPU used: RTX TITAN)

Training with MVTecAD and testing with VisA resulted in the following performance: 2-shot: 0.820 (AUROC), 0.837 (AUPRO), paper: 0.858(AUROC), 0.877(AUPRO) 4-shot: 0.832 (AUROC), 0.849 (AUPRO), paper: 0.877(AUROC), 0.902(AUPRO) 8-shot: 0.827 (AUROC), 0.848(AUPRO), paper: 0.887(AUROC), 0.904(AUPRO)

In addition, I did not change the default settings.

There might be an issue with the command I used, so I am attaching it as well. (I trained with all classes of MVTecAD and tested by VisA, The shot was directly modified in main.py.)

train command: python main.py --normal_json_path D://InCTRL-main//data//mvtec_json//AD_json//bottle_normal.json D://InCTRL-main//data//mvtec_json//AD_json//cable_normal.json D://InCTRL-main//data//mvtec_json//AD_json//capsule_normal.json D://InCTRL-main//data//mvtec_json//AD_json//carpet_normal.json D://InCTRL-main//data//mvtec_json//AD_json//grid_normal.json D://InCTRL-main//data//mvtec_json//AD_json//hazelnut_normal.json D://InCTRL-main//data//mvtec_json//AD_json//leather_normal.json D://InCTRL-main//data//mvtec_json//AD_json//metal_nut_normal.json D://InCTRL-main//data//mvtec_json//AD_json//pill_normal.json D://InCTRL-main//data//mvtec_json//AD_json//screw_normal.json D://InCTRL-main//data//mvtec_json//AD_json//tile_normal.json D://InCTRL-main//data//mvtec_json//AD_json//toothbrush_normal.json D://InCTRL-main//data//mvtec_json//AD_json//transistor_normal.json D://InCTRL-main//data//mvtec_json//AD_json//wood_normal.json D://InCTRL-main//data//mvtec_json//AD_json//zipper_normal.json --outlier_json_path D://InCTRL-main//data//mvtec_json//AD_json//bottle_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//cable_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//capsule_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//carpet_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//grid_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//hazelnut_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//leather_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//metal_nut_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//pill_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//screw_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//tile_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//toothbrush_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//transistor_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//wood_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//zipper_outlier.json --val_normal_json_path D://InCTRL-main//data//mvtec_json//AD_json//bottle_val_normal.json D://InCTRL-main//data//mvtec_json//AD_json//cable_val_normal.json D://InCTRL-main//data//mvtec_json//AD_json//capsule_val_normal.json D://InCTRL-main//data//mvtec_json//AD_json//carpet_val_normal.json D://InCTRL-main//data//mvtec_json//AD_json//grid_val_normal.json D://InCTRL-main//data//mvtec_json//AD_json//hazelnut_val_normal.json D://InCTRL-main//data//mvtec_json//AD_json//leather_val_normal.json D://InCTRL-main//data//mvtec_json//AD_json//metal_nut_val_normal.json D://InCTRL-main//data//mvtec_json//AD_json//pill_val_normal.json D://InCTRL-main//data//mvtec_json//AD_json//screw_val_normal.json D://InCTRL-main//data//mvtec_json//AD_json//tile_val_normal.json D://InCTRL-main//data//mvtec_json//AD_json//toothbrush_val_normal.json D://InCTRL-main//data//mvtec_json//AD_json//transistor_val_normal.json D://InCTRL-main//data//mvtec_json//AD_json//wood_val_normal.json D://InCTRL-main//data//mvtec_json//AD_json//zipper_val_normal.json --val_outlier_json_path D://InCTRL-main//data//mvtec_json//AD_json//bottle_val_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//cable_val_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//capsule_val_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//carpet_val_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//grid_val_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//hazelnut_val_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//leather_val_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//metal_nut_val_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//pill_val_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//screw_val_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//tile_val_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//toothbrush_val_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//transistor_val_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//wood_val_outlier.json D://InCTRL-main//data//mvtec_json//AD_json//zipper_val_outlier.json

test command(repeated for each class): python test.py --val_normal_json_path D://InCTRL-main//data//visa_json//AD_json//macaroni2_val_normal.json --val_outlier_json_path D://InCTRL-main//data//visa_json//AD_json//macaroni2_val_outlier.json --category macaroni2 --few_shot_dir D:/InCTRL-main/data/visa_inference/visa/8/

Is there an issue with the way I used the command?