detectors_review icon indicating copy to clipboard operation
detectors_review copied to clipboard

The results of LID for detecting PGD-L∞(8) is much lower than the paper

Open CycleBooster opened this issue 1 year ago • 1 comments

HI! I want to get the results of each type of adversarial example detection as the baseline. Because the max_iter should be 5 in my paper rather than the default number 100, I run the code to generate the results. However, when I ran the code (only comment out the code for other attacks, the max_iter is still 100), the results of LID for detecting PGD-L∞(8) was much lower than the paper (the followingscreen shot). image

The results should be DR=94.39 and FPR=1.81, but I got DR=66.13 and FPR=11.29. Is the setting of the paper is different from the code? Or do you know anything may be wrong? the code I fork and change is here: https://github.com/CycleBooster/detectors_review

By the way, I change common/utils.py and generate_adv.py. Otherwise, the newest version of art can't run.

CycleBooster avatar Apr 20 '23 07:04 CycleBooster

I train the model and LID again and get a much different result. image Although the result still lower than the paper, it becomes much better! Does that mean the accuracy of LID is very unstable?

CycleBooster avatar Apr 20 '23 08:04 CycleBooster