SIFID from paper unreproducible

[ThisIsIsaac]

I've modified calculate_sifid_given_paths(path1, path2, batch_size, cuda, dims, suffix) slightly in sifid_score.py s.t. the function calculates mu and sigma of a single real image and claculate mu and sigma of 50 fake images generated from the single real image. This was an attempt to decrease variation of mu and sigma of the generated output:

def calculate_sifid_given_paths(path1, path2, batch_size, cuda, dims, suffix):
    # some code...
    # ...


    # get mu and sigma of single training image
    ref_m, ref_s = calculate_activation_statistics([img_path], model, batch_size, dims, cuda)

    fid_values = []

    # get fid of multiple images that were based on the single reference image,
    # then calculate distance between each one and ref_m, ref_s
    for i in range(len(files2)):
        generated_m, generated_s = calculate_activation_statistics([files2[i]], model, batch_size, dims, cuda)
        fid_values.append(calculate_frechet_distance(ref_m, ref_s, generated_m, generated_s))

    return fid_values

The values I get are too small compared to the pubilshed SIFID:

When using balloons.png as training image, I get 1.650822e-05 for n=N and 1.0670579e-05 for n=N-1 with 50 generated samples.

The published SIFID are:

[tamarott]

In the paper we compared a single image sample with it's corresponding real image, for 50 different images (you can find all the data in the Download folder, we took them from the "places" dataset). Different experiment settings and/or other real images are likely to give different SIFID values.

[ThisIsIsaac]

So this is a natural variance? The score I got is over 1000 times smaller so I assumed there was an error in SIFID calculation. What is the specific SIFID you got for ‘balloons.png’? Could you share the random seed so that I can reproduce the results?

Thanks

[15732031137]

@ThisIsIsaac Hello ! Thank you for sharing your thoughts here! I have a question for you, I do n’t understand how to calculate the SIFID, I trained Balloons.png and generated 50 pictures. I want to know that the code in this article calculates one picture at a time, so that it counts a total of 50 times, or calculate one real picture and the 50 pictures it produces at once, and the real image file name and fake image file said by the author What does the same name mean? At the beginning, I thought the author's program was to calculate the sfid of the real picture and the 50 pictures generated at one time, but putting these 50 pictures in a folder, I couldn't change the file names of the 50 pictures to the same If I can only calculate one real picture and one fake picture at a time, should I change the file names of the two pictures to the same?
Looking forward to your reply, I wish you a happy life!

[15732031137]

@ThisIsIsaac I understand

[tportenier]

I also struggle reproducing the numbers in Table 2 from the paper. If I run your script on the data in user_study/fake_high_variance and user_study/fake_mid_variance I get 1.6010857 and 1.6310933 respectively (the values printed by the script to stdout, i.e., the average over the 50 samples). Does this make sense? Shouldn't the number be lower for mid_variance compared to high_variance? The values in Table 2 are more than one order of magnitude lower.

Thanks!

[15732031137]

@tportenier If you want to reproduce the results in the text, you need to change the picture format to jpg

[tportenier]

The pictures are already in jpg, this is not the reason.

[simcoster]

for us, commenting out the line in sifid_score.py that reads "images /= 255" did the trick, the values were already between 0 and 1

[tamarott]

I just checked the code again: I get exactly the same numbers as in the paper without changing anything in the code.

[benfei]

I also struggle reproducing the numbers in Table 2 from the paper. If I run your script on the data in user_study/fake_high_variance and user_study/fake_mid_variance I get 1.6010857 and 1.6310933 respectively (the values printed by the script to stdout, i.e., the average over the 50 samples). Does this make sense? Shouldn't the number be lower for mid_variance compared to high_variance? The values in Table 2 are more than one order of magnitude lower.

Thanks!

We had a similar issue. It was fixed by sorting the files lists (PR #136).

[EliotChenKJ]

Hi, thank for contribute this briliant work, but there may be one bug in SIFID calcalation code(in get_activations function): images = np.array([imread(str(f)).astype(np.float32) for f in files[start:end]]) images /= 255 the images read by imread function are already between [0, 1], i think that we don't need to divide it by 255 once more. This is also why the SIFID value provided in paper is so small.