What transformations suit ImageNet-V Evaluation? (Error rate 100%)

[f-amerehi]

Hi @Heathcliff-saku,

Firstly, thank you for the dataset and codes. I'd like to evaluate some fine-tuned models (like ResNet, DenseNet) on the ImageNet-V data. However, the accuracy shows 0. To debug, I simply loaded the default ImageNet_1K checkpoints and used the following code to see their accuracy. In the (first page) of repository, it says that DenseNet's accuracy on ImageNet-V is ~20. However, the following code still shows an error rate of 100. I looked into the evaluation code and believe the normalization is correct. Is there any specific normalization required for the test phase on ImageNet-V? Also, should I set the number of classes to 1000 (ImageNet) or 100 (ImageNet-V)? I would appreciate any thoughts.

Many Thanks.

import torch
from torchvision.datasets import ImageFolder
from torchvision import transforms
from torchvision.models import densenet121, DenseNet121_Weights
from torch.utils.data import DataLoader
from torchmetrics import Accuracy
from tqdm import tqdm


normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
test_transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    normalize
])


imagenetV = 'C:/datasets/ImageNet-V'
test_dataset = ImageFolder(root=imagenetV, transform=test_transform)
test_loader = DataLoader(test_dataset, batch_size=128, shuffle=False)


model = densenet121(weights= DenseNet121_Weights.IMAGENET1K_V1)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
model.eval()


accuracy = Accuracy(task="multiclass", num_classes=1000).to(device)  # tested both 1000 and 100


with torch.no_grad():
    for images, labels in tqdm(test_loader, desc="Evaluation on imagenet-V"):
        images, labels = images.to(device), labels.to(device)
        outputs = model(images)
        preds = torch.argmax(outputs, dim=1)
        accuracy.update(preds, labels)

error_rate = 1 - accuracy.compute().item()
print(f"Error rate on ImageNet-V dataset: {error_rate*100:.4f}")


#Output shows
#Error rate on ImageNet-V dataset: 100.0000