ViViT strange results using scenic checkpoints.

[namhar]

We are using ViViT base checkpoint and code provided in scenic and obtain strange inference results on K400 test set (no training, just inference from the checkpoint).

• When we use the full K400 test set as the test set (~33k samples) Results: I0222 07:01:57.562714 140194510305024 logging_writer.py:35] [7650] test/accuracy=0.002614,test/accuracy_top_5=0.012980, test/loss=5.991464

• We obtain similar results on the full test set using each of the following model's respective checkpoints: ViViT-B/16x2, ViViT-B/16x2 FE, ViViT-L/16x2 FE

• eg. for ViViT-B/16x2 unfactorized we use the following checkpoint: https://storage.googleapis.com/scenic-bucket/vivit/kinetics_400/vivit_base_16x2_unfactorized/checkpoint

• Similarly obtaining strange test results when using a small amount of the K400 training data as the test set.

• If we train ViViT after loading a ViT checkpoint, the model does learn (accuracy climbs and loss decreases for training) however we similarly obtain strange results for inference on validation (2 orders of magnitude lower than train)

Platform: Linux environment with GPU's. Data preprocessed using the indicated preprocessing instructions (using DMVR) including generating tfrecords.

Any help with figuring out what we might be missing would be greatly appreciated.

[Nada-Baili]

Same here! I am getting suspiciously low accuracy results on Kinetics400 using a pretrained ViVit 16x2 FE (Inference only).

[Fredbcx]

Same here! I am getting suspiciously low accuracy results on Kinetics400 using a pretrained ViVit 16x2 FE (Inference only).

Can you please tell me how did you load the model, i'm struggling with tensorflow

[Nada-Baili]

@Fredbcx I downloaded a checkpoint from the model zoo https://github.com/google-research/scenic/tree/main/scenic/projects/vivit. You need to place the checkpoint in the folder of the model. For example, if you want to run vivit with ViT-Large, then place the checkpoint in vivit_large_factorised_encoder (the folder will be created automatically). One trick that I found after many trial and errors is that the saved checkpoint's name must start with "checkpoint_". I hope this helps!

[namhar]

We have found the issue. It has to do with the label mappings not being aligned between train and validation/test. (ie. class strings -> class indices conversion's do not match); so its also related to DMVR.

Our workaround was to generate a unique label mapping for our processed data, save that and pass that in which lets us train and test with reasonable results so far.

However it is difficult to run the scenic provided checkpoints since we do not know what the mapping was. Although, I suppose we could look at the class probs on the training data, argmax and try to recreate them ... but its not an elegant solution.

[Mohamedellebody]

@Nada-Baili I saw that you worked on vivit. could you help with this issue, please? https://github.com/google-research/scenic/issues/679