Method of temporal_transform and Backward

[johook]

Thank you for your excellent work. I have two questions: @yaorong0921

1. The dataloaders for incabin and outcabin apply temporal_transform in different ways in model_inside.py and model_outside.py respectively (train_incabin: UniformRandomSample, val_incabin: UniformEndSample, train_outcabin: UniformIntervalCrop, val_outcabin: UniformIntervalCrop). When combining incabin and outcabin, should the same approach be followed? or should we unify the temporal_transform?
2. When calculating and performing backward on the loss, I'm curious if the loss for incabin and outcabin is combined for backward, or if they are processed separately. Thank you.

[johook]

Additionally, in dataset.py, for the get_training_set, the n_samples_for_each_video for inside and outside differ, being 1 and 10 respectively, which results in different lengths for the dataloaders. Have these been unified as well? @yaorong0921

[yaorong0921]

Hi @johook, Sorry for the late reply. I have been on trips over the past few weeks.

1. When training the combination, I extracted the features from two branches with the trained model for each second (from 0 to 5) , and saved the features. For all subsets. No augmentation is used when extracting and training the combination network using these features.
2. I trained the incabin and outcabin models separately, which also answers your first and third questions, why the data augmentation and sampling parameters might be different.
3. You can find the script for the dataloader/train/combination network. Hope it helps you understand better how the combination works.