Question about the AVSS pre-training

[SitongGong]

When training the model on the AVSS Datasets, we find that the MIOU is about 20 with Res50 backbone and is about 30 with PVT-v2 backbone at 11 epochs. Could you please confirm if this is a normal occurrence? We have completed training for a total of 30 epochs, and in the subsequent 20 epochs, we observed an increase of approximately 6 points.

[vvvb-github]

Yes, this should be normal. The results you described are even slightly higher than mine. 

------------------ 原始邮件 ------------------ 发件人: "vvvb-github/AVSegFormer" @.>; 发送时间: 2023年11月7日(星期二) 晚上10:51 @.>; @.***>; 主题: [vvvb-github/AVSegFormer] Question about the AVSS pre-training (Issue #4)

When training the model on the AVSS Datasets, we find that the MIOU is about 20 with Res50 backbone and is about 30 with PVT-v2 backbone at 11 epochs. Could you please confirm if this is a normal occurrence? We have completed training for a total of 30 epochs, and in the subsequent 20 epochs, we observed an increase of approximately 6 points.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[SitongGong]

Thanks so much. Now I have another question. During training, do you use the model with the best performance on the validation set for testing on the test set?

[vvvb-github]

Yes, and the performance on two different sets may differ slightly.

------------------ 原始邮件 ------------------ 发件人: "vvvb-github/AVSegFormer" @.>; 发送时间: 2023年11月10日(星期五) 下午4:37 @.>; @.@.>; 主题: Re: [vvvb-github/AVSegFormer] Question about the AVSS pre-training (Issue #4)

Thanks so much. Now I have another question. During training, do you use the model with the best performance on the validation set for testing on the test set?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[SitongGong]

Ok, I actually made some improvements by incorporating the approach of ReferFormer on top of your model. Surprisingly, the model achieved performance of 66.1 and 79.87 on the validation set, but on the test set, it dropped significantly to 58.1 and 69.14. I didn't expect the gap between the two to be this large.

[vvvb-github]

Congratulations! Indeed, there is a significant difference between the two datasets, especially MS3, which may be due to the dataset being too small..

------------------ 原始邮件 ------------------ 发件人: "vvvb-github/AVSegFormer" @.>; 发送时间: 2023年11月10日(星期五) 下午4:49 @.>; @.@.>; 主题: Re: [vvvb-github/AVSegFormer] Question about the AVSS pre-training (Issue #4)

Ok, I actually made some improvements by incorporating the approach of ReferFormer on top of your model. Surprisingly, the model achieved performance of 66.1 and 79.87 on the validation set, but on the test set, it dropped significantly to 58.1 and 69.14. I didn't expect the gap between the two to be this large.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[SitongGong]

Thank you. But it seems that our current results may not be suitable for inclusion in the paper because the performance on the test set is not satisfactory, whereas results from other papers are typically evaluated on the test set.

[SitongGong]

How long have you pre-train the AVSegFormer on AVSS Datasets approximately?

[vvvb-github]

About 2 days.

发自我的iPhone

------------------ Original ------------------ From: SitongGong @.> Date: Sun,Nov 19,2023 6:34 PM To: vvvb-github/AVSegFormer @.> Cc: vvvb @.>, Comment @.> Subject: Re: [vvvb-github/AVSegFormer] Question about the AVSS pre-training(Issue #4)

How long have you pre-train the AVSegFormer on AVSS Datasets approximately?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[SitongGong]

Only use 1 V100 GPU and the batch size is set to 4?

[SitongGong]

I trained my model with 1 A800 GPU and the batch size is set to 4, but it may takes 9 days on the AVSS Datasets.

[SitongGong]

Happy new year! One more thing I'd like to confirm: Does the calculation of the F-score metric have any relationship with the batch size? Would adjusting the batch size of the test set to 1 affect fairness