Inquiry on Incorporating Future Data in Training for StreamPETR.

[kkkcx]

Hello,

I would like to seek your guidance on whether StreamPETR can also incorporate future data into training, similar to Sparse4D-v3-offline. If possible, could you kindly advise how I should modify the config or training code to achieve this objective? Thank you very much for your assistance!!

[exiawsh]

You should modify the training code. And use sliding window to train the offline model. https://github.com/exiawsh/StreamPETR/blob/2315cf9f077817ec7089c87094ba8a63f76c2acf/projects/mmdet3d_plugin/datasets/nuscenes_dataset.py#L91 The index of nuscenes dataset is organized in chronological order. The sliding window I implemented only loads past frames. You can add future indices to the list.

[kkkcx]

Thank you very much for the response. I noticed that in the code, when in seq_mode, self.queue_length is set to 1, indicating that the original training uses a queue length of 1.

https://github.com/exiawsh/StreamPETR/blob/2315cf9f077817ec7089c87094ba8a63f76c2acf/projects/mmdet3d_plugin/datasets/nuscenes_dataset.py#L36-L41

I would like to inquire if increasing the queue length to a larger number, such as 4 or 8, and including some future frames might enhance the model's performance?

Additionally, could you kindly advise where I should adjust the corresponding queue_length when testing the model? Thank you so much!

[exiawsh]

No. When use seq_mode the training seq_len is controled by seq_split_num. queue_length = 40 (frames per scene in nuscenes) // seq_split_num. When testing, the queue_length = 40.

[kkkcx]

Oh, thank you for the correction. I am not yet very familiar with the code. Following your guidance, how should I incorporate future frames during the training and testing phases to enhance the model when in seq_mode?

[kkkcx]

Thank you once again for the guidance. I understand that I should train using the sliding window mode. I have another question: should I also modify the code for the testing phase to include future frames? I noticed that there is no queue_length parameter mentioned for testing.

https://github.com/exiawsh/StreamPETR/blob/2315cf9f077817ec7089c87094ba8a63f76c2acf/projects/mmdet3d_plugin/datasets/nuscenes_dataset.py#L118-L130

[exiawsh]

You should modify the test pipeline and modify prepare_test_data. You are in the right way. Another alternative approach is to train an online model that does not include future frames. Save the query features of the online model, and then load the memory of the online model for interaction during training and testing.