Hi, I'm following your CoTracker3, and I found that the previous Kubric dataset for CoTracker2 is not suitable for CoTracker3, as CoTracker3 requires a `seq_len` of 64, while CoTracker2 only needs a `seq_len` of 24. I am wondering how to create a Kubric dataset with 64 frames. Thank you for your guidance!

[ZeroWen]

          Hi @nikitakaraevv , I'm following your CoTracker3, and I found that the previous Kubric dataset for CoTracker2 is not suitable for CoTracker3, as CoTracker3 requires a `seq_len` of 64, while CoTracker2 only needs a `seq_len` of 24. I am wondering how to create a Kubric dataset with 64 frames. Thank you for your guidance!

Originally posted by @sinkers-lan in https://github.com/facebookresearch/co-tracker/issues/8#issuecomment-2439941747

[nikitakaraevv]

Hi @ZeroWen, that's the modified movi_def_worker.py file that we used to generate this dataset. It actually contains 120 frames per sequence, but we just sample 64 frames during training. https://gist.github.com/nikitakaraevv/14235feb8d60251fb271421bfb0460b7

That's the command to launch data generation with this file:

python ./kubric/challenges/movi/movi_def_worker_long_kubric.py \
--max_motion_blur=2.0 --camera=linear_movement --max_camera_movement 8 --randomize_focal_length \
--min_num_dynamic_objects 0 --max_num_dynamic_objects 10 --output_dir ./generated_data

@fusheng-ji @zbw001 @sinkers-lan

[sinkers-lan]

@nikitakaraevv Thanks for your reply! It helps a lot.

[Superviro]

So the basic dataset preparation process for CoTracker3 is to generate a 120-frame movi-f dataset first, then run challenges/point_tracking/dataset.py to generate point tracking annotations, am I comprehending correctly? By the way how many 120-frame videos are generated for training CoTracker3? @nikitakaraevv

[nikitakaraevv]

Hi @Superviro, yes, that's correct. We generate 10k 120-frame videos

[sunjiating]

你好@ZeroWen，这是movi_def_worker.py我们用来生成这个数据集的修改后的文件。它实际上每个序列都包含120帧，但我们在训练期间只采样64帧。https ://gist.github.com/nikitakaraevv/14235feb8d60251fb271421bfb0460b7

这是使用此文件启动数据生成的命令：

python ./kubric/challenges/movi/movi_def_worker_long_kubric.py \
--max_motion_blur=2.0 --camera=linear_movement --max_camera_movement 8 --randomize_focal_length \
--min_num_dynamic_objects 0 --max_num_dynamic_objects 10 --output_dir ./generated_data

@fusheng-ji @zbw001 @sinkers-lan

@nikitakaraevv Hello, I would like to ask why after executing this command, it only generates a dataset with 120 frames and then stops. How should I execute it to generate a 10k video dataset? Also, have you made any modifications to the code for generating annotations in challenges/point_tracking/dataset.py?

[nikitakaraevv]

Hi @sunjiating, yes, I forgot to mention that it only generates one sequence. We just submit 10k such batch jobs when generating the dataset. The code for annotations haven't changed

[sunjiating]

@nikitakaraevv Thank you for your reply. As a beginner, I have encountered some difficulties in the data generation process. Below is a modification I made after some trial and error: First, I used the movi_def_worker_long_kubric.py script you provided to generate the videos, with the output_dir set to ./generated_data/0000, ./generated_data/0001, and so on. Then, I modified the BUILDER_CONFIGS in /kubric/challenges/movi/movi_f.py: MoviFConfig( name="512x512", description="Full resolution of 512x512", height=512, width=512, num_frames=120, validation_ratio=0.025, train_val_path="./generated_data", test_split_paths={ } ) For some key-value reading errors that occurred, I handled them following the approach in the movi_e.py code.

Finally, I made the following modifications to the create_point_tracking_dataset function in dataset.py:
import sys sys.path.append(r'/kubric') import challenges.movi.movi_f ds = tfds.load( 'movi_f', data_dir=r"/kubric/my_data", shuffle_files=shuffle_buffer_size is not None, # download=False, **kwargs) I had to comment out the download=False line, otherwise, I would get an error. With this change, I was able to generate the tfrecord and annotations files.

I would like to ask if there are any mistakes in my modifications. Additionally, I noticed that the co-tracker2 tfrecord files were directly downloaded. The movi_f-train.tfrecord file has 1024 records, and the movi_f-validation.tfrecord file has 128 records. My question is whether it is possible, like the co-tracker2 setup, to use the movi_def_worker_long_kubric.py script to generate 1024 video samples, convert them into tfrecord, and then use dataset.py to generate 10k annotations?Thank you for your guidance!