run code at about [ 3170/125000], [W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]

[Emibobo]

I20240724 09:37:05 4020743 dino_fl helpers.py:102] Training [ 3490/125000] eta: 14:39:00 lr: 0.0000 (0.0000) wd: 0.0407 (0.0402) mom: 0.9920 (0.9920) last_layer_lr: 0.0000 (0.0000) current_batch_size: 2.0000 (2.0000) total_loss: 0.0078 (2.4457) stage_local_mse_loss: 0.0039 (0.0062) stage_global_mse_loss: 0.0039 (0.0062) dino_local_crops_loss: 9.8579 (9.6866) dino_global_crops_loss: 1.2324 (1.2248) ibot_loss: 1.8275 (2.2796) time: 0.400124 data: 0.003072 max mem: 14504 [W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]