mmtracking
mmtracking copied to clipboard
Published
20 hours ago •
open-mmlab
open-mmlab
[Community] Add `IDOL`, `VITA` to project
Hi,
Thank you for the fantastic work on developing Mask2Former, IDOL, and VITA using mmtracking. I can successfully run Mask2Former. However, I run into some issues running IDOL and VITA.
For IDOL, I get errors during the training stage. And I get the following error:
Traceback (most recent call last):
File "tools/train.py", line 119, in <module>
main()
File "tools/train.py", line 115, in main
runner.train()
File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1678, in train
model = self.train_loop.run() # type: ignore
File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/runner/loops.py", line 264, in run
self.run_iter(data_batch)
File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/runner/loops.py", line 287, in run_iter
outputs = self.runner.model.train_step(
File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/model/base_model/base_model.py", line 114, in train_step
losses = self._run_forward(data, mode='loss') # type: ignore
File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/model/base_model/base_model.py", line 314, in _run_forward
results = self(**data, mode=mode)
File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/supreeth/mmtracking_vis_sota/mmtrack/models/mot/base.py", line 106, in forward
return self.loss(inputs, data_samples, **kwargs)
File "/home/supreeth/mmtracking_vis_sota/projects/VIS_SOTA/IDOL/idol_src/idol.py", line 53, in loss
losses = self.track_head.loss(x, data_samples)
File "/home/supreeth/mmtracking_vis_sota/projects/VIS_SOTA/IDOL/idol_src/models/idol_query_track_head.py", line 488, in loss
losses = self.loss_by_feat(*loss_inputs)
File "/home/supreeth/mmtracking_vis_sota/projects/VIS_SOTA/IDOL/idol_src/models/idol_query_track_head.py", line 534, in loss_by_feat
losses_mask, losses_dice, min_cost_qids = multi_apply(
File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmdet/models/utils/misc.py", line 219, in multi_apply
return tuple(map(list, zip(*map_results)))
File "/home/supreeth/mmtracking_vis_sota/projects/VIS_SOTA/IDOL/idol_src/models/idol_query_track_head.py", line 681, in loss_by_feat_single
loss_dice = self.loss_dice(
File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmdet/models/losses/dice_loss.py", line 137, in forward
loss = self.loss_weight * dice_loss(
File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmdet/models/losses/dice_loss.py", line 47, in dice_loss
a = torch.sum(input * target, 1)
RuntimeError: The size of tensor a (8) must match the size of tensor b (600) at non-singleton dimension 0
Regarding VITA, I can train the model successfully. However, during evaluation, I get the following error:
Traceback (most recent call last):
File "tools/test.py", line 88, in <module>
main()
File "tools/test.py", line 84, in main
runner.test()
File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1724, in test
metrics = self.test_loop.run() # type: ignore
File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/runner/loops.py", line 417, in run
self.run_iter(idx, data_batch)
File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
return func(*args, **kwargs)
File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/runner/loops.py", line 436, in run_iter
outputs = self.runner.model.test_step(data_batch)
File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/model/base_model/base_model.py", line 145, in test_step
return self._run_forward(data, mode='predict') # type: ignore
File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/model/base_model/base_model.py", line 314, in _run_forward
results = self(**data, mode=mode)
File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/supreeth/mmtracking_vis_sota/mmtrack/models/mot/base.py", line 108, in forward
return self.predict(inputs, data_samples, **kwargs)
File "/home/supreeth/mmtracking_vis_sota/projects/VIS_SOTA/VITA/vita_src/vita.py", line 106, in predict
_frame_queries, _mask_features = self.seg_head.predict(feats)
File "/home/supreeth/mmtracking_vis_sota/projects/VIS_SOTA/VITA/vita_src/models/vita_seg_head.py", line 217, in predict
all_frame_queries, mask_features = self(x)
ValueError: too many values to unpack (expected 2)
Can you please help me with this @pixeli99?
Thanks!
I'm glad you are interested in this part of code, but unfortunately, the current code support for VITA is not complete, which will be repaired after the GenVIS reproduction is completed. If you need to do research, you can try the official repo first. The reproduction is expected to be completed in two weeks ✅ In addition, if you have any suggestions on the code or want to join me, I am also very welcome ！😺
Thank you for the swift reply. That's great to know! I will checkout the official code repo once. I will let you know if I have any suggestions on the code!
Hi @pixeli99 ! Is there any update on the IDOL/ VITA code yet? I am looking forward to trying them! Thanks!