Axel Mendoza

Ok thanks for the response. I see a command line example to train on custom temporal data: ```bash python src/train.py with \ deformable \ tracking \ mot17 \ full_res \...

After trying this, I faced an assert here: https://github.com/timmeinhardt/trackformer/blob/686fe60858e4f61497a4544b95f94487614d0741/src/trackformer/datasets/crowdhuman.py#L12 I fixed it by changing `train_split=train` to `crowdhuman_train_split=train`. Then I noticed that by leaving `mot_path` to the default `data/MOT17` it was...

I got rid of the `full_res` option and I was able to perform 50 iterations. ```bash python src/train.py with \ deformable \ tracking \ crowdhuman \ resume=models/mot17_train_deformable_private/checkpoint.pth \ output_dir=models/boxy_train_deformable \...

By the way, is it possible to perform the validation on static data ? (on CrowdHuman like dataset)

This is the stack trace: ```bash Traceback (most recent call last): File "/home/bigbro/anaconda3/envs/trackformer/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/home/bigbro/anaconda3/envs/trackformer/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/bigbro/.vscode/extensions/ms-python.python-2021.5.842923320/pythonFiles/lib/python/debugpy/__main__.py", line...

Unfortunately, it occurs after a few iterations. Ok I will try to debug the code in this file. I discovered that in Visdom, my training images seemed cut. Apparently is...

> You should debug the data flow and check if everything works as when training on CrowdHuman. For example is this code after the if statement executed properly: > >...

I noticed that when I face the assert, the `out_bbox` as well as `out_probs` contains Nans. I suppose it is the prediction of the model. https://github.com/timmeinhardt/trackformer/blob/686fe60858e4f61497a4544b95f94487614d0741/src/trackformer/models/matcher.py#L71 Same thing when I...

I noticed that the Nans first appears in the `query_embed` variable: https://github.com/timmeinhardt/trackformer/blob/686fe60858e4f61497a4544b95f94487614d0741/src/trackformer/models/deformable_detr.py#L167 Do you have any clue where this is updated ?

You were right ! I might forgot to set a lr to 0. When I comment the optimizer step, it successfully runs a whole epoch.