Detr module doesn't work out of the box

[rdilip]

There are a variety of issues with the code suggested in the README of the detr baseline. Some of these are resolvable, but it feels like this should be updated accordingly. I haven't yet gotten the detr implementation to train successfully, so I'm not able to make a pull request yet. Would someone be able to assist in getting a minimal version up and running?

Here the changes I've made thus far to get rid of errors.

1. flax needs to be set to version 0.5.3, so flax.optim is available. (Probably it should be updated to work with optax).
2. There is no call to builder.download_and_prepare in scenic/projects/baselines/detr/input_pipeline_detection.py (line 169), so the coco dataset isn't downloaded.
3. By default, the scenic package looks for a file instances_val2017.json in the package installation directory itself, rather than allowing the user to specify it. This becomes a bit annoying if you pip install as the README on the scenic github suggests. I think this could be fixed by adding a keyword argument annotations_loc in the call to DetrGlobalEvaluator (line 418 in trainer.py), maybe by adding annotations_loc to the config file? Either way, it would be ideal if scenic just automatically downloaded the annotations file, since right now it isn't specified anywhere.
4. The training loop asks for a meta_data property, but doesn't provide it. I think this is because this version of the project is still dependent on train_lib_deprecated; switching to train_lib seems to solve this.
5. The compilation of the module pmap_train_step takes a very long time, and eventually just gets killed.

As a general comment, it feels like the detr codebase has a lot of internal dependencies within scenic that lead to breaking changes. I think it would be helpful if there was a minimal file just to get spun up (perhaps getting rid of a lot of the training loop logic, since that seems to be where a lot of the errors arise and I think most people looking to extend would write custom training loops anyway). If someone can help me just get a basic version running I'd love to make the changes + do a pull request.

[MostafaDehghani]

Thank you @rdilip for your feedback and for listing all the issues. We are aware of almost all these and planning to fix them up, but it'll take some time (hopefully these will be fixed before end of the year). In the meantime, we open-sourced the Deformable DETR, in which many of these issues are resolved. I'll keep this issue open and will ping back when these are fixed.

Thanks again for using scenic and to care enough to spend time and write detailed comment for us.

[MasterSkepticista]

@MostafaDehghani Are you accepting community PR contributions for this? Edit: @rdilip I have created #1062 addressing all these. Understand this thread is stale, but you may give it a try if you like.