mmaction2
mmaction2 copied to clipboard
open-mmlab
Maybe we should support wild backbones/models ?
Describe the feature
Add a wild_backbones folders that stores the backbones directly copied from the original papers/repositories. They are just about to work and could be maintained by any contributors.
Motivation I personally have sucessfully run the I3D, X3D, MViT, MViT2, VideoMAE as the backbones with the mmaction2 by intergrating the codes from the original repos. I want to share my codes because I was once eager to these backbones. But my implementations is just about to work, and defitenly do NOT meet the mmaction code requirement.
Therefore, I think it would be very meaningful to have wild_backbones that allow contributors to easily share their implementations without the need of following the strict code requirement.
@makecent Thanks for your suggestions. We will consider this. Are you interested to contribute?
@makecent Thanks for your suggestions. We will consider this. Are you interested to contribute?
I'd like to share the backbones implemented by myself. While I am not confident on how to structure the wild_backbone.
The initial idea would be just add a folder named wild_backbones under mmaction/models, which contains some .py files, e.g., MViT.py, just like the backbone folder.
@makecent I am thinking about two possible folder structures. Not sure which one is better, could you have a look?
Option 1: the contrib folder is under the models folder, together with the backbones,heads, etc folders, so that we can have third party codes for different functions.
├── models
│ ├── backbones
│ ├── heads
│ ├── contrib
│ │ ├── __init__.py
│ │ ├── MViT.py
Option 2: each folder under the models folder can have a contrib folder if any:
mmaction
├── models
│ ├── backbones
│ │ ├── c3d.py
│ │ ├── contrib
│ │ │ ├── __init__.py
│ │ │ ├── MViT.py
│ ├── heads
@hukkai I am in favor of the option1 because it's simple. Furthermore, in some cases it may be NOT good to put a trhid party codes in a single py file. The dependency subfolder could also be put in the contrib folder.
For instance, we can implement ALL slowfast models by adding a slowfast package under the models/contrib. This slowfast pacakage only contains necessary py files to build slowfast models, and the slowfast config files. After that, we can easily build any slow models with mmaction2.
I will create a demo PR for this.
I just made a demo PR. It allows us to use all slowfast models.
To use them, the package pytorchvideo and fvcore are required.
@makecent It seems we cannot merge the codes with
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.... I need to talk with some seniors about this.
@makecent I think we can begin with the MViT model and make the inference available. I will work on this in the following two days.