Support Swin Transformer Model

[tanzhenyu]

Related to https://github.com/keras-team/keras-cv/issues/668, this is another transformer based model which we'd like to be supported. For simplicity, focus on only classification task, having in mind that future detection tasks might rely on it.

For contributors, please follow the contribution guide.

And specifically, on the API Design Guide, we'd like to see a brief description of how the model would look like before implementation, preferrably conforming to the tf.keras.application style.

[ariG23498]

I would like to work on this!

[ariG23498]

Hey @tanzhenyu

I got around reading the contriution guideline and the API design document. I was unable to find a guide to contribute a model inside of keras_cv.

My main doubts:

• Do we need to port the weights of the model (here SwinT in PyTorch)? If yes (which I hope should be the case), how would I do that?
• Do we also need to validate the weight porting process by validating it wrt the task (classification here) on a benchmark dataset?
• How do we check the correctness of the model?

[bhack]

Hey @tanzhenyu

I got around reading the contriution guideline and the API design document. I was unable to find a guide to contribute a model inside of keras_cv.

My main doubts:

• Do we need to port the weights of the model (here SwinT in PyTorch)? If yes (which I hope should be the case), how would I do that?
• Do we also need to validate the weight porting process by validating it wrt the task (classification here) on a benchmark dataset?
• How do we check the correctness of the model?

It would be useful to have something like: https://github.com/keras-team/keras/blob/master/CONTRIBUTING.md#contributing-to-keras-applications

[bhack]

Also, do we have a duplicate ticket with https://github.com/keras-team/keras-cv/issues/636 ?

[tanzhenyu]

Hey @tanzhenyu

I got around reading the contriution guideline and the API design document. I was unable to find a guide to contribute a model inside of keras_cv.

My main doubts:

• Do we need to port the weights of the model (here SwinT in PyTorch)? If yes (which I hope should be the case), how would I do that?
• Do we also need to validate the weight porting process by validating it wrt the task (classification here) on a benchmark dataset?
• How do we check the correctness of the model?

Yes we do expect porting the weights and verifying the results against the original paper. For the guide, @bhack 's reference is most relevant.

[ayulockin]

@ariG23498 will you mind if I join you on this PR?

[ariG23498]

I have no issues collaborating with you on this one!

[bhack]

Yes we do expect porting the weights and verifying the results against the original paper. For the guide, @bhack 's reference is most relevant.

We have @sayakpaul https://github.com/sayakpaul/swin-transformers-tf for this

Then we need to build the training phase.

[DavidLandup0]

To do transformer-based models we'll need new layers (reusable) and blocks (might be reusable). KerasCV is currently missing blocks to rapidly build transformer-based models. Might be worth taking a detour to work on those components first

[ayulockin]

I agree with @DavidLandup0.

[tanmaypandey7]

Hi! Are you guys @ayulockin and @ariG23498 still working on this? I would like to help if it's alright.

[IMvision12]

Is Anyone working on this??

[github-actions[bot]]

This issue is stale because it has been open for 180 days with no activity. It will be closed if no further activity occurs. Thank you.