Steven Walton

Here to confirm the issue still exists (more than a year after the report). Thank you Budlee for the fix.

Hi everyone, I just wanted to jump in here before this gets pushed, as one of the developers of NAT (@alihassanijr is the lead dev and may have additional comments)....

Not being a rust programmer, I take didn't think about this. Forgive me for my naivety, I'm sure I'm about to make it more apparent lol `curl https://sh.rustup.rs -sSf |...

Hey @leochand101, I don't work in this space anymore so unfortunately I can't give you too much advice. These tutorials were specifically made as I was learning Geant and trying...

Hi, thanks for bringing this up. There is an error in the code. I traced the issue with `pdb` and found that if [you look at kwargs here](https://github.com/SHI-Labs/Compact-Transformers/blob/main/src/cct.py#L146-L153) you'll see...

@alihassanijr I believe this was introduced in 5b21843. What was the reasoning behind this? Same issue is going to exist in the evaluate example. I'm not sure what the best...

We have not tried this, nor some of the other pre-training techniques. We'd be curious about your results. Our paper is more about demonstrating that ViTs can work on small...

I'm not quite sure what's going on without looking too closely, but you can see [here](https://github.com/SHI-Labs/Compact-Transformers/blob/main/src/utils/embedder.py#L26) that we basically only call torch's embedding which [expects longs](https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html). This just looks like...

https://medium.com/mlearning-ai/load-pre-trained-glove-embeddings-in-torch-nn-embedding-layer-in-under-2-minutes-f5af8f57416a

I'm sorry, I meant to get to this sooner. I have code to visualize the attention maps for both Swin and NAT located [here](https://github.com/SHI-Labs/StyleNAT/blob/main/src/analysis.py#L225). If you use these attention maps...