Support State Space Models

[abhinavkulkarni]

Hi @casper-hansen,

Please add support for state space models such as the recently released StripedHyena released by TogetherAI (and authors of Flash Attention 2). These models supposedly do well in really long contexts and are much easier to train and infer compared to transformers-only models such as Llama 2. Please note, that the StripedHyena model has some layers made up of SSM blocks whereas others utilize usual transformer blocks.

More details here: https://www.together.ai/blog/stripedhyena-7b

Thanks!

[casper-hansen]

Hi @abhinavkulkarni, thanks for posting this. I talked with the Striped Hyena team and I am looking to implement it. I have already started on a branch below, but needs more testing.

https://github.com/casper-hansen/AutoAWQ/tree/striped_hyena