Zhiyuan Li

Please try to squash merge :)

Your suggestion makes a lot of sense. Some of these changes were introduced by the edittor. I'll try to first limit the changes to chunkrwkv6 and fix the test

[checkrwkv6.tar.gz](https://github.com/user-attachments/files/16608127/checkrwkv6.tar.gz) Here are the codes that compare CUDA with FLA.

Also, this pull request fixed https://github.com/sustcsonglin/flash-linear-attention/issues/29 The problem was introduced by bfloat16 when calculating dq and dk. By converting to float32 when necessary and using tf32 as much as possible,...

> @uniartisan Hi, just make some reviews, could you have a check? hi. I can't see any comments, could you tell me where could I have a check?

 Could you give me a review like this? https://github.com/sustcsonglin/flash-linear-attention/pull/44/files/4a3e2bb1d699c7e41ead7adc2f2403fb3e79ceb6 I can't see your msgs :(

> @uniartisan Can you see my updated comments between the lines? Sorry, I don't know what's going on. I still cannot see you review comments. Maybe you can directly post...

@yzhangcs Hello, I hope finds you well. I have successfully synchronized all the latest changes to your project. Given your expertise and valuable insights, I was wondering if you could...

> Hi, can you authorize this branch to me so that I can make some updates Of course!!! Sorry for my late reply. I will try it :)

You can try： ```python from transformers.modeling_utils import no_init_weights with no_init_weights(): model = AutoModelForCausalLM.from_pretrained() ``` And I will fetch up to bo's init to RWKV7 in the next few days