Rasmus

I think the example conversion script is perhaps not very good. One thing that helps a lot is to use the Datasets `.map()` to batch tokenize the dataset. I'm not...

> Hi @rlrs ! Could you share the script to transform the weights from HF to dcp? Thanks! I'm using a modified script based on gpt-fast, will paste it here....

Thanks for asking over there. I didn't try to download the weights from anywhere other than HF, but I would be a bit surprised if there's some simple transformation you...

As discussed in the HF issue, there is indeed a permutation of the weights that causes the two implementations to be equivalent. I don't believe anything needs to be done...