Fangkai Jiao

Has the work of adding BERT done?

4

Hi, do you have finished the word of adding bert? Could you please share the results? Thank you very much!

The optimizer may have a bad performance on reading comprehension model?

4

Great thanks to your work!  The line with orange color is baseline using Adam as optimizer and the line with blue color is the baseline using AdaBound. I think...

Would you release the data on google drive?

3

Indeed, google cloud is not provided to developers in China so that I can't run the script with defalut storage space..... Would you release other solution for download the data?...

Problem for downloading data of reddit

4

Hi, great thanks to your contribution! I try to use `python demo.py --data full` to download the reddit data. For I don't want to train the model now I didn't...

Hello. It seems that the performance of slqa is worse than bidaf(with self-attention?). I wanna know that are there some problems implementation slqa? In my own implementation, it seems also...

There are several tricks to implement to improve the performance and they won't be added util Sep. 19 for some more important projects. I opened this issue to avoid forgetting...

Compatibility with Deepspeed, Fairscale, or Torch zero-redundancy optimizer

1

Wonderful work! May I know the compatibility with ZeRO mechanism? E.g., Torch redundancy optimizer, deepspeed zero-1 to zero-3, and fairscale FSDP. Becaused I noticed that QLoRA relies on particularly implemented...

I know that qlora can help train LLaMA-65B with less than 40GB memory. But I find that the quantization process should be completed on GPU, which means that you should...

Seems not compatible with DeepSpeed (perhaps also FSDP)

5

Hi, appreciate to your awesome work! When I trying to introduce GaLore AdamW optimizer to Gemma training, it seems that it is not compatible with deepspeed with Zero stage as...

Hi I have question. How could I use this parser if I wanna challenge the **Dialog State Tracking** task? I see your code while searching on github. The task definition...