NickGao96 comments

Repositories
Issues
Comments

Results 2 comments of


                                            NickGao96

How is the contrastive data pipeline implemented?

> As mentioned in the [readme](https://github.com/CStanKonrad/long_llama/tree/main/fine_tuning#misc) the instruction fine-tuning does not use FoT. In fact, it can be thought of as a "modified" FoT with `cross_batch=1` because: > > *...

Pippy ddp2pipe example doesn't work for pipeline

I observed similar ERRORs with official examples, using python 3.8 / PyTroch 1.13 / CUDA11.6 / NVIDIA A100. Did you managed to solve this problem or find any candidate cause...