Cal Mitchell

I do not need streaming, this unblocks my use case.

@SLR722 thank you for the basic copy/paste error finds. @RdoubleA, I'll address your questions and comments one by one: > How do you plan to extend packing functionality to all...

@ebsmothers yes, it is most likely an issue of how I'm saving the dataset, but I believe the [`save_to_disk()`](https://huggingface.co/docs/datasets/en/package_reference/main_classes#datasets.Dataset.save_to_disk) function is quite popular in the HF ecosystem. Whenever I try...

@ebsmothers @RdoubleA @kartikayk, thank you all very much for taking my use case and problems into account! I really appreciate it. Since you are all asking my opinion on various...

Closing this PR in favor of #875

The PR from @vinjn above ^^ worked for me.

Copying and pasting my response to your original issue: > Hi, I am having this same issue. However, the code in question is in the Huggingface Transformers lib, not in...

After some initial research, the functionality I'm imagining is already implemented for Llama 2 in the [`convert_llama_weights_to_hf()`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py) function in HF Transformers. I [opened an issue to add support for Llama...

Looks like [someone at HF is already working on converting that script to support Llama 3](https://github.com/huggingface/transformers/pull/30334).

@kartikayk, thanks for the quick follow up. A peft integration would be very handy, but the other thing is more pressing for me right now. I did come across that...