llm-foundry icon indicating copy to clipboard operation
llm-foundry copied to clipboard

Published 20 hours ago • verified publisher icon mosaicml

`convert_dataset_json.py` does not respect parameter `--split`

Open matthiasgeihs opened this issue 1 year ago • 1 comments

https://github.com/mosaicml/llm-foundry/blob/bd8127252c660e45ed01413645d29427f86c085a/scripts/data_prep/convert_dataset_json.py#L204C4-L204C4

out=os.path.join(args.out_root), should be out=os.path.join(args.out_root, folder_split), as in convert_dataset_hf.py.

matthiasgeihs avatar Aug 01 '23 09:08 matthiasgeihs

I will work on fixing this now

tbarton16 avatar Aug 02 '23 22:08 tbarton16