unitxt icon indicating copy to clipboard operation
unitxt copied to clipboard

LoadFromIBMCos does not work if datafile has '/' in name

Open yoavkatz opened this issue 1 year ago • 0 comments

data_dir=“tuning-data-cleared/ceramic/mixtures_02.26.2024/code/dolphin_coder”, data_files={“train”: “train/part0.jsonl”, “test”: “val/part0.jsonl”},

This causes failures.

Possible solution:

  1. Raise an error if '/' in data_files, but allow fusing of datasource (see #707)
  2. Handle '/' in paths. (simpler for users)

yoavkatz avatar Mar 25 '24 11:03 yoavkatz