dpo_ov7b.sh imports data_processing which is missing

[bluesky333]

In train_dpo.py line 41, it imports data_processing which is removed in the latest commit. from data_processing.utils import load_jsonl, load_json ModuleNotFoundError: No module named 'data_processing'

[xushilin1]

+1

[Liuziyu77]

same issue. Any solution?

[jbjeong91]

+1

[tobiapoppi]

+1

just remove the import line and add these two functions

def load_jsonl(filepath):
    """Load a JSONL file and return a list of parsed JSON objects."""
    data = []
    with open(filepath, 'r', encoding='utf-8') as f:
        for line in f:
            try:
                json_object = json.loads(line.strip())
                data.append(json_object)
            except json.JSONDecodeError as e:
                print(f"Error decoding JSON on line: {line.strip()} - {e}")
    return data


def load_json(filepath):
    """Load a JSON file and return the parsed data."""
    with open(filepath, 'r', encoding='utf-8') as f:
        return json.load(f)