llm-foundry
llm-foundry copied to clipboard
log eval dataset misconfiguration
Make Eval Dataset Misconfiguration Errors Visible through Mosaic Logger
Wraps eval dataset creation with a mosaic logger try/catch.
In Train context
mpt-125m-chinchilla-regression-6OSoWo
(the log trace indicates that the error is caught and the context attribute is set)
[rank7]: │ 623 │ │ if mosaicml_logger is not None: │
[rank7]: │ 624 │ │ │ e.context = 'TrainContext' │
[rank7]: │ 625 │ │ │ mosaicml_logger.log_exception(e) │
[rank7]: │ ❱ 626 │ │ raise e
In Eval context
mpt-125m-chinchilla-regression-6OSoWo
(the log trace indicates that the error is caught and the context attribute is set)
[rank0]: │ 656 │ │ │ if mosaicml_logger is not None: │
[rank0]: │ 657 │ │ │ │ e.context = 'EvalContext' │
[rank0]: │ 658 │ │ │ │ mosaicml_logger.log_exception(e) │
[rank0]: │ ❱ 659 │ │ │ raise e
Flagging that the dataset type won't be added in the
convert_delta_to_json.py
andconvert_text_to_mds.py
. I think we would have to parse the datapath to get the train split to wrap the error if you'd like to take a stab! Also happy to do it :) thanks so much Milo!
I am happy to do this, but let's split it into a separate PR so this one stays small.