llm-foundry icon indicating copy to clipboard operation
llm-foundry copied to clipboard

log eval dataset misconfiguration

Open milocress opened this issue 9 months ago • 1 comments

Make Eval Dataset Misconfiguration Errors Visible through Mosaic Logger

Wraps eval dataset creation with a mosaic logger try/catch.

In Train context

mpt-125m-chinchilla-regression-6OSoWo (the log trace indicates that the error is caught and the context attribute is set)

[rank7]: │   623 │   │   if mosaicml_logger is not None:                                │
[rank7]: │   624 │   │   │   e.context = 'TrainContext'                                 │
[rank7]: │   625 │   │   │   mosaicml_logger.log_exception(e)                           │
[rank7]: │ ❱ 626 │   │   raise e 

In Eval context

mpt-125m-chinchilla-regression-6OSoWo (the log trace indicates that the error is caught and the context attribute is set)

[rank0]: │   656 │   │   │   if mosaicml_logger is not None:                            │
[rank0]: │   657 │   │   │   │   e.context = 'EvalContext'                              │
[rank0]: │   658 │   │   │   │   mosaicml_logger.log_exception(e)                       │
[rank0]: │ ❱ 659 │   │   │   raise e  

milocress avatar May 07 '24 00:05 milocress

Flagging that the dataset type won't be added in the convert_delta_to_json.py and convert_text_to_mds.py. I think we would have to parse the datapath to get the train split to wrap the error if you'd like to take a stab! Also happy to do it :) thanks so much Milo!

I am happy to do this, but let's split it into a separate PR so this one stays small.

milocress avatar May 13 '24 16:05 milocress