modelcards
modelcards copied to clipboard
Handle invalid model-index instead of throwing data away
Right now, if model-index
is not formatted correctly in the remote repo when using ModelCard.load(repo_id)
, it will tell you the model-index is invalid and that we're not loading eval results.
For context, under the hood modelcards.CardData
uses a list of EvalResult
to represent the model index, which is stored at CardData.eval_results
. When you turn card data to dict, it doesn't return the list of EvalResult
, but instead formats it to a valid model index and assigns it to the model-index
key of the metadata dict.
The code can be seen here:
https://github.com/nateraw/modelcards/blob/a45b85bab516531813b0f80db4399a56de4447b9/modelcards/card_data.py#L151-L167
So, maybe in the case that the found 'model-index' is invalid, we assign it to card_data['model-index']
so it can still be accessed. If the user tries to update the eval_results
, we'll use that, but if not, the previous model-index
will still be in tact.
Perhaps we can still create the eval_results
if formatted somewhat correctly (throwing a warning that its invalid). This way it could make it easier for a user to fix invalid model-index in their existing cards.