Validation errors are not shown correctly during autograding

[tmetzl]

nbgrader_version=0.9.3 and older

When autograding a notebook with metadata errors, the error messages are not helpful and/or misleading. Here is an example notebook with a duplicate grade_id:

from nbformat.v4 import new_notebook, new_markdown_cell
from nbgrader.preprocessors import CheckCellMetadata
from nbgrader.nbgraderformat import MetadataValidator

nb = new_notebook()

cell = new_markdown_cell(
    metadata=dict(
        nbgrader=dict(
            grade_id="my_id",
            grade=False,
            solution=False,
            locked=True,
            schema_version=3
        )
    )
)

nb.cells.append(
    cell
)

nb.cells.append(
    cell
)

Now when the MetadataValidator is used to validate the notebook there is a descriptive error message that gives the correct information about the error:

MetadataValidator().validate_nb(nb)

Output:

---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
---> 27 MetadataValidator().validate_nb(nb)

File /opt/conda/lib/python3.11/site-packages/nbgrader/nbgraderformat/v3.py:110, in MetadataValidatorV3.validate_nb(self, nb)
    108 grade_id = cell.metadata['nbgrader']['grade_id']
    109 if grade_id in ids:
--> 110     raise ValidationError("Duplicate grade id: {}".format(grade_id))
    111 ids.add(grade_id)

ValidationError: Duplicate grade id: my_id

When the CheckCellMetadata preprocessor is used the error is undescriptive:

CheckCellMetadata().preprocess(nb, {})

Output:

---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
---> 28 CheckCellMetadata().preprocess(nb, {})

File /opt/conda/lib/python3.11/site-packages/nbgrader/preprocessors/checkcellmetadata.py:18, in CheckCellMetadata.preprocess(self, nb, resources)
     16     msg = "Notebook failed to validate; the nbgrader metadata may be corrupted."
     17     self.log.error(msg)
---> 18     raise ValidationError(msg)
     20 return nb, resources

ValidationError: Notebook failed to validate; the nbgrader metadata may be corrupted.

When autograding a notebook like this I actually get this misleading error originating from here: https://github.com/jupyter/nbgrader/blob/main/nbgrader/converters/base.py#L421

AutogradeApp | ERROR] One or more notebooks in the assignment use an old version 
    of the nbgrader metadata format. Please **back up your class files 
    directory** and then update the metadata using:
    
    nbgrader update .

This might be related to #1629.

[shreve]

So this exception is actually correct, but is misleading because there's some undefined behavior underneath. When there are duplicate grade_ids, the nbgrader metadata object gets cleared out. I keep encountering this issue and need to track down the root issue.

In the latest occurrence, a student duplicated a grade cell, so in the submitted notebook we have two cells with the following metadata:

...
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "51cd13ae2d9fef48072f6fe9ff4af3fc",
     "grade": false,
     "grade_id": "logs",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
...
   "metadata": {
    "nbgrader": {
     "cell_type": "code",
     "checksum": "51cd13ae2d9fef48072f6fe9ff4af3fc",
     "grade": false,
     "grade_id": "logs",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
...

But then by the time we run MetadataValidator().validate_nb(nb), the first cell only contains the following metadata:

    "metadata": {
        "deletable": false,
        "nbgrader": {}
    }

This results in a SchemaTooOldError being raised because metadata["nbgrader"].get("schema_version", 0) < 3.

Still looking into why this is the case. Hopefully a PR will follow soon.