ui: show misconfigured tests repository in frontend
We already detect when processing the tests repository whether or not there are any problems, such as malformed json files or missing json files.
Here are some possible rules:
- Always required for each lab assignment:
assignment.json(top-level folder only) - Should we require one of:
criteria.jsonortests.json?
If a json file is malformed, then we should alert the teaching staff in their frontend.
Similar, if the above rules are not satisfied. Maybe we need to have an exclusion mechanism to avoid unnecessary alerts for cases like the internal folder that does not contain an assignment, but extra library functions etc.
We could also provide a notification to students that their committed push did not trigger any test execution of a known assignment. This is sometimes relevant when a change is pushed to the root folder or another folder without a corresponding tests/folder/assignment.json file. For example, if someone creates a new folder mylab which does not exist in the tests repository.
However, such notifications could be noisy for some courses/assignments, so we need to figure out a way to surface these issues without them becoming an annoyance. Maybe the exclusion mechanism can be used to silence notifications.