`make_report`: how to handle manually written files

[dberenbaum]

In https://github.com/iterative/example-repos-dev/pull/117, inserting a live.make_report() call will fail because there is a manually written precision_recall.json file in the live dir:

$ python src/evaluate.py model.pkl data/features
INFO:dvclive:Report path (if generated): /Users/dave/Code/example-repos-dev/example-get-started/build/example-get-started/evaluation/train/report.html
Traceback (most recent call last):
  File "src/evaluate.py", line 76, in <module>
    evaluate(model, train, "train")
  File "src/evaluate.py", line 62, in evaluate
    live.make_report()
  File "/Users/dave/Code/dvclive/dvclive/live.py", line 195, in make_report
    html_report(self.dir, self.summary_path, self.html_path)
  File "/Users/dave/Code/dvclive/dvclive/report.py", line 77, in html_report
    renderers.extend(get_plot_renderers(dvclive_path / Plot.subfolder))
  File "/Users/dave/Code/dvclive/dvclive/report.py", line 51, in get_plot_renderers
    row["rev"] = "workspace"
TypeError: 'str' object does not support item assignment

I think this can be resolved by renaming the file to prc.json or the top-level JSON key to precision_recall, but dvclive should probably handle this kind of error more gracefully. Maybe it can skip that plot but still generate the rest of the report?

[daavoo]

but dvclive should probably handle this kind of error more gracefully. Maybe it can skip that plot but still generate the rest of the report?

I cut corners based on the assumption that DVCLive would be the one logging plots so we would control the format.

I am not sure about enabling people to manually put files inside the output folder. If we want to allow it, we need to replicate the parsing used in DVC (https://github.com/iterative/dvc/blob/main/dvc/render/converter/vega.py#L62) which allows for arbitrary top-level keys in JSON.

[dberenbaum]

Putting as p2 for now. Feel free to follow up if you disagree.