Issue

When the experiment yaml file sets merge_with_nonprivate: true, fuzzbench produces an error message saying "Error generating HTML report".

An experiment named 2022-03-08-11-02-07 shows the existence of this issue with merge_with_nonprivate: true. Another experiment, named 2022-03-09-14-01-27-dongge proves this error can be avoided by setting merge_with_nonprivate: false.

Traceback

"Traceback (most recent call last):
  File "/work/src/experiment/reporter.py", line 73, in output_report
    generate_report.generate_report(
  File "/work/src/analysis/generate_report.py", line 244, in generate_report
    detailed_report = rendering.render_report(experiment_ctx, template,
  File "/work/src/analysis/rendering.py", line 48, in render_report
    return template.render(experiment=experiment_results,
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/local/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "/work/src/analysis/report_templates/default.html", line 68, in top-level template code
    {% if experiment.rank_by_median_and_average_rank.size < 2 %}
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 471, in getattr
    return getattr(obj, attribute)
  File "/work/src/analysis/experiment_results.py", line 287, in rank_by_median_and_average_rank
    return self._ranking(data_utils.benchmark_rank_by_median,
  File "/work/src/analysis/experiment_results.py", line 266, in _ranking
    key=self._relevant_column),
  File "/work/src/analysis/experiment_results.py", line 152, in _relevant_column
    return 'edges_covered' if self.type == 'code' else 'bugs_covered'
  File "/work/src/analysis/experiment_results.py", line 145, in type
    raise ValueError(
ValueError: Cannot mix bug benchmarks with code coverage benchmarks.
"