core-v-verif

core-v-verif copied to clipboard

Symptoms: Working in parallel on vplans is very impractical. Reviewing the diff of vplan changes is difficult.

Cause: The excel vPlans are binary.

Background: The XIF vplan, and other vplan reviews before it. We want to change parts of the XIF vplan, but Thales might also be about to make changes to it, so how can we work simultaneously on it? We want to review vplan changes, but the file is binary and not easily diff'able.

Solutions?: Save as text-based odf spreadsheet? Use markdown instead? Something else?

This is a perennial topic. At the start of the CV32E40P we considered all these options. Markdown tables were rejected due to poor support for tables. ODF is an option, but people missed some of the features available with XLSX formats.

In the end, we opted to "waste" space on GitHub by having individual reviewers submit feedback as comments in the spreadsheet and submit them to GitHub with a unique filename (e.g. xif_dvplan_robin_review.xlsx). The reviewers updated the primary DVplan spreadsheet and deleted the others when the review was complete. Wasteful, but workable.

Ideally, I would like to use GoogleSheets for this with the spreadsheet on a GoogleDrive. the final version would then be moved to GitHub. Unfortunately, not all OpenHW Group members allow their employees to access GoogleDrive from the corporate network.

Hi @JeanRochCoulon. I wanted to continue the Mattermost discussion about excel vplan reviews. Is it ok to continue it here on github, so we have a later reference to the discussion?

Reviewing the DVPlan in an Excel file is not convenient. Modifications cannot be tracked and comments are not easy to read and modify. Moreover committing a binary file is not in the github spirit: too much space, no tracking, licence dependency,... Why not editing the DVplan in Yaml file and transform it into a html table? Some python library provides this feature. This would be better! Cheers - Jean-Roch Coulon

The idea of Yaml files is good! This bypasses the problem of "Markdown tables were rejected due to poor support for tables"? The "struct-like" items in a yaml files should map pretty much 1-1 to the columns we already have in spreadsheets?

Is there a way to "sanity check" a yaml vplan against a template? To ensure that it follows our standard format. This would also solve our problem of the vplans being very inconsistent in layout and formatting and use of cell values, multi-row merged cells etc.

What about "ODF is an option, but people missed some of the features available with XLSX formats"? What features are we talking about? Would people still missing these features if we go for the yaml solution?

@JeanRochCoulon, @MikeOpenHWGroup, I tried to think about how this would work in yaml.

Below is a fraction of the xif vplan written in yaml, just as an experiment to see how this goes. The following scheme should be scalable to arbitrary choices of multi-column cells etc with a lot of creative freedom while still maintaining a rigid structure. It is not super readable(?), but by using some sort of DOM editor (picture related) it might be usable, and I don't know how a rendered html table of this would help.

features:
- feature: issue interface
  sub features:
  - sub feature: issue_req signals stable
    feature descriptions:
    - feature description: >
        Interface/Issue interface chapter:
        The “instr” and “mode” signals remains stable
        during an issue Request transaction.
      verification goals:
      - verification goal: >
          For issue transaction which lasts one cycle,
          as signals change value on rising edge,
          the feature is always true.
      - verification goal: >
          For issue transaction which lasts more than one cycle,
          check that “mode” and “instr” cannot be modified
          by an instruction when transaction issue is in process
        pass/fail criterias:
          - pass/fail criteria: ...
  - sub feature: mode signal value
- feature: commit interface
  sub features:
  - sub feature: ...

Thanks for this @JeanRochCoulon. This is a good illustration. My views on this:

1. The effort required to write a DVplan in a Spreadsheet or YAML is about the same.
2. Reading a DVplan is much easier as a Spreadsheet.
3. Providing review feedback is about the same.
4. Revision control of the YAML format is much better.

So if each of those items is worth one point in a football game, the "Spreadsheet vs. YAML" is a tie. Let's try "Google Sheets" penalty kicks to see if we can break this tie.

The benefit of Google Sheets is that the document is on the Google cloud. We can keep it there during the DVplan writting phase and reviews. All versions of the file are always accessible and edits and review comments are always visible. Sheets provides reasonably good tools for reviews. When we are done with the review, a release version of the spreadsheet can be placed on GitHub.

Of course, the problem with this idea is that not all members of the OpenHW Group have equal access to Google Sheets.

Hello, by implementing a yaml2html converter, we obtain the following html :

I can provide the converter script if needed.

In the coming days, we will compare the spreadsheet to the yaml methodology, I keep you in touch !

Cheers

Hello @mike, @silabs-robin, @silabs-hfegran, Excel and Yaml have been tried in Thales/INVIA, the two methods present some limitations. That's why we are investigating another methodology based on an internal custom tool called Verification Plan Tools (VPTOOL). This tool is very appreciated by Thales verificators. But we have to confirm that this tool meets the CVA6 requirement. in this case, we could present it to know if it can also ease your dvplan edition. This tool can be shared it as open source tool. I will schedule a discussion point probably next week. Cheers

Closed as stale. (Might resume if it becomes relevant again.)