Assign static IDs to DTS E2E tests to be able to add them to OSFV Dashboard

[DaniilKl]

The problem you're addressing (if any)

Currently DTS E2E tests are being generated dynamically according to the template https://github.com/Dasharo/open-source-firmware-validation/blob/d326bfb29d5a3fb374d4c728efb786b61aa8eb2a/dts/dts-e2e.robot#L23

The tests IDs are generated dynamically too. While it is still possible to identify the test case by its name, the OSFV Dashboard needs IDs.

Describe the solution you'd like

The DTS E2E test is being defined by the three fields:

${platform}    ${workflow}    ${release}

Every of the three fields are being defined in the platform-configs/*.robot files. Can we define the IDs in the same place, so every combination possible of the three fields above will have a unique ID?

Where is the value to a user, and who might that user be?

We could:

• Unify the tests results storage and presentation with Dasharo via OSFV Dashboard.
• Present the tests to clients via Dasharo/osfv-results or OSFV Dashboard.

Describe alternatives you've considered

Using DTS E2E tests names insted of the IDs in the OSFV Dashboard.

Additional context

No response

[DaniilKl]

@m-iwanicki, what do you think?

[m-iwanicki]

Does ID has to have some specific format? We could use hash of ${platform}, ${workflow}, ${release}. While there exists chance of collision it's very unlikely unless we use very short hash or trim sha256

There is also possibility of hardcoding those values e.g.

E2E<${platform_id}><${workflow_id}><${release_id}>:

[DaniilKl]

The ID format is defined by OSFV Dashboard, check https://github.com/Dasharo/open-source-firmware-validation/blob/develop/docs/adding-and-naming-test-cases.md .

[m-iwanicki]

It's a three digit number with leading zeros. Identifies test cases in a test suite. Typically test cases are numbered incrementally starting from 001.

The biggest problem here might be three digit number. We won't be able to use E2E<${platform_id}><${workflow_id}><${release_id}> format in that case as I'm pretty sure we have more than 10 platforms and workflows (or soon we will have). I need to think on it a little longer as I'm unsure how to resolve it.

[DaniilKl]

It's a three digit number with leading zeros. Identifies test cases in a test suite. Typically test cases are numbered incrementally starting from 001.

The biggest problem here might be three digit number. We won't be able to use E2E<${platform_id}><${workflow_id}><${release_id}> format in that case as I'm pretty sure we have more than 10 platforms and workflows (or soon we will have). I need to think on it a little longer as I'm unsure how to resolve it.

Why not use the ID incrementally without dividing it into <${platform_id}><${workflow_id}><${release_id}>, is there other advantages than better IDs interpretation by human? We would have 1000 combinations then.

[m-iwanicki]

Why not use the ID incrementally

Because we would need to keep list or json with mappings between test name and id. We could do it but maybe another way would be better.

Use 2 digits for platform, 2 digits for workflow and 1 digit for release e.g.:

E2E<${platform_id}><${release_id}>.2<${workflow_id}>.

Not sure if even this will be allowed as:

All the environment_ids of all the tested environments are defined as Robot Framework variables in the os-config/environment-test-ids.py file.

and in this file:

# 2xx - Linux
ENV_ID_UBUNTU = "201"
ENV_ID_FEDORA = "202"
ENV_ID_QUBES = "203"
ENV_ID_TRENCHBOOT = "204"
ENV_ID_XCP_NG = "205"

So I guess those xx are also reserved

[DaniilKl]

@BeataZdunczyk, can we use E2E<${platform_id}><${release_id}>.2<${workflow_id}> or E2E<${platform_id}><${release_id}>.0<${workflow_id}> in the way @m-iwanicki, described it above?

[m-iwanicki]

Currently template test names are made up of 3 parts:

${platform} ${workflow} - ${release}

• platform - platform we are emulating during test, it's the robot filename in platform-configs e.g. novacusom-nv41pz (without extension)
• workflow - easy to understand name of what we are testing e.g. SeaBIOS Update, UEFI->Heads Transition, etc.
• release - either DCR or DPP. The only difference is whether we are inputting DPP keys at the beginning of workflow

Example implementation of IDs: Lets say IDs are made up of 6 digits: xyz.abc

• In OSFV there are platform-config files, in each platform file we will set DTS_PLATFORM_ID to unique value. This value will be used as first 2 digits (xy)
• In default.robot we define all possible releases. Each release would have it's own unique ID e.g. DTS_RELEASE_ID. This will be used as 3rd digit (z). Currently we only support 2 releases (DCR and DPP)
• In default.robot we define all possible workflows. Each workflow would have it's own unique ID e.g. DTS_WORKFLOW_ID. This will be used as last 2 digits in the second ID part (bc). Currently we support 8 workflows.

To make sure there are no duplicate IDs template tests should have different suite_id e.g. E2ET from normal tests were IDs are hard-coded.

---

Another way: Create file with list of IDs matching test name e.g.:

001.201: <platform> <workflow> - <release1>
002.201: <platform> <workflow> - <release2>

When generating template test, find a line that contains test name (after <ID>:) and prepend found ID.

[DaniilKl]

@macpijan, do you approve the idea?

[3mkusiak]

@macpijan ping

[macpijan]

In default.robot we define all possible releases. Each release would have it's own unique ID e.g. DTS_RELEASE_ID. This will be used as 3rd digit (z). Currently we only support 2 releases (DCR and DPP)

According to the latest naming convention, this would be edition: https://docs.dasharo.com/dasharo-naming-convention/#description with DCP, DPP, DEP allowed values (where only the first two ones make sense right now for DTS).

Lets say IDs are made up of 6 digits: xyz.abc

I think a is not covered in this proposal.

This proposal is so much different from the ID convention used for Dasharo tests in OSFV, that we need a separate one for DTS anyway it seems. So let's just keep the DTS tests and ID separate, and document what the ID convention means (if we want it to be meaningful).

If we wanted to be a bit closer to what we have for Dasharo tests, I would swap your proposal. So first you would have a workflow (case ID), then the environment, which in case of the DTS, is a combination of platform + release edition.

But since these should be kept separated anyway, we can stick to what suits better from the DTS testing perspective.

[DaniilKl]

Lets develop our own naming convention then.

Templated E2E tests

E2ETXYZ, where:

• X is a 3-positional number for platform.
• Y is a 2-positional number for workflow.
• Z is a 1-positional number for release.

Other E2E tests

E2EX.Y, where X is a 3-positional number for a test group ID (e.g. a group for FUM tests or a group for credentials tests), and Y is a 3-positional number for a test in the group ID.

@m-iwanicki, what do you think?

[m-iwanicki]

Why do you want to change other E2E tests? The only problem is with template ones.

[DaniilKl]

Why do you want to change other E2E tests? The only problem is with template ones.

Other tests already adhere to the convention I explained in my comment, e.g.:

E2E007.001 Check credentials are being saved correctly

So there is nothing to change for them. I just wanted to explicitly state it, so it will be clear, that it should be added to the DTS tests naming convention.