Test data for validation of existing message specifications

[treiher]

Test data for all existing specifications should be added. For each message of a specification file a set of valid and invalid messages should be added to validate the correctness of the message specification using the tool developed in Componolit/RecordFlux-specifications#7.

[rssen]

For packages that import other specifications but do not use messages from these packages as refinements, the coverage report will never output a 100% coverage (or anything > 0 for the imported specifications). For example, ARP (arp.rflx) imports ipv4.rflx and ethernet.rflx, but does not define a refinement arp -> ethernet (of cause not), but uses values defined in ethernet.rflx (e.g. Pro : Ethernet::Ether_Type). Therefore, the coverage report for arp.rflx looks like below. I think this report format is a bit confusing, since one would expect one coverage report per specification file.

--------------------------------------------------------------------------------
                     RecordFlux Validation Coverage Report
Directory: /home/rs/Dev/RecordFlux-specifications
--------------------------------------------------------------------------------
File                                          Links       Used        Coverage
ipv4.rflx                                        25         25         100.00%
ethernet.rflx                                    12          0           0.00%
arp.rflx                                         10          0           0.00%
--------------------------------------------------------------------------------
TOTAL                                            47         25          53.19%
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
                     RecordFlux Validation Coverage Report
Directory: /home/rs/Dev/RecordFlux-specifications
--------------------------------------------------------------------------------
File                                          Links       Used        Coverage
ipv4.rflx                                        25          0           0.00%
ethernet.rflx                                    12         12         100.00%
arp.rflx                                         10          0           0.00%
--------------------------------------------------------------------------------
TOTAL                                            47         12          25.53%
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
                     RecordFlux Validation Coverage Report
Directory: /home/rs/Dev/RecordFlux-specifications
--------------------------------------------------------------------------------
File                                          Links       Used        Coverage
ipv4.rflx                                        25          0           0.00%
ethernet.rflx                                    12          0           0.00%
arp.rflx                                         10         10         100.00%
--------------------------------------------------------------------------------
TOTAL                                            47         10          21.28%
--------------------------------------------------------------------------------

[treiher]

Sounds like a bug in the validator. Could you please open a ticket for that in the RecordFlux repository? Is that also related to https://github.com/Componolit/RecordFlux/issues/833? Can we get 100 %, coverage, if there are multiple independent messages in one specification file?

[rssen]

Can we get 100 %, coverage, if there are multiple independent messages in one specification file?

As far as I remember, the output is on a per-message basis. Only the message specification with the name passed as an argument to validate is checked. The unit-test test_validate_spec solves this by invoking the tool separately for each message in a package, but one would get a separate coverage report for each message. ~~The output is misleading here as we have the column "File", which implies the whole specification is checked.~~ I think ee made this decision, because you could have e.g. IPv4 Options messages, that cannot occur independently.

[treiher]

The output is misleading here as we have the column "File", which implies the whole specification is checked.

We should also look for a better solution here. This issue should be also recorded in a RecordFlux ticket. Could you please take care of that?

[rssen]

I need to partly correct myself on this. The coverage report contains the reached coverage over the entire file. So "File" is not misplaced. However, 100% coverage could never be reached in a single coverage report if there are two completely separate messages defined in a specification. This is because the validator works on a single message per run, as stated above.

The problem with the multiple coverage reports is caused by the unit test. It runs for every package of a pyrflx container and every message of a package, so in case of arp.rflx it runs for ARP::Packet_IPv4, Ethernet::Frame, IPv4::Packet and IPv4::Options independently, thus 3 outputs (no output for Options, as there are no example messages yet). So it seems like 3 outputs for one specification, but it is really 3 outputs for 3 individual messages.