create test cases checking expected outcomes

[riebl]

At the moment, we have a number of smoke tests that just abort if something goes utterly wrong. However, smaller problems or changes in behaviour may be undetected.

I would like to have the following aspects under test (non-exclusive):

• [ ] a station's position is properly initialised and updated (see #59)
• [ ] generated messages are actually transmitted by the radio
• [ ] CBR calculation produces reasonable results

[AshFungor]

Hey @riebl!

Can you update me on status for this issue? I wish to start helping out on an actual code for Artery, this enhancement seems like a good starting point :)

[riebl]

Unfortunately, there is no progress on these topics. The Docker containers you contributed are definitely a great foundation for building GitHub Action workflows that execute tests. Let me briefly share my rough idea of how such test cases could work:

• Test script firing up an Artery scenario (possibly building upon run_artery.sh)
• Record simulation data in OMNeT++ output vectors and scalar files (maybe SQLite format as it is easy to process)
• Asserting test cases' constraints by evaluation of the recorded data

[AshFungor]

Hey, @riebl!

I've been trying to get what we want to with the data. Setting scalar and vector recording options to true and changing default recording class to the one that writes databases is rather simple.

But what we want to do with the resulting database? Are we looking for some sort of regression testing, when we just mark current master (and data produced by scenarios) as canonical and compare incoming changes (data, produced by changed code, to be precise) against it? Or some sanity check on the scalars & vectors?

I'll work on just the recording for now.

[riebl]

At first, I would like to check for some expected behaviours. OMNeT++ has some built-in fingerprinting which can be useful for regression testing. However, it is tricky to find the right balance between setting it too strict (CI complains all the time) and too relaxed (does not catch errors).

Assuming we have the test vectors, some sanity checks could look like these:

1. Check that vector contains as many unique station ids as vehicles are moving around (for a scenario where all vehicles are equipped)
2. Check that vector contains at least one transmission per second for each vehicle (minimum CAM rate)
3. Check that recorded min/max x-y-coordinates of vehicle span a rectangle of sufficient size (thus assuring that vehicle nodes are really moving)
4. Check that recorded CBR values vary (i.e. not all 0.0)

I think it is okay if some checks are bound to a specific simulation configuration, e.g. requiring 100% penetration rate etc.

[AshFungor]

Hey, @riebl! This is not really connected with test cases themselves, but should we extend run_artery.sh script to support scenarios that have their services defined under scenarios/certain-scenario? Right now script adds only global artery shared libraries like libtraci.so and libartery_core.so and things like scenarios/highway-police/libartery_police.so are left out. Same for NED folders.

[AshFungor]

extend run_artery.sh script to support scenarios that have their services defined under scenarios/certain-scenario

One way to do that is to save all custom services in scenarios directory and then extract binary targets and NED folders from them, just the same way we do with artery target.

Another way is to generate a (possibly) unique script for each scenario, extracting only from artery and features that were defined for scenario.

[AshFungor]

Hey, @riebl! Had a small question regarding individual scenario tests. Some of our scenarios demonstrate certain features (sometimes directly, sometimes conditionally with specific configs) that require optional dependencies and more than one run of simulation for a single scenario. Should we allow these multiple runs per scenario?

Take highway-police for example

[General]
network = artery.inet.World

*.traci.core.version = -1
*.traci.launcher.typename = "PosixLauncher"
*.traci.launcher.sumocfg = "highway.sumocfg"

*.node[*].middleware.updateInterval = 0.1s
*.node[*].middleware.datetime = "2018-03-19 10:00:00"
*.node[*].middleware.services = xmldoc("services.xml")

[Config sumo-gui]
*.traci.launcher.sumo = "sumo-gui"

[Config storyboard]
*.withStoryboard = true
*.storyboard.python = "story"
*.node[*].middleware.services = xmldoc("services-storyboard.xml")

[Config envmod]
network = artery.envmod.World
*.traci.mapper.vehicleType = "artery.envmod.Car"
*.node[*].middleware.services = xmldoc("services-envmod.xml")
*.node[*].environmentModel.sensors = xmldoc("sensors.xml")
*.node[4].environmentModel.FrontRadar.fovRange = 200.0 m
*.node[4].environmentModel.FrontRadar.fovAngle = 20.0

[Config storyboard-gui]
extends = storyboard, sumo-gui

Ideally, I think we should test General, envmod and storyboard configurations. I propose to introduce similar test config structure: define common options top-level and override/add options under scenario config tags, like

scenario:
  common-option-1: "some-value"
  common-option-2: "some-value"

  storyboard:
    overriden-option-1: "new-value"

  envmod:
    extended-option-3: "some-value"

  tested_configurations: ["General", "envmod", "storyboard"]

This might slow down CI though.

[riebl]

I agree with you. In your example configuration, though, it may be difficult to distinguish the "options" and "configurations".

[AshFungor]

My thoughts after working on https://github.com/riebl/artery/pull/366:

• We should add separate type of tests, which are basically services' parameters validations. Upon loading such a test, runner should extract a matrix or plain 1D vectors with parameters' range and then run certain scenario X times validating that no combination of parameters' values break services or metrics.
• We should probably hook testing on every feature branch, for that we need:
  • add CI action for building Dockerfile with Artery
  • add CI action for test running

CI actions are also a nice place for formatters and linters, like clang-format and clang-tidy.

[AshFungor]

Also, our doc might move to this repo, with a CI action to update it.