Generators: semantics of `#testgroup/testdata.yaml` key

[thorehusfeldt]

In a Nutshell

Q: The contents of a testcase’s testdata.yaml file be overwritten by the entries in #testcase/"testdata.yaml". Should this happen

1. for the entire file or
2. for the individual keys of testdata.yaml (such as grading or input_validator_flags).

This makes a difference when something else also has set some keys of testdata.yaml.

Example

A hypothetical top-level po-scoring key in generators.yaml takes a list of integers for implementing the scoring tradition of the Swedish Informatics Olympiad PO like this:

po-scoring:
  - 12 # score for testgroup 1, tiny inputs (brute-force should pass)
  - 24 # score for testgroup 2 (straightforward quadratic-time DP)
  - 64 # score for testgroup 3 (needs FFT)

From this, a (hypothetical) tool generates the necessary testdata.yaml entries in the rest of the testdata (this involves setting grading, grader_flags (to min, ignore_sample, sum), accept_score, and range in various ways at data, data/sample, data/secret, and data/secret/group1, …2 and …3.) All of this is simple but fragile, and the tool just takes care of it.

In addition, the generators.yaml file might include an entry like

data:
  secret:
    group1: # brute-force should pass
      testdata.yaml:
        input_validator_flags: --max_n 20

Expected behaviour

I would expect the following contents of data/secret/group1/testdata.yaml:

accept_score: "12"
range: 0 12
grader_flags: min
input_validator_flags: --max_n 20

Possible semantics

For each key k specified by a #testdata/problem.yaml entry, its value defines the value of the testdata.yaml file in the given testcase. If no such file yet exists, create it. It is an error(? warning?) if a file already exists that already specifies k.

Current documentation

Here’s what we say right now:

testdata.yaml: Optional yaml configuration that will be copied to testdata.yaml in this directory.