Test result format

[iteadam]

How and where should the results from the tests be defined?

1. Where should it pretty print? Inside the test-container or in the js?
2. What's the output format?

Design goals:

1. Easy to understand the output without any other framework, as raw output from the test itself (i.e. docker run ..)
2. Easy for someone to read the test to browse through the different test results to understand what the test does.
3. Easy to convert to human-understandable language including localisations.

[iteadam]

Regarding 1 I feel like it might be best to let the person writing the test decide which tools to use for creating the text. If they're used to Python and use that for the test then it would make sense for them to use that to format the output

[irony]

I think the actual test output should be fairly technical, something like:

{status: "OK", code: 200, message: "DNS is checked and verified"}

or:

{status: "FAIL", code: 501, message: "DNSSEC is not correctly configured"}

the codes could be parsed into more human understandable format through a translation json:

codes_sv.json

{
  200: { title: "DNS kontrollerad och verifierad", mitigation: ""},
  501: { title: "DNSSEC ej korrekt konfigurerad", description: "DNSSEC är viktigt för att säkerställa att ingen tar över domänen och låtsas vara samma webbplats.", mitigation: "Se [example.org](https://example.org) för mer information om hur du konfigurerar DNSSEC"},
}

[irony]

Format for the codes: Maybe easier to just have a string constant instead. "OK": {... }, "DNSSEC_ERROR": {...} ?

[irony]

We should try to implement these and see how it feels. Probably the rest of the details will reveal themselves when we write the messages.

[iteadam]

Oh, I didn't think about localisation, then we can't just have the test return all the messages.

I would still like to keep the messages close to the tests, since they're coupled (if you change one you probably have to change the other) and because they explain the tests to someone reading the test code. Maybe just json-files next to the test named something predictable

[iteadam]

Mockup of what I imagine output looking like localised to swedish

• ❌ https.sv.title: Säker överföring via HTTPS https.sv.description: HTTPS krypterar besökarnas uppkoppling, detta är viktigt för att blah blah....
  • ✔️https.sv.existance.pass: Kommunikationen är krypterad, mycket bra!
  • ❌https.sv.tls_version.fail: Era servrar stödjer en gammal version av krypteringsalgoritmen TLS. Detta är dåligt då den har kända problem blah blah... [+] Lösning https.sv.tls_version.fix: För att ändra era servrar så de inte blah blah... [+] Detaljer https.sv.tls_version.raw: "Server returned Cypher: TLS1.2, TLS1.1, BLAH231, FOFO...
  • ✔️https.sv.cipher_support.pass: Ni stödjer bara säkra krypteringsmetoder! [+] Detaljer https.sv.cipher_support.raw: 'ECDHE-RSA-AES256-GCM-SHA384 \n DHE-RSA-AES128-GCM-SHA256'

Test output to drive this could then be

{ 
  id: 'https', 
  passed: false, 
  output: [ 
    { name: 'existance', passed: true },
    { name: 'tls_version', passed: false, raw: 'Server returned Cypher: TLS1.2, TLS1.1, BLAH231, FOFO...' },
    { name: 'cipher_support', passed: true, raw: 'ECDHE-RSA-AES256-GCM-SHA384 \n DHE-RSA-AES128-GCM-SHA256' }
  ]
}

[mull]

I think this kind of feedback (internet.nl) is pretty useful:

Being able to tell the use "Here is what we saw" is pretty useful so that they can match it in their admin panel (or whatever) of their registrar/webserver/etc.

If we want to have that level of presentation then we'll need some sort of markup language rather than plain strings.

[iteadam]

Couldn't you tell that my strings were markdown? ;D

[mull]

Oops 😅

A problem with markdown strings is that, as in the example above, we not only need basic formatting but also loops. A table of DNS-servers and their response/status is going to be tricky and would require some sort of string interpolation.

Last night I thought about using JSX for constructing the responses, pseudo example:

<Result score="pass">
  <Header>{t('https.sv.title)}</Header>
  <Description>
    <Table>
      {someItems.map(i => (<Table.Item>{i.value}</Table.Item>)}
    </Table>
  </Description>
</Result>

Not too complicated. WDYT?

[iteadam]

(It was just a joke about how regular text is valid markdown ;)

Maybe if we allow the tests to return a template with strings that we then replace with the text? Perhaps that what your meant with interpolation.

[mull]

(Damnit :D)

So we agreed on passing a finished markdown strings from the test. I'll have an example of that in #19.

What the docker container should return we don't need to specify, since this is passed from the worker.

I'll keep it simple and skip groupings for now.