Document how to write test suite generators

[kytrinyx]

This is not documented anywhere that I can find.

[Insti]

Has anyone worked out how to do it yet?

We're working on it over in https://github.com/exercism/xruby/issues/485 but we're not done yet, and I'm in no way confident enough at this point to be able to say in any non-ruby specific way: "This is how you write a generator"

[ErikSchierboom]

I've created some generators for the C# track, but nobody else has had a look at them unfortunately. They work well, but I'm not sure about the structure of the code.

[petertseng]

I'm assuming we would like these to be language-independent? In that case the most that can probably be said is:

• Parse the JSON
  • You can refer to the schema to know what the generic structure is
• Look for cases
• Output them in the target language
• It's nice to include the x-common version
• property tells you what's being tested
• Read the comments to see if there's anything interesting

[kytrinyx]

I'm assuming we would like these to be language-independent?

Yes, this is at a conceptual level. If someone wanted to write their own generator from scratch in a track that didn't have them, what would they have to know? What might they be confused about?

[wobh]

I've been thinking about this for the lisp track and it seems like the JSON structure of the test data is a point of common concern with #11, #26, and #28.

[kytrinyx]

@wobh I was going to ask you to elaborate, but I think I understand what you mean. All of those documents will probably want to refer to the canonical data documentation, right?

[ireddick]

I wrote a problem-specific one over on the JavaScript track here (was some time ago now).

I remember looking at the ruby and scala tracks to see how things were being done before starting.

Having a look over things as they are now, I can see that there has been an improvement made to the canonical JSON data structure, so this particular generator's data parser will now fail.

What I can remember puzzling about while I was writing it:

1. Where to put the generator files (both for running them, and for them to have access to the canonical data in x-common).
2. What to name the generator files (at the time prefixing with "example" was recommended if the file was going into the same dir as the example and the spec).
3. What is the .version file and do we need to generate one.
4. What dependencies are acceptable in the generator code.
5. Do we want to group the tests (the JS track uses spec-style tests, so theoretically they could be nested in contexts representing the top level cases).
6. What to do with non-canonical tests (we appended the existing ones to the end of the suite in this one).
7. Do we need to have tests (maybe relevant in light of the breaking change in JSON structure I mentioned earlier 😃 ).
8. How will the generator be run (the comment block at the start of the generator I linked to was added to help guide future maintainers).

I hope this helps a little.

[wobh]

@kytrinyx sorry for the late reply, but, yes. Right. When I get time for xlisp, again, I'll make some notes about this.

[kytrinyx]

@ireddick these are great notes, thanks. @wobh appreciate it!