Add a check that the output is valid YAML

[nonprofittechy]

It might be useful to have the Weaver check that the interview isn't broken at the end of the generation process. Ideally this would be put into comprehensive test cases that are in the GitHub action and not something that a user would ever need to see, but one way to achieve that goal would be to use @plocket's testing framework and verify that the "error" screen isn't reached. In that case, building this as an interactive screen in the interview would be a useful first step.

Valid YAML is a very low bar but we have encountered bugs that produced invalid YAML (e.g., the over-long description bug). It's a pain to download the interview, install it, and then find out it doesn't work when simple checks could obviate the need. We can certainly add additional checks--e.g., valid mako for all question blocks. Gold standard would be using Docassemble's method to pre-run the interview in the playground, but that may be too slow.

[nonprofittechy]

could also use ast.parse() on code blocks and mako.template.render() on question/subquestion fields.

[nonprofittechy]

This really just needs to be part of the kiln test, not a feature for normal use. Or even better, a unit test once we have a better API for the Weaver that doesn't require the Docassemble interactive frontend.

[nonprofittechy]

I found https://stackoverflow.com/questions/71334373/is-there-a-way-i-can-validate-yaml-files-on-github but when I ran yammlint on a real repo, it had a lot of "errors" that seem incorrect:

1. missing final line
2. No --- in a YAML that has only one document
3. Lines over 80 characters
4. spaces at the end of lines (which are confusing, but actually mean something in markdown)

I don't know if we want to enforce that picky YAML style on our interviews. Just make sure it's parsable.

Will keep this open for a bit, but I'm not sure how to use this in the Weaver just right now. Maybe just for unittesting with some predefined interviews so we know that we didn't break the weaver when we change the mako template somehow?

[BryceStevenWilley]

Agreed that yamllint isn't going to be what we want.

This was a project that @plocket and I were working on independently: I took a manual approach, with the intention of scaling up to something that could be used as a language server in VS Code, and plocket was using JSON schemas to build a schema that would work for docassemble YAMLs.

Both approaches have been pretty fast in my experience.

[nonprofittechy]

OK, I'll wait for your results then!

[plocket]

I'm not sure, but it's also possible ALKiln could handle this with a bit of fancy footwork in the action. It might need some added features from ALKiln, but I think not. For the action I'm thinking:

1. Write a test with a special tag (@generated?) for later testing the yml file
2. Run ALKiln (with a not @generated) to generate a .yml file and download it. That download may still exist after the step in a folder in the repo. We'd have to work out if we can find the path to it with a glob.
3. Run ALKiln again with the tag @generated to run just the tests for the generated file.

It might be faster than waiting for the various validation approaches, and maybe more reliable if da changes without our noticing or if we've made some kind of subtle mistake.