mdn
Prose end-to-end testing with recipes
We've had at least two errors now where the prose sections are busted. https://github.com/mdn/stumptown-experiment/issues/32 and https://github.com/mdn/stumptown-experiment/issues/37
There's not really anything obviously wrong with the Markdown in those files. Maybe there might be, but that's the point.
What I would like to have is something that reads the recipes, understands which sections are mandatory and then validates the built JSON files for each of these.
Let's keep it simple and make an end-to-end test exclusively for this. It'd need to first run the npm run build-json ... and then iterate over the produced .json files and do checks including loading in the recipe.
@wbamberg I'm happy to take a stab at this one. But before I do I would appreciate your input on my issue description.
This will definitely be great to have.
-
how stable do the data structures need to be before we do this? I think realistically the structures will continue to change for a while yet as we learn more, and it will be a pain to have to keep updating the validation. At some point we just have to accept that pain, but we should try for a very basic level of stability before we do this. Perhaps after https://github.com/mdn/stumptown-experiment/pull/41 is resolved?
-
we need docs for the structures that can be used to write the validation. I have a pretty good idea of what the structures should be, but need to write it down (and in the course of writing it down I'll probably realise I don't have as good an idea as I thought :).
-
this becomes essential at the point we are adding substantial amounts of content. We're not at this point yet, but maybe we will be soon.
I'm a huge proponent of never writing unit tests that test the implementation. In this context, it means to NOT write validation code/scripts that nitpick around inside the content. What really matters is; do the scripts (e.g. build-json.js) produce content that matches stuff the recipes require.
Let's not make something that is over-engineered. I don't know if we need a spec but it would be good to get the ball rolling on something that does some validation on the prose. When we discover it's insufficient we can beef it up with more complex checks.
Last, there are, at the moment, more exciting features to work on that works towards a great MVP for structured content.
By the way, I think this is the best description of an MVP ever: https://twitter.com/jopas/status/515301088660959233?s=20
I had thought this issue was about testing the content, since both the issues you linked are content errors, and over in https://github.com/mdn/stumptown-experiment/pull/44 you commented that this issue is the one 'to actually add "content linting" '.
If this issue is to add content linting, then yes it should nitpick inside the content. If it's some integration testing that the particular set of content we have right now produces the particular set of pages we want, then no it shouldn't. And that's fine and also a useful thing to have. But we still need content linting.
This is about content linting. But, depending on the angle or how you phrase it, the validation can be considered end-to-end testing because the constituents might pass but when used in a bigger context is fails.
I.e. instead of poking inside the .md files, I'd like some system that looks at what's produced from the .md files.
We have a content structure for code examples. Each example can include a title. If the title is present it's used as a heading for that example (like https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video#Simple_video_example).
Should the title be mandatory? In the current content, it isn't: some example omit titles, especially but not exclusively when there is only one example. But the current rendering code assumes a title will always be present.
We ought to resolve this: deciding whether every example must have a title is a content design question. I think probably titles should be optional, but really that's beside the point. The point is that the system should help authors write correct content: meaning that if titles are mandatory, then the system should flag it when they omit a title. Also that it should be clear to the rendering code what assumptions it can make about the content (for example, whether it can rely on finding a title).
Whether it does this by "poking around inside the .md files" or in some other way, I guess I don't care much.
Lots of details to iron out. If we rely on the final rendering to spot problems with the content, it might complicate things and the error messages are possibly going to be cryptic. If we write advanced and specific validators we might be wasting our precious time too since everything is still quite loose. We'll just have to keep poking at it.
I'm still not entirely sure what is involved when you say "end-to-end testing".
To illustrate my view on this, let me describe how I've done various testing in BCD and how I would imagine a similar story for stumptown, but as it is early on, maybe we come to different conclusions and architectures. For BCD at least, I look at three main test areas:
-
"Content linting" BCD contains structured information, so upon querying it, it must return certain things in a structure that is consistent and well designed. BCD authors have to comply to the structural rules otherwise the machines/APIs/anything can't really get information out of it. Writing lints and inventing content rules help the authors to be explicit about what content we want to provide. "Should MDN/BCD provide polyfill code or a url to a polyfill?" becomes a question and we're making an explicit decision and code it into our content design / content linter. Consumers of MDN content (MDN the website, MDN in VSCode, MDN devtools, ...) know what to expect from us and what not. Practically, BCD designed a data schema for that. Authors only provide compat data in that structure and have to play by these rules. There is a linter and the build will fail whenever you add information/content that does not comply. In stumptown, we don't have such a fine grained data schema yet, but we have recipes. This is because stumptown deals with docs generally and not with (compat) data specifically and it is because we're still early with this project. In my opinion, stumptown should fairly soon code up content linting (at least for our HTML test docs), so that we have structured content and fulfill promises as a reliable content provider much like BCD is a (somewhat) reliable structured data provider.
-
"Formatting linting" In BCD, we use JSON as the authoring format and we are fairly strict about how to format it. The main reason for this is that mass-changes over a consistent authoring format is much easier than doing it on inconsistent files. For stumptown, I think I would want consistent markdown (e.g. one way to declare headings ), so that when I decide to change the structure to comply with a new content rule, I can do so cleanly across the whole content.
-
"Integration tests" BCD is used by "MDN the website", so will be stumptown hopefully. So, "MDN the website" should test the integration and rendering of the BCD information. This currently happens in KumaScript. So, a KumaScript macro has the rendering code, and there is test code that checks if the renderer always renders the correct tables given very different compat data that comes from BCD. For stumptown, I would imagine there is a project that wants to render "MDN the website" out of the content from stumptown, BCD, and potentially other sources. Maybe the goal of "mdn2" is to be the new "MDN the website" code!? I believe mdn2 should test if it integrates and renders correctly, but it should also rely on certain promises the content provides (bcd, stumptown, etc) make and if the content providers don't deliver, well then that is a problem and the content providers should fix that and get their promises right (in their "content linting"). I don't think this is mdn2's job, as next month there will be a new content consumer, say "MDN in devtools", and they want rely on the same content provider promises.
Also, it seems like there is an additional layer that BCD doesn't have: build-json takes the md authoring format and creates, well, json data out of it. I guess that is then the stumptown-json layer, which makes new promises and comes with its own data schema, which ought to be tested if it fulfills its promises.
Here again, I don't think it should be testing what "content linting" tests, because if I want to create a stumptown-graphql layer next week, I want to rely on a consistent content design in the markdown files, too, right? Or, do you envision that other layers and consumers will always have to work with the stumptown-json layer? Is the markdown then only an authoring convenience and all promises we make to stumptown consumers should be made in the stumptown-json layer?
I believe mdn2 should test if it integrates and renders correctly but it should also rely on certain promises the content provides (bcd, stumptown, etc) make
Generally, what we want is that someone wants to make an edit contribution on stumptown and type "Shirt description" or they edit the "example.atttributes", they make PR and then the PR should run checks. Those checks should fail and raise awareness. (i.e. "Short description" not "Shirt description" and "example.attributes" not "example.atttributes"). Those errors should prevent a merge to master and then mdn2 (or "MDN in devtools") can continue to assume that every document has a "short_description" and every example has a "example.attributes" without having to be too defensive around it.
Instead of making distinctions between "linting" and "formatting" and "tests" I prefer the word "integrity" (or "integrity check"). Consumers will have to be aware of recipes and if the recipes say that the doc is supposed to have a prose.short_description the stumptown should make sure that's always true. How to do that is a deeper discussion(*).
(*) I.e. write scripts that inspect the .md files and the .yaml files, or scripts that run build-json, make it a lot pickier, and expect an 0 exit code.
Maybe the goal of "mdn2" is to be the new "MDN the website" code!?
Yes, more or less.
I believe mdn2 should test if it integrates and renders correctly, but it should also rely on certain promises the content provides (bcd, stumptown, etc) make
Yes, I agree.
Or, do you envision that other layers and consumers will always have to work with the stumptown-json layer? Is the markdown then only an authoring convenience and all promises we make to stumptown consumers should be made in the stumptown-json layer?
Exactly. I think the MD is an authoring convenience and stumptown-json is the thing we make promises about. So for example we ought to be free to change the way we organize attribute MD files as long as the JSON output is the same.
(similarly, for example, in BCD you can decide how the JSON is split up among files, and that's not visible to your consumers).
Generally, what we want is that someone wants to make an edit contribution on stumptown and type "Shirt description" or they edit the "example.atttributes", they make PR and then the PR should run checks. Those checks should fail and raise awareness. (i.e. "Short description" not "Shirt description" and "example.attributes" not "example.atttributes"). Those errors should prevent a merge to master and then mdn2 (or "MDN in devtools") can continue to assume that every document has a "short_description" and every example has a "example.attributes" without having to be too defensive around it.
Yes, I think we are all in agreement on this.