markdown
markdown copied to clipboard
texmacs
Automate tests
Currently tests/run.scm just converts files in subdirectories but doesn't check anything. We need to compare the output against the reference, either with diff or using scheme. diff might be better if we have CI, because installing and running texmacs headless for CI might range from complicated to an ordeal. Conversions could be done with a commit hook, and the pipeline would fail if tm files are newer than the md files. It's not rock solid automation, but better than nothing.
In TeXmacs there is some code to do testing. See e.g. for the HTML converter. Wouldn't be better to integrate the testing in Scheme, in particular to perform the comparison of the output at the level of trees? I never did CI, how does it work? Do we have a stable image or the image is created every time the process run? Run headless on Unix is not a problem using a virtual X server, we do already for the TeXmacs server in Bonn.
Yes, I saw the tests, but it seemed quite specific to that converter and disentangling it way too much work (at least at first sight). I wasn't aiming so much for unit tests as for "larger" regression tests, where I test a bunch of functionality and how it ties together. Just a compromise for the sake of simplicity since for this simple converter it often suffices (sufficed?) to see that some conversion failed in order to spot the problem. If anyone is willing to do the heavy lifting, we can have more fine-grained tests too, of course.
As to the CI: github runs its "actions" (pipelines) in runners hosted by them, basically VMs with some stuff preconfigured. There are Linux, Windows and Mac runners, so it might just be easier to use the latter, but e.g. on ubuntu, one can just run apt-get to update and install packages. Because that's quite wasteful to do on each push to the repo, and because there is a 2000 minute/month time limit for running actions, one could pre-build a docker image (with e.g. the latest TeXmacs stable release), push it to dockerhub and use it instead of the default environments provided by github.