Difficulty running any test built on dune's `diff` action

[WardBrian]

trafficstars

I am trying to run my codebase's tests (Same project as #28), which include a lot of things like

(rule
 (alias runtest)
 (action
  (diff pretty.expected pretty.output)))

Where the test produces some pretty.output, and pretty.expected is checked in to the repo (so-called "gold" tests)

The difficulty is one run of these tests produces all the .output files in the _build folder. This means that if the first mutation causes a test failure, all the later ones will as well, because they're just comparing to the same test output.

This is definitely related to #16 #18, but it seems different enough I wanted to open my own issue for it.

I've got my currently-not-working branch here: https://github.com/stan-dev/stanc3/compare/master...WardBrian:stanc3:mutaml?expand=1

The way this currently proceeds if you run make mutation-test is that the first few runs timeout, while they're creating output files, and then the rest all fail because those output files exist and will not compare equal to the .expect files

[WardBrian]

I got closer by having the command passed to the metaml runner be a .sh file that contains

#!/bin/bash

find _build/default/test -name '*.output' -delete || true
dune runtest test/integration/ --force

Now it seems like my bigger issue is #32

[WardBrian]

Even after manually upping the timeout, still can't get this style of test to work. I'm confirming that the executable is receiving the environment variable, but it does not seem to really be mutating anything, as the tests all 'pass', but manually applying one of the mutation diffs that gets printed causes them to fail rather quickly

[WardBrian]

I was eventually able to get this to work by making a copy of the instrumented binary somewhere outside of _build, and then hard-coding that path into my tests. Not something viable for production, but to run this one-off it worked