Unexpected behavior when `*.in` exists but `*.out` does not

[pwinckles]

I just started using this library for testing a cli that is used for manipulating files. I absolutely love how painless it's going to make testing, but I did notice a few surprising behaviors that I thought I'd report.

1. If a *.in directory exists and a *.out directory does not exist, then the tests will write output to *.in.
2. When dump is used under the same conditions as point 1, the output is not written to the dump directory and is still written to *.in.

I'm not sure if these behaviors are intentional or not, but behavior 1 is problematic because it destroys the test fixture files that are in the input directory, making the test non-repeatable. Behavior 2 also contaminates the test, but is additionally surprising because I ran the command with the intention of copying the output into *.out but it was never generated. The only way to get the content to generate is to have an existing, empty *.out directory, run dump, and then copy the dump output.

Finally, tangentially related, I understand that overwrite only deals with writing stdout and stderr, but it would nice if there was a similar command for syncing the contents of *.out without having to do a dump-copy.

Thanks for making this great tool!

[epage]

There is a .toml setting

fs.sandbox = <bool>

This controls whether we create a temp directory or whether things can run in the *.in directory. This setting defaults to true only when there is an *.out/. You can manually set it yourself.

The idea behind this choice is if you are just doing read-only filesystem operations, you don't pay for the cost of a temp directory while if you are modifying the file system, you are most likely going to be wanting to verify the results of it.

[epage]

Finally, tangentially related, I understand that overwrite only deals with writing stdout and stderr, but it would nice if there was a similar command for syncing the contents of *.out without having to do a dump-copy.

overwrite will overwrite existing files in *.out/. trycmd makes the assumptions that

1. You are unlikely to be verifying every changed file (e.g. in my static-site generator when testing RSS, let's not test all of the other files)
2. TRYCMD=overwrite writing out files that the user isn't testing will lead to them accidentally being tested locally just by doing subsequent runs or tested by everyone by the user running git add .. I've found users frequently will add untracked files to a repo, assuming that they are meant to be tracked.

Yes, it does mean that we are hurting the "initialize *.out/ case. I either

• Dump and copy it over
• touch files I want
• copy an existing test (I also have a rename.sh).

[pwinckles]

Thanks for the response!

Small correction to a typo in your original answer in case anyone reads this in the future, it's fs.sandbox rather than fs.snapshot.

Setting it does indeed work as described. Personally, I think I would want it to default to sandboxed and require an override to not run it in a sandbox, as I find that behavior less surprising and less destructive, but maybe the majority of test cases are read-only operations.

[Edit] But, I guess the default to not run in a sandbox is really only a problem when creating a new test case.