Support alternative file loaders

[indygreg]

(Filing this issue in response to discussion on #214.)

I frequently find myself brushing up against limitations in the file formats that trycmd currently supports. (I am one of the top historical contributors to Mercurial, whose expressive .t test format / harness was the inspiration for cram. cram exposes a subset of what Mercurial's .t format supports (for better or for worse). And trycmd a subset of cram's. So from my perspective the feature set of trycmd is somewhat limited.)

It seems I'm not alone. Scanning the issue tracker for this project reveals a handful of other users wanting extensions to the file format.

I'd like to propose a new feature (which I initially proposed in #214): support for custom file loaders. The way it works today is trycmd is the lone arbiter deciding how tests are defined and loaded from files. My idea is to provide an API that allows customers to bring their own file formats / loaders. This requires opening up the TryCmd struct to the public API (#217).

If customers are able to define their own file formats, they are able to experiment with extensions to the current file format or completely different file formats entirely. All while reusing the internals of trycmd to run tests. This could lead to a world where people proposing new file format features are encouraged to experiment on their own projects first before coming back to this project with a clear demonstration of value to justify inclusion in the official file format.

As a tangible example of this, using trycmd with my patches from #214, I built a custom test runner that supports running .trycmd tests from a filesystem sandbox of the crate root - something not possible with .trycmd tests using the public API available today. This only scratches the potential of what's possible.

[epage]

Similar to #217, this is solution focused. Let's list out the use cases you are trying to solve with this solution so we can evaluate what the right solutions are for this.

[indygreg]

My over-arching use case is:

• Implement file format features (.trycmd format extensions if you will) that don't yet exist [without being blocked on trycmd to accept a new feature and publish a release.

Random ideas on the top of my mind - some of these probably have existing open issues:

• Introduce % as an extensible directive syntax to influence state.
  • e.g. % cwd foo can change the current working directory.
  • e.g. % require windows as a way to specify constraints on where the test can run.
  • e.g. % if macos... % endif for inline conditional execution.
  • e.g. % alias foo=bar to define command or substring aliases.
  • e.g. % shell bash to instruct the runner to execute all commands in the same shell (this is how Mercurial's test harness works natively).
• Support lines beginning with < following $ lines as a way to define multiline stdin to a command.
• Support basic | syntax in $ lines to define command chains. Useful for executing command A to pipe to command B.
• Support a special output matching syntax that can span multiple lines.
• Support output matching syntax to mark a line as conditional.
  • Or dependent on conditions, such as Windows only.

(I'm not formally proposing these - they are just ideas.)

My point is there are infinite possibilities to define / extend the test definition format. I'm likely not going to convince trycmd maintainers to accept them without a lot of effort: maintainers are justifiably skeptical about complexity! But if the trycmd API allows customers to extend the file format with ease, it encourages this experimentation outside the crate and provides a more natural pathway for inclusion upstream for good ideas that have demonstrated value in other projects.

[epage]

Most of that wouldn't work with just manual construction (#217) or alternative file loaders (this issue) but tie into all of the behavior of trycmd.

For some % directives (and |), those could be handled if we had a more proper shell to run in. I actually would like to explore using an embeddable shell. I wish there was a more modern TCL. Nushell might be a possibility though its a bit heavy in dependencies and likely wasn't designed for this use case.

For <, that could be handled by a custom parser.

Matching is built directly in, so any changes there would require major re-works.

My point is there are infinite possibilities to define / extend the test definition format

I hope the above showed why concrete cases are needed rather than infinite possibilities. Flexible code is hard to write and maintain and usually is extended in the wrong direction. We need a concrete idea of what extensions are needed.