shelltestrunner
shelltestrunner copied to clipboard
simonmichael
need to review/improve file formats
A new release has been stalled for a while, mainly because I'm not sure how good the proposed new file formats are. Here's the old format 1. I wanted to be able to reuse input for multiple tests in a file, so I added format 2, presumably to replace format 1 entirely after a transition period. I also wanted a lighter format that's easier to write and to copy/paste to/from a shell session, so format 2b is a step in that direction.
Reviewing these after some time, I find them confusing. Maybe they just need better docs & examples.
Here's another possibility, let's call it format 3 for a minute, that's still closer to a shell window:
# A CMD that reads standard input, followed by STDIN, then expected STDOUT/STDERR/EXITCODE as usual.
$ hledger -f- reg --pivot code
2017/1/1 (224) pro
a $-10
b $10
>>>
2017/01/01 pro code:224 $-10 $-10
code:224 $10 0
>>>2
>>>= 0
This has the advantage that the CMD and STDIN part (excluding the dollar sign) is a valid shell command, pasteable to and from a shell window. But it doesn't allow reuse of input for multiple tests. We shouldn't have too many formats (one would be best) but perhaps this is one we should support.
During my experience of using shelltest I didn't missed the feature to specify stdin for multiple tests. It depends on the type of program You are testing and in some cases it can save some time and lines. I agree with You that this feature can be confusing and in some cases it may even spoil some other tests by letting the program read stdin that suppose not to happen.
I like the idea that tests can be copied to cmd. and run without modifying the line.
When we are thinking about new format lets develop a notation for general test options / annotations that is general enough to add new features to the tests without braking the format. The first time I mentioned that was when I described how to specify the test description that is shown when the tests are running.
Here's a review of current and proposed formats, including a new markdown-compatible one for testing examples in documentation.
Test file format 1 / classic (implemented)
One or more tests, each specifying its input.
OPTIONAL WHITESPACE/#COMMENT
COMMAND LINE
OPTIONAL <<< INPUT
OPTIONAL >>> OUTPUT/REGEX
OPTIONAL >>>2 STDERR/REGEX
>>>= STATUS/REGEX
0 OR MORE ADDITIONAL TESTS
The command line and exit status check are required.
stdout/stderr are not checked implicitly.
Comment & whitespace lines before and after tests are ignored.
Test file format 2 / reusable input (implemented)
One or more test groups, which are input followed by one or more tests.
# COMMENTS OR BLANK LINES
<<<
INPUT
$$$ COMMAND LINE
>>>
EXPECTED OUTPUT (OR >>> /REGEX/)
>>>2
EXPECTED STDERR (OR >>>2 /REGEX/)
>>>= EXPECTED EXIT STATUS (OR >>>= /REGEX/)
# COMMENTS OR BLANK LINES
ADDITIONAL TESTS FOR THIS INPUT
ADDITIONAL TEST GROUPS WITH DIFFERENT INPUT
All parts are optional except the command line.
When unspecified, stdout/stderr/exit status are tested for emptiness.
The <<< delimiter is optional for the first input in a file.
Without it, input begins at the first non-blank/comment line.
Input ends at the $$$ delimiter. You can't put a comment before the first $$$.
The >>> delimiter is optional except when matching via regex.
Expected output/stderr extends to the next >>>2 or >>>= if present,
or to the last non-blank/comment line before the next <<< or $$$ or file end.
Two spaces between $$$ and the command protects it from -w/--with.
!/REGEX/ negates a regular expression match.
Test file format 2b / reusable input, short delimiters (implemented)
Same as format 2, but with short 1-2 character delimiters.
Test file format 3 / pasteable command & input
Intended to mimic what you could copy from/paste into a shell window. A dollar sign, a one-line command that reads standard input, followed by INPUT lines then the usual EXPECTED STDOUT/STDERR/EXITCODE specifiers. Eg:
$ hledger -f- reg --pivot code
2017/1/1 (224) pro
a $-10
b $10
>>>
2017/01/01 pro code:224 $-10 $-10
code:224 $10 0
>>>2
>>>= 0
Test file format 4 / markdown friendly
Here's one that aims to be not unlike command examples in markdown documentation, so you will be able to (in simple cases) test the examples in your docs. Basically, the lightweight format 3 above in literal blocks. Any markdown literal block delimited with backticks whose first line begins with $ is a test.
4.1 Simple form. Here no input is specified, stderr is expected to be empty, and exit code is expected to be 0.
```
$ COMMAND LINE
EXPECTED STDOUT
```
4.2 Testing stderr:
Here stdout is expected to be empty, and exit code is expected to be non-zero. To test for a specific exit code, add >>>= CODE.
```
$ COMMAND LINE
>>>2
EXPECTED STDERR
```
Some ways we could provide input:
4.3:
```
$ COMMAND
INPUT
>>>
STDOUT
```
4.4:
```
$ COMMAND
INPUT
<CTRL-D>
STDOUT
```
4.5 input first. This would permit multiple tests reusing the same input, though a change of command delimiter would be needed to allow input/output containing lines beginning with $.
```
<<<
INPUT
$ COMMAND
STDOUT
```
4.6 and of course there's (posix-specific):
```
$ printf "multi\nline\ninput\n" | COMMAND
STDOUT
```
Tests within a file are stateful, eg tests can see files created by a previous test. You can put dummy setup/teardown "tests" at start and end of the file.
Content between tests (outside of ``` blocks) is ignored. Except, a paragraph immediately before a test block (no intervening blank line) could be treated as metadata, with the first line being the test name and any remaining lines specifying tags, etc.
There are lots of different formats and as I remember You prefer when there are less. I suggest to end up with 2 formats - the classic for backward compatibility and the new version with as consistent format as possible (few or no exceptions how to specify stuff). Consistent formats maintainability is higher.
At first I liked format nr. 3 but how often is it really needed? I still like it but is it worth of modifying the format? When I am writing new tests I mostly begin from the command line and after that I copy the command to my test file. It does not work with using of macros! In my opinion the copy paste is mostly needed when tests are failing and You want to examine the case as fast as possible by copying the actual command that was executed. In that case all the macros (and executable specified with parameter --with) are resolved and debugging can start. Maybe the actual command executed when failing should be updated by embedding the stdin? For example "echo 'plah' | mytool -n".
The format nr. 4 is awesome but I suggest to write a preprocessor that transforms "some doc file" to shelltest file that is parsed. When writing a preprocessor the actual test format stays the same thus making shelltest more maintainable (the main functionality is not changed and the chance of making new bugs (to test parser) is not so high). With that approach its not hard to introduce more document types (e.g. man page, pdf, html ...) that may contain some examples.
I suggest to use consistent style when embedding metadata. As you told the metadata should be written before the test block but in some cases we don't want to show the metadata in documentation (its making document ugly and confusing) and in that cases we just can't test the examples. I suggest to specify metadata inside a comment block that is located before the actual test. I also suggest to specify stdout, stderr and exitcode within the metadata block as in many cases the real output of a command is not displayed in documentation.
<comment char> <metadata tag> = <metadata>
<comment char> <metadata tag> = <metadata>
<shelltest bock>
For example.
Regular shelltest file:
# @desc@ = My test
# @tag@ = something
$ test.exe -plah
Test from markdown:
[//]: # (@desc@ = Test example nr 1)
[//]: # (@tag@ = something)
```
$ test.exe -plah
```
Test from man page:
.\" @desc@ = Test example nr 1
.\" @tag@ = something
\fB$ test.exe -plah
Formats 2 and 2b above have been shipped (with 2b renamed to 3) as additional builtin formats in shelltestrunner 1.9.
I hope to convert hledger's tests to the new format 3 and experiment with the lighter syntax. No migration/conversion support has been shipped yet, so this will be a manual job.
Still on the wishlist is to support, one way or another, testing shell command examples in markdown documents. I manually extracted shelltestrunner's README examples as tests here.
Hello, I'm not sure if I should create another issue for that or just write it here, so I'll try the latter first:
I'm writing a program which prints file paths to stdout. By default, it utf-8 percent-encodes these paths, so that they stay within single lines and don't put the terminal into a weird state with control characters. However, I would also like the output to be easily consumable by scripts. The aforementioned format is lossy, because on unix-like systems filenames do not have a defined encoding (so they aren't necessarily valid utf-8) --- file paths on these systems are just arrays of non-zero bytes. AFAIK this is really the only constraint if we don't assume what particular filesystem is used. In this situation, it isn't really clear how to serialize such a thing into text... Luckily (or not) stdout isn't really text; again it's just a stream of bytes. So naturally, it's common for several common unix utilities to provide options to use the 0 byte (aka ASCII NUL character) instead of 10 (aka "newline", aka ASCII LF character) to separate records, in my case file paths (with a short prefix, but that's irrelevant for the issue I'm describing), see find(1) (option -print0), xargs(1) (option -0), grep(1) (option -Z). I decided to go with this approach, since it gives me an unambiguous mapping between this stream of bytes and a list of file paths; it should also be trivial for scripts to handle this output --- they don't have to reimplement utf-8 percent-decoding and the established tools already handle this kind of input gracefully.
However, right now it isn't particularly easy to input NUL-terminated lines into shelltestrunner's file formats. Two possible solutions to these problems that I see right now are:
- Add a format which accepts C-style character escapes (so I can e.g. use
\0to insert the NUL character) and ignores newlines (so that during test editing users don't have to deal with overly-long lines) - Add a format which replaces every newline character with a NUL when parsed
Thank you for your time.
Hello @yakubin, thanks for the discussion. I think this deserves its own issue, would you mind moving your comment to a new issue.
Update on the 2018 discussion above: many of hledger's tests have since been converted to format 3 (https://github.com/simonmichael/shelltestrunner#format-3, example: valuation.test). I did not get around to adding a conversion helper, so I've converted individual test files by hand, when needed. I've found format 3 quite useful and pleasant and I consider it the preferred format. Format 4 (markdown files with embedded examples) is not yet implemented and still much needed.
Format 4 (markdown files with embedded examples) is not yet implemented and still much needed.
I think like @taavi-valjaots that it would be better to have a slim consistent shelltest syntax without extras for orgmode (* comments only for folding, see #23 ) and markdown (fenced code blocks). It is easier to understand, maintain, and less error-prone. In my experience, the parser already gives strange error messages or unexpected results sometimes related to shared input and missing empty lines after tests... I would try to not make the syntax even more complex.
I use mostly v3.
A preprocessor would be a nice thing for markdown.
For orgmode, it should be very little effort, to write an ox-shelltest.el file for babel code block execution on export. The --print option can also help, to generate the actual results on C-c C-c in the orgmode code block.
the parser already gives strange error messages or unexpected results sometimes related to shared input and missing empty lines after tests
If you find any examples of this, please report (as new issues) and I'll investigate.
Update for 2023: I just finished converting hledger's remaining format 1 tests to format 3, fixing several issues in --print to get better results. I see the * org comments were also removed some time ago. And I clarified the guidance on formats a little in the readme. So I think this can be closed.