Clarify the number of test cases run.

[Wilfred]

Given a file:

Simple commands:

  $ echo foo
  foo
  $ printf 'bar\nbaz\n' | cat
  bar
  baz

Multi-line command:

  $ foo() {
  >     echo bar
  > }
  $ foo
  bar

If I run it:

$ cram test_cases.t -v
test_cases.t: passed
# Ran 1 tests, 0 skipped, 0 failed.

This confused me, because there are two separate tests in this file. Cram's test count seems to only be the number of files executed:

$ cram test_cases.t exit.t
..
# Ran 2 tests, 0 skipped, 0 failed.

Could cram report the number of test cases too? Something like this:

$ cram test_cases.t
.
# Ran 1 test (2 specs), 0 skipped, 0 failed.

[mrak]

I would like this, too.

[roman-neuhauser]

i do not think this number should be reported because it does not carry the information you ascribe to it, yet it would be misinterpreted in that and other ways.

for start, @Wilfred's example contains four " $" lines: a function definition, a call to that function, and two echo / printf calls. the comment then calls for "Ran 1 test (2 specs)", but that seems wrong, it should be at least 3. but, should it be in fact 3 or 4?
should cram decide which shell constructs imply a "spec"? which shell are we discussing anyway?

let's have a look at one of the test files in roman-neuhauser/theresa:

setup::

  $ . $TESTDIR/setup

  $ export fake_zstat=1

  $ mkzstat -oLH st fubar -- uid 0 gid 0 mode 060640
  $ mkzstat -oLH st snafu -- -x 1
  $ mkzstat -oLH st lol -- mode 020644
  $ mkzstat -oLH st . -- mode 040755

  $ fake -x 1 getpwent -Nqu nobody


test::

  $ theresa blockdev fubar
  $ theresa blockdev snafu
  FAIL: blockdev snafu does not exist
  [1]

  $ theresa blockdev .
  FAIL: . is a directory
  [1]

  $ theresa blockdev lol --owned-by whoever
  FAIL: lol is a chardev
  [1]

  $ theresa blockdev fubar --owned-by nobody
  FAIL: blockdev fubar is owned by root, not nobody
  [1]

  $ theresa blockdev fubar --owned-by root --in-group wheel --mode 666
  FAIL: blockdev fubar has mode 0640, not 666
  [1]

what rules do you propose to decide which of those are actual "specs"?

finally, ignoring the setup / test distinction, you should realize that the number of " $" lines in a file ranges from signals-a-problem to meaningless. stateful or side-effectful line (environment variables, filesystem, etc) code means each " $" line may modify beharior of subsequent lines; OTOH any number is fine as long as the code is pure functions from arguments to output on stdout / stderr.

to conclude: almost every time i've wanted to see this indicator of my progress in adding "specs", i was actually digging myself a hole of brittle tests. the psychological pressure to factor the tests into more files to get the feedback my inner monkey craves is a benefit.