Add an option to exit on first failue

[jan-matejka]

Add an option to exit on first failue so cram --fail-fast ./tests exits on first failing test case and one can focus only the single test case failing output.

[roman-neuhauser]

This is implemented in Dram (https://git.sr.ht/~rne/dram), see https://git.sr.ht/~rne/dram/tree/master/t/fail-fast/opt.t:

setup::

  $ for x in a b d e; do
  >   printf -->$x '  $ true\n'
  > done

  $ printf -->c '  $ false\n'


test::

  $ dram -Dfv a b c d e
  . a
  . b
  ! c

  tests: 5, skipped: 2, failed: 1
  [1]

[aiiie]

@roman-neuhauser I’ve been kind of too busy with life to think about programming.

I’ve always been kinda intimidated by D, but I wish someone would take over the project and go in whatever direction they like. If you’re interested in extra responsibility for no reason, I’m happy to talk directly!

[aiiie]

And so you know, I might’ve mentioned this in my tests, but my original goal was to be able run the Mercurial test suite’s shell tests. I’m sure their format has significantly diverged in the past few years though so I’d never want to break backwards compatibility for Cram.

I’ve also had the goal for the tests to be runnable with identical output in Bash, Zsh, and Dash. So I recommend testing your tests in those shells.

[roman-neuhauser]

@roman-neuhauser I’ve been kind of too busy with life to think about programming.

I’ve always been kinda intimidated by D, but I wish someone would take over the project and go in whatever direction they like. If you’re interested in extra responsibility for no reason, I’m happy to talk directly!

Hey!

Sure, my email is in my profile.

[roman-neuhauser]

And so you know, I might’ve mentioned this in my tests, but my original goal was to be able run the Mercurial test suite’s shell tests. I’m sure their format has significantly diverged in the past few years though so I’d never want to break backwards compatibility for Cram.

I had a look at Mercurial's tests a few weeks ago and it seemed to have gone some ways indeed. I can't parse the second sentence, sorry. :) Anyway, to riff on the theme of BC: I've been using Cram for many years, and loved the simplicity, aesthetics, and stability of the test format all the way. I take stability of interfaces very seriously (http://rants.sigpipe.cz/durable.html), and should Dram become the next version of Cram, things like escaping metacharacters in (glob) lines will have to be figured out.

I’ve also had the goal for the tests to be runnable with identical output in Bash, Zsh, and Dash. So I recommend testing your tests in those shells.

Dang, NOCLOBBER strikes again! Thank's for the headsup, I have it fixed locally and will push shortly.

for s in sh bash dash mksh posh zsh; do DRAM_SHELL=/bin/$s make check; done

now passes.

[roman-neuhauser]

Oh, if you're saying that running Mercurial's current test suite is a non-goal for you, then rest assured that it's the same for me.