Counter examples missing from failing tests

[frekw]

Hi!

When running tests that fail, ExCheck fails to print the failing counter example (which would be the expected behaviour, I assume).

For example, changing the sample to this:

  def prop_push_list do
    for_all({x, y}, {int(), list(int())}) do
      result = push(x, y)
      Enum.at(result, 0) == x and Enum.count(result) == Enum.count(y)
    end
  end

which is expected to fail, the output is the following:

ExCheck.SampleTest
  * test verify sample property (24.8ms)...........................................................................................
..........
  1) test verify sample property (ExCheck.SampleTest)
     test/sample_test.exs:5
     Expected truthy, got false
     code: ExCheck.check(ExCheck.Sample)
     stacktrace:
       test/sample_test.exs:6: (test)

  * test verify sample property with iteration parameter (20.5ms)

  2) test verify sample property with iteration parameter (ExCheck.SampleTest)
     test/sample_test.exs:9
     Expected truthy, got false
     code: ExCheck.check(ExCheck.Sample, 10)
     stacktrace:
       test/sample_test.exs:10: (test)

which doesn't provide me with much help for actually tracking down why a test is failing. Is this expected behaviour?

I'm running Elixir 1.4.0.

[avocade]

👍

[pdawczak]

I can confirm I have similar problem. The test fails as expected, but there is no hint regarding the problem:

[sotojuan]

Can anything be done @parroty? This can really make debugging tests hard.

[aerosol]

FWIW https://github.com/parroty/excheck/pull/37 solves it.

BTW I'll work on getting https://github.com/triqng/triq/pull/28 merged so that no IO capturing will be necessary in excheck.

[zkessin]

I merged triqng/triq#28

[aerosol]

Submitted first pass on https://github.com/parroty/excheck/pull/41

[parroty]

Sorry about late responses, and thanks for the PRs.

@aerosol: Could you help me understanding the relationship between #37 and #41? (Should I merge both? or just #41?).

[aerosol]

@parroty I pushed #37 to my branch and built stuff on top of it, effectively removing the module modified in #37. I could rewrite the commit history in #41, it has some rough at edges I've been meaning to take care of anyway. Sounds like a plan?

[aerosol]

cc @obrok

[parroty]

That sounds good (I understood that just merging #41 would be enough, and as long as original issue can be resolved, rewriting the history may not be required).

I can see one comment fixme. no counter example here in the PR. Is it ready? (or are you still working on this item?)

[obrok]

@parroty please just close #37 when #41 is merged

[aerosol]

Thanks @obrok

I can see one comment fixme. no counter example here in the PR. Is it ready? (or are you still working on this item?

@parroty Yes, I'd like to work on this PR a bit more. If you're OK with the general idea, I'll finish up a few things and ping you for a merge.

[parroty]

Great, thanks @aerosol @obrok.