Stop running tests on first test failure

[sol]

Let's assume the current setup

src/Util.hs
src/Foo.hs
src/Bar.hs
test/UtilSpec.hs
test/FooSpec.hs
test/BarSpec.hs

where both Foo and Bar import Util.

Now, if I hack on Util and have a failing test in UtilSpec, arion will still run FooSpec and BarSpec after UtilSpec. I would prefer if it would just stop after running UtilSpec until the tests for Util are green.

[mwotton]

I think this is a bigger issue than that - we got into it a little in https://github.com/karun012/arion/issues/9

[mwotton]

i think these basic rules apply:

1. if you change a test file, that should always trigger a run of that test file.
2. if a set of tests are run, and some subset pass, our scope should narrow: the ones that passed can be popped on a stack, and not run again until all the currently failing tests pass.
3. when all the tests in a run pass, we pop the stack and run the stored tests - if some of them fail, we follow rule 2.

[sol]

@soenkehahn and I hashed out a pretty straight forward algorithm that solely relies on GHCi's :reload and Hspec's --rerun. It is not trivial to implement, but should still be quite feasible. @soenkehahn, can you chime in?

[karun012]

@sol Waiting to hear more about the algorithm you guys hashed out

[soenkehahn]

Sorry for the delay.

As I understand the central idea of arion is that it understands the connection between some module Foo and its test module FooSpec and then can rerun only the tests for that module. While I like the idea, I think it wouldn't improve my workflow by much. (And there are certain cases where it would even hurt, by missing failing tests in other test modules.)

So @sol's and my idea is this: You have a tool that you point at let's say test/Spec.hs. At startup it opens a ghci session, tries to compile the module with all its dependencies, looks for a declaration of spec :: Spec in the module and if that exists it executes the test suite. If anything in the file system changes, it reloads and runs the tests again.

• If compilation of a module fails it sets the focus on that module. On any file change it tries to reload that module. On success it looks for a declaration of spec :: Spec in the focused module. (It might be a module from src in which case it probably doesn't have one, but it might be e.g. test/FooSpec.hs and then it should have that declaration.) If the declaration is found it runs those tests. If they succeed it pops the stack of focuses and goes from there (with reloading and testing).
• If any test fails it sets the focus on that specific test case in that test module. On any file change, it reloads that module and executes that single test case. If that succeeds it again pops the focus stack.

So it's basically @mwotton's 2nd and 3rd point combined with compilation. There are lots of details that we haven't figured out of course. But I hope I sketched the basic idea.

[sol]

Note that we get setting focus on failing tests for free by using --rerun

Sent from my iPhone

On 10 Apr 2015, at 11:19 am, Sönke Hahn [email protected] wrote:

Sorry for the delay.

As I understand the central idea of arion is that it understands the connection between some module Foo and its test module FooSpec and then can rerun only the tests for that module. While I like the idea, I think it wouldn't improve my workflow by much. (And there are certain cases where it would even hurt, by missing failing tests in other test modules.)

So @sol's and my idea is this: You have a tool that you point at let's say test/Spec.hs. At startup it opens a ghci session, tries to compile the module with all its dependencies, looks for a declaration of spec :: Spec in the module and if that exists it executes the test suite. If anything in the file system changes, it reloads and runs the tests again.

If compilation of a module fails it sets the focus on that module. On any file change it tries to reload that module. On success it looks for a declaration of spec :: Spec in the focused module. (It might be a module from src in which case it probably doesn't have one, but it might be e.g. test/FooSpec.hs and then it should have that declaration.) If the declaration is found it runs those tests. If they succeed it pops the stack of focuses and goes from there (with reloading and testing).

If any test fails it sets the focus on that specific test case in that test module. On any file change, it reloads that module and executes that single test case. If that succeeds it again pops the focus stack.

So it's basically @mwotton's 2nd and 3rd point combined with compilation. There are lots of details that we haven't figured out of course. But I hope I sketched the basic idea.

— Reply to this email directly or view it on GitHub.

[karun012]

@soenkehahn @sol From what I understand this is a completely different approach. I like the approach, but I think instead of changing the central idea of Arion it will be nice to implement this as a separate mode.

When @mwotton came up with the idea of focusing on one test at a time I was telling him that we should call it zen-mode #9. I say we take this route and create a new mode in which Arion works like you guys described. At some point if everyone feels that the normal mode where it associates source and test files is not being used anymore we can get rid of it. Until then I would like to keep both modes. What do you guys think?

[soenkehahn]

@karun012: Yes, it's a very different approach.

The idea to implement this as a different mode in arion sounds doable. But I wonder if it's not a case of scope creep. This could also be implemented in a completely different tool.

In any case it's nice to see some people put some thought into this. :)

[soenkehahn]

@karun012: Btw, did you come up with the term zen-mode or are there other tools that call it like that?

[karun012]

@soenkehahn zen-mode? I haven't used a tool that uses the term. I am fine with having it as a separate mode in arion. I like the name arion :).

[sol]

I gave this a head start at https://github.com/hspec/autospec/pull/1. It does not set focus yet, and it does not reload on file modifications.

@karun012 I like the name arion too. But for now I just want to iron out this idea. If all of us agree that this is the right approach, then I'm fine with merging it back to arion (or replacing arion with it). But for now I want to be free to experiment.

[karun012]

@sol Nice. I feel it will be a good feature for arion. I can add you as a contributor so that you don't have to make pull requests for everything. Let me know. On Apr 13, 2015 5:04 PM, "Simon Hengel" [email protected] wrote:

I gave this a head start at hspec/autospec#1 https://github.com/hspec/autospec/pull/1. It does not set focus yet, and it does not reload on file modifications.

@karun012 https://github.com/karun012 I like the name arion too. But for now I just want to iron out this idea. If all of us agree that this is the right approach, then I'm fine with merging it back to arion (or replacing arion with it). But for now I want to be free to experiment.

— Reply to this email directly or view it on GitHub https://github.com/karun012/arion/issues/29#issuecomment-92542285.

[karun012]

@sol done :) You can push directly to arion whenever you want.