Ability to selectively run tests.

[benkolera]

I've made a smallish tweak to doctest to give the user the ability to be able to only run the doctests that they want.

It's only by modules and line numbers, so it is not perfect, but it was the simplest change to some kind of selectivity in (that I could think of, anyway).

doctest --dt-select=Foo       src/*.hs # All tests in the Foo module
doctest --dt-select=Foo:22    src/*.hs # Doctest on line 22 of module Foo
doctest --dt-select=Foo:22-25 src/*.hs # Doctest between lines 22 and 25 inclusive.

I'm not attached to the implementation (esp. how I avoided pulling in some parsing deps for the arg parsing) so if you like the idea and want me to change it before merging it, just let me know.

[aavogt]

good idea. The same flag might be better it also supported the following:

--dt-select=Foo:bar # tests in the documentation for Foo.bar
--dt-select=Foo:bar-baz # tests attached to those functions (and those in between)
--dt-select=Foo:22- # tests line 22 to end?
--dt-select=Foo:-22 # tests up to line 22
--dt-select=dist/test/Foo-0.0.0.0-$testsuitename.log # tests that failed?
--dt-select # implicitly will look for that testsuite.log?

Also, it seems for now that you get no warning that the string you pass to --dt-select is invalid (you just get all the tests run).

[benkolera]

I agree that being able to put in function names would be awesome, but I'm not sure that it is easy to work into there. You know the code better than I do, so correct me if I'm wrong, but I couldn't see a great way to get at the function names other than just grepping the string in all of the expressions & properties or doing something crazy in Extract to give more information about the doctest other than just the line number. Would simple text searching (e.g. Data.List.isInfixOf) of the string be sufficient? I don't think that will be hard, but it seemed kind of ugly and controversial to be putting in my first pull request to doctest. ;)

Can definitely put the --dt-select=Foo:-22 and --dt-select=Foo:22- options in there. Good idea!

Those last two options are interesting. I'll look into what it would take to do those.

And yeah, the warnings are a bit crap since they have accumulated a lot of 'special case cruft' given that I started with just a module and line number and expanded it to the three options it is now without thinking too much. Thinking about it now, though, I have a way to tidy it and will get onto that.

[benkolera]

Take a look at this commit: https://github.com/benkolera/doctest-haskell/commit/de763acac59ed43604e2af074d7f7ab7053648c3

Which does running tests by function name (made it <Module>. rather than <Module>:<Func> to disambiguate between functions and line numbers).

It's pretty naive, but it works for the simple cases that I have tested. Let me know if I should merge this down to master to bring it into this PR.

[aavogt]

Maybe it's better to use : instead of . to separate modules from their contents in all cases? I don't see a realistic case in which a top-level declaration name can be confused with a line number. On the other hand it's probably uncommon to have a qualified data type name look the same as a module name too.

Consider --dt-select=A.B when you have two modules:

module A where
-- |
-- >>> 1+1
-- 3
data B = B

{-# LANGUAGE TypeOperators #-}
-- | >>> "A.B"
-- "A"
module A.B where
-- | >>> "another test"
-- ""
data a ::: b = a ::: b

-- | You can attach doctests to declarations
-- like the following, but nobody does this?
-- >>> 1+1
-- 2
1 = 2

I'll take another look tomorrow.

[benkolera]

The only reason why I chose . instead of using ":" is because it made it a lot easier in the parser to still give an error message if the user did something silly like --dt-select=Foo:bar-22. If we use ":" for everything then it'll just mean that anything that doesn't look like "-12", "12-", "12" or "12-14" will be treated as a function name.

But thinking about it now, my implementation is busted because the parser eager looking for the "." in the string. E.g:

--dt-select=A.B.baz will actually get parsed into TestSelector "A" (FunctionName "B.Baz")

As such, I committed this to the function-names branch: https://github.com/benkolera/doctest-haskell/commit/8a8e48e467c18a1aa32878aa039a0e9ec92eb7f7

[sol]

Hey @benkolera, sorry for getting back late! This feature definitely makes sense! Sadly, I currently don't have any spare cycles to review it. That will hopefully change in about 6 weeks from now, but if you want to polish things up, we will definitely find a way to get this on master (maybe @kazu-yamamoto can chime in).

I think we should simplify the user interface.

matching by filename and line

Note that we always have to run all examples within a given Haddock comment. So if we e.g. specify line 22 and we have

 19 -- |
 20 -- >>> let x = 23
 21 --
 22 -- >>> x
 23 -- 23

We should run the examples from line 20 and 22.

Similarly, if we specify a range x-y, we first determine which Haddock comments intersect with this range, and then run all the examples from all the Haddock comments.

# run all examples from Haddock comment that spans over line 22 of src/Foo.hs
# if there is no Haddock comment at line 22, select the next Haddock comment in
# the file (e.g. if there starts comment at line 25, select this)
doctest -isrc src/Foo.hs:22

# run all examples from Haddock comments that intersect with lines 23-28.  If
# there are no Haddock commets, behave like src/Foo.hs:23
doctest -isrc src/Foo.hs:23-28

matching by module

Currently if you specify e.g. Foo.hs, we run all examples from module Foo and all examples from all modules imported by Foo. Maybe we could consider to change this to "all examples from module Foo".

# run all examples from module Foo
doctest -isrc src/Foo.hs
# this also works
doctest -isrc Foo

(note that we can rely on GHC to resolve module names to filenames, as needed for doctest -isrc Foo)

matching by function name

I'm not sure if this is currently easy to do, and I'd probably defer this for now. But my preferred syntax would be

# run all examples for Foo.foo
doctest -isrc Foo.foo

corner cases: ambiguous name for module/type/constructor

I would tend to not distinguish between names for modules/types/constructors on a syntactical level. Rather, I would just run all matching examples.

# if there is a module Foo.Bar, run all examples from that module
# in addition, if there is a type Foo.Bar, run all examples for that type
# in addition, if there is a constructor Foo.Bar, run all examples for that constructor
doctest -isrc Foo.Bar

Again, I would probably defer that for now and focus on a sound implementation for filename/line number.

[decentral1se]

Bump! This would be cool!!!