doctest

doctest copied to clipboard

I have been using cabal-doctest to build a doctest testsuite (which is then run via cabal test). However, cabal-doctest requires custom setup in the .cabal file with a Cabal dependency, which is bad for the dependency footprint.

So I am trying the "new" recipe (not soo new anymore):

cabal repl -w doctest

This fails some doctests (that should pass) because I get mysterious type-defaults warnings (see below). I can workaround this via

cabal repl -w doctest --repl-options=-Wno-type-defaults

(UPDATE: the OP used --ghc-options which does not work reliably, see discussion to follow.)

Couldn't this problem fixed generally by not counting GHCI warnings as part of the test result?

Example:

src/BNFC/Backend/Agda.hs:677: failure in expression `numberUniquely ["a", "b", "a", "a", "c", "b"]'
expected: [(Just 1,"a"),(Just 1,"b"),(Just 2,"a"),(Just 3,"a"),(Nothing,"c"),(Just 2,"b")]
 but got: 
          ^
          <interactive>:109:1: warning: [-Wtype-defaults]
              • Defaulting the following constraints to type ‘String’
                  (Show a0) arising from a use of ‘print’ at <interactive>:109:1-45
                  (Ord a0) arising from a use of ‘it’ at <interactive>:109:1-45
                  (IsString a0) arising from a use of ‘it’ at <interactive>:109:1-45
              • In a stmt of an interactive GHCi command: print it
          [(Just 1,"a"),(Just 1,"b"),(Just 2,"a"),(Just 3,"a"),(Nothing,"c"),(Just 2,"b")]

We can see here that this implementation of doctests does not deal properly with {-# LANGUAGE OverloadedStrings #-}.

I can workaround this via

cabal repl -w doctest --ghc-options=-Wno-type-defaults

Actually, I cannot confirm that --ghc-options=-Wno-type-defaults has any effect. I was under the impression that this was the workaround, but actually I get both false positives and false negatives. So, this flag is likely irrelevant.

It must be other effects that fix this sporadic problem. Like running for a second time, building again, whatever. I cannot see the pattern.

A reliable workaround is to add this to the .cabal file:

ghc-options: -Wno-type-defaults

@andreasabel I think we want to make sure that --ghc-options=-Wno-type-defaults works as expected, or at the very least understand what exactly is happening here.

Can you check if --ghc-options=-Wno-type-defaults takes effect for a regular cabal repl ghci session?

If no, we want to figure out how to convince cabal to pass GHC options to cabal repl.

If yes, we want to understand what's going on Doctest's side, likely in ghci-wrapper.

To try to simulate this in GHCI/cabal repl I'd need to know what doctest makes out of a doctest expression e. The error above mentions a print it...

I tried to construct a little reproducer:

• doctest-issue.cabal:

cabal-version:              2.0
name:                       doctest-issue
version:                    0.0.0

build-type:                 Simple

library
  hs-source-dirs:           src
  exposed-modules:          OverloadedStrings

  default-language:         Haskell2010
  default-extensions:       OverloadedStrings
  ghc-options:              -Wall

  build-depends:            base

• src/OverloadedStrings.hs

module OverloadedStrings (foo) where

-- | Fooking do noothing.
--
-- >>> foo ["a", "b", "c"]
-- ["a","b","c"]
--
foo :: Ord a => [a] -> [a]
foo xs = xs

However, in this case the type-defaults warning can be successfully suppressed:

$ cabal build && cabal repl -w doctest --ghc-options=-Wno-type-defaults
... (ok)

I shrank too much, it seems.

I have played 2 hours now with this issue and I found that cabal build/repl are stateful in an awful way so that I can reach both states where:

1. I see no error despite not giving --ghc-options=-Wno-type-defaults.
2. I see the error despite giving --ghc-options=-Wno-type-defaults.

All this depends on the history of (at least) the following actions. a. cabal clean (silent transition) b. cabal build (silent transition) c. cabal repl -w doctest (accept or reject depending on whether tests pass) d. cabal repl -w doctest --ghc-options=-Wno-type-defaults (accept or reject)

Research question:

1. Document an action sequence that ends in state 2.
2. Figure out the language (action sequences that lead to accept), and if exists finite state automaton generating this language.
3. If you are not successful in question 2, try complete the picture by new actions. (Maybe by interrupting actions c/d.)

This sounds frightening. If you can come up with a repro then I guess we need a Cabal issue for this (a repro without -w doctest would be ideal, but even if the repro requires -w doctest it's still something that shouldn't happen).

Here is an example run, organized in three lines:

1. Action (b,c,d).
2. Doctest result:- (didn't run), o (success), x(failure).
3. Whether result is pathological:- (expected), 1 (bad 1., see above), 2(bad 2.).

bcddbdcdbdcdccbccd
-xxx-ooo-ooooo-xxx
--22--1---1-11---2

This random sequence supports the following thesis:

If the first cabal repl -w doctest after cabal build comes with --ghc-options=-Wno-type-defaults, then the doctest will succeed and continue to succeed, no matter if this option is included in subsequent invokations of cabal repl -w doctest or not. Symmetrically, if the option is omitted on the first doctest invokation, the test will fail and continue to fail, whether this option is passed later or not.

Wer lesen kann, ist im Vorteil. RTFD:

cabal repl TARGET loads all of the modules of the target into GHCi as interpreted bytecode. In addition to cabal build’s flags, it additionally takes the --repl-options and --repl-no-load flags.

To avoid ghci specific flags from triggering unneeded global rebuilds these flags are now stripped from the internal configuration. As a result --ghc-options will no longer (reliably) work to pass flags to ghci (or other repls). Instead, you should use the new --repl-options flag to specify these options to the invoked repl.

Ok, this explains the erratic behavior. (What I do not understand, though, is why cabal repl happily takes --ghc-options on the command line, and then does WHAT EXACTLY with them?)

@sol wrote:

@andreasabel I think we want to make sure that --ghc-options=-Wno-type-defaults works as expected, or at the very least understand what exactly is happening here.

So, answer is, it does not work, but --repl-options=-Wno-type-defaults seems to work as expected.

(This it the first time I see --repl-options, but frankly, I never use cabal repl.)

Reported cabal repl --ghc-options upstream:

• https://github.com/haskell/cabal/issues/8638