Support cabal test-suite with shelltestrunner

[fmind]

When I develop executables as part of a Haskell project, I have to do and repeat some manual tests to ensure that the executable behaviour is correct (e.g. status code, error message, output ...). This is tedious, and I think it could be automated nicely with shelltestrunner.

My proposal would be to support shelltestrunner in Cabal file as a test-suite.

Example of a Cabal test-suite with shelltestrunner support:

test-suite shelltest
hs-source-dirs: test
main-is: shelltest.hs
type: exitcode-stdio-1.0
default-language: Haskell2010
build-depends: base, shelltestrunner

test/shelltest.hs

module Main where

import Test.ShellTest (shelltest)

main :: IO ()
main = shelltest ["test/shelltest/"]

Shelltestrunner test files would be gathered in test/shelltest.

Shelltestrunner configuration could be set directly in shelltest.hs or in an external configuration file.

[simonmichael]

Sounds nice! Would you like to make a start on it ?

[fmind]

At the moment, the whole project is in one file.

I would need to split it into an executable and a library.

Do you think it makes sense ? Is it something acceptable ?

[simonmichael]

Sure, that’s fine.

[orlitzky]

FWIW, I do basically the same thing now (this is my ShellTests.hs):

module Main
where

import System.Process (
   CreateProcess( env ),
   createProcess,
   shell,
   waitForProcess )
import System.Exit ( exitWith )

main :: IO ()
main = do
  -- Get a CreateProcess object corresponding to our shell command.
  let createproc = shell "shelltest test/shell/*.test"

  -- But clear its environment before running the command.
  let empty_env_createproc = createproc { env = Just [] }

  -- Ignore stdin/stdout/stderr...
  (_,_,_,hproc) <- createProcess empty_env_createproc

  -- Now run the ProcessHandle and exit with its result.
  result <- waitForProcess hproc
  exitWith result

Granted that yours looks a lot nicer. Depending on libraries with cabal works a lot better than depending on executables, too. For example, hackage can't tell you that my project depends on "the shelltest executable being in $PATH".

[simonmichael]

Thanks for this idea back in 2017. Are you still using it ? Is there anything we need to do here ?

[simonmichael]

Perhaps add as a reproducible cookbook example to the https://github.com/simonmichael/shelltestrunner/blob/master/README.md ?

[orlitzky]

Yep, still using it, but struggling to remember what this issue is about =)

I think all the original report is asking for is a way to "do what the shelltest program would do," but within a library function. If there was a shelltest function provided by a library under Test.ShellTest, then you could use the sample code in the first comment rather than the real code in my previous comment.

Here's a sketch:

1. Refactor the bulk of shelltest's main function into Test.ShellTest.shelltest
2. Update shelltest's main to call that new function.
3. Ship Test.ShellTest as a public library.

Then anyone who wants to use shelltest as part of a cabal test suite can call the library function instead of the executable (which is a bit more trouble).

[simonmichael]

Ah, so make shelltestrunner usable as a haskel library ? That I can understand, thanks for clarifying. I agree this could be useful.

[andreasabel]

In general, an executable can be used in a testsuite by cabal test if it is declared as build dependency.
For an example, testing "myself" (agda2lagda) using goldplate, see: https://github.com/andreasabel/agda2lagda/blob/93d317c87bddf973e3fa6abd5bf24dcfa9a797f0/agda2lagda.cabal#L104-L115

So, add build-tool-depends: shelltestrunner:shelltestrunner to the testsuite stanza. Then you can system "shelltestrunner" ... in the Main of the testsuite. (Example: https://github.com/andreasabel/agda2lagda/blob/93d317c87bddf973e3fa6abd5bf24dcfa9a797f0/test/Tests.hs#L5)

[orlitzky]

Ah, build-tool-depends does indeed solve half of the problem. I am somehow just learning about it today. But keep in mind that this issue predates cabal-2.0 =)

Using system "shelltestrunner" isn't a great solution though. It's not really portable -- you have to feed it a command that will work on every possible shell, including Windows' cmd.exe -- and it's also a bit inefficient in that it creates a whole new shell process (sourcing .rc files and whatnot) to launch the command.

With shelltestrunner, I usually have multiple .test files in some subdirectory of tests/. What I'd like to do is to use a portable solution to find them all...

import System.FilePath.Find ((==?), always, extension, find)
find_sources :: IO [FilePath]
find_sources = find always (extension ==? ".test") "tests/shell"

and then test them within the same process, without involving an OS shell at all, e.g. (in pseudocode):

main :: IO ()
main = do
  sources <- find_sources
  shelltest $ sources

[orlitzky]

Ah, build-tool-depends does indeed solve half of the problem.

This was premature :skull_and_crossbones:

The "configure" phase happily proceeds when the required build-tool-depends are missing. In this case...

$ runghc Setup.hs configure --enable-tests -v3
...
Searching for shelltestrunner in path.
Cannot find shelltestrunner on the path

returns success (zero) on the command-line, rather than dying immediately as it would if a required library were absent.

[andreasabel]

$ runghc Setup.hs configure --enable-tests -v3

I have not tested this since I am always using the cabal command. Is there an issue on this problem in the cabal issue tracker?

[orlitzky]

I have not tested this since I am always using the cabal command. Is there an issue on this problem in the cabal issue tracker?

This one is similar in spirit: https://github.com/haskell/cabal/issues/5464