Multiple tests within a single session

[jacg]

trafficstars

Consider a test suite (for example, https://github.com/digital-asset/ghcide/blob/5257eb7d9d1f8793e6dbb06ca63ad5a29e0e958a/test/exe/Main.hs#L593) which contains many tests of the form

oneTest :: Position -> Range -> TestTree
oneTest originatingPosition targetRange = sessionToTestTree $ do
  doc <- openDoc' "Foo.hs" "bar" sourceCode
  found <- getDefinitions doc originatingPosition
  found `containsDefAtExpected` targetRange

sessionToTestTree :: Session () -> TestTree
sessionToTestTree s = runSessionSomehow ... s ...

containsDefAtExpected :: [Location] -> Range -> Session ()
containsDefAtExpected [Location{_range = found}] expected = 
  liftIO $ expected @=? found

The key point is that sourceCode (and hence doc) is invariant across all these tests; what differs is the location in that document where getDefinitions is invoked, and the definitions that are expected to be found. Consequently, the test suite wastes lots of time opening a bunch of sessions containing identical documents, one for each test.

How can a single session be opened and shared across multiple tests, where each test will make a separate query about the document which can then be reported as independent passes/failures by the test suite?

[lukel97]

I presume you're asking if it's possible to use one session for multiple Hspec specs? As far as I can tell, we need some way of calling it/describe (stuff in SpecWith) within lsp-test's Session. Maybe it would be possible with a MonadBaseControl instance for Session/SpecWith? In the meantime it's always possible to just make multiple assertions within the one session, but I think this is something worth looking into

[jacg]

I presume you're asking if it's possible to use one session for multiple Hspec specs?

Well, in my specific case it's HUnit (via Tasty) rather than Hspec (so the tests are based around @=? rather than describe and it - I've edited the code outline above, to show these details) but yes, that's the general idea.

As far as I can tell, we need some way of calling it/describe (stuff in SpecWith) within lsp-test's Session.

liftIO allows me to call @=? within a Session, but running that session turns it into a single IO (), which corresponds to a single test. I want to run many independent tests in a single session.

Tasty provides a withResource for sharing resources across multiple tests, but I would need to share a half-evaluated Session for it to be any use.

Maybe it would be possible with a MonadBaseControl instance for Session/SpecWith?

I'm not sufficiently familiar with MonadBaseControl, I'll have to find some time to look into it.

In the meantime it's always possible to just make multiple assertions within the one session

Sure, but it results in monolithic tests, rather than nice granular ones. This leaves me with the choice between granular but slow vs fast but monolithic. I'm greedy, so I want both fast and granular. Hence ...

I think this is something worth looking into

... I agree.