ghc-exactprint
ghc-exactprint copied to clipboard
alanz
Test Suite Failure with GHC 8.2.2
I'm getting a test suite failure building ghc-exactprint with GHC 8.2.2:
Running 1 test suites...
Test suite test: RUNNING...
GHC82
### Failure in: 1:Round-trip tests:0:ghc710:20:Control.hs
tests/Test/Common.hs:123
Control.hs
### Failure in: 1:Round-trip tests:0:ghc710:77:Internals.hs
tests/Test/Common.hs:123
Internals.hs
Cases: 1764 Tried: 1764 Errors: 0 Failures: 2
Counts {cases = 1764, tried = 1764, errors = 0, failures = 2}
Test suite test: FAIL
Test suite logged to: dist/test/ghc-exactprint-0.5.6.1-test.log
0 of 1 test suites (0 of 1 test cases) passed.
I've got this failure with the master branch and several versions from hackage:
- 0.5.6.1
- 0.5.6.0
- 0.5.5.0
I mostly got these results building with Nix, so it's possible that there is some misconfiguration on the Nixpkgs side. However, I also got the same failure building a clone of the repo directly with cabal.
It passes CI, and I just tried
$ cabal unpack ghc-exactprint
$ cd ghc-exactprint-0.5.6.1/
$ cabal install --dependencies-only --enable-tests --with-compiler=ghc-8.2.2
$ cabal configure --enable-tests --with-compiler=ghc-8.2.2
$ cabal test
and got
...
Linking dist/build/test/test ...
Running 1 test suites...
Test suite test: RUNNING...
Test suite test: PASS
Test suite logged to: dist/test/ghc-exactprint-0.5.6.1-test.log
1 of 1 test suites (1 of 1 test cases) passed.
Interesting. This means there's something about the Nix setup causing the tests to fail, but I don't know what that could be. I'll poke around a bit more to see if I can find the issue.
Do you have any hunches?
Is there some way to get more detailed output about what caused the test case to fail?
Looks like it might be a macOS issue of some sort. One of my colleagues followed the same build steps as I did on Linux and did not encounter the test failure.
To see the failure, should be simple, but we re-use some result files.
So edit https://github.com/alanz/ghc-exactprint/blob/ghc-8.6/tests/Test.hs#L135 to comment out the last test category, i.e.
return $ TestList [
internalTests,
roundTripTests
,
transformTests
, failingTests
, noAnnotationTests
-- ,
-- prettyRoundTripTests
]
Then when the tests fail, you can compare https://github.com/alanz/ghc-exactprint/blob/ghc-8.6/tests/examples/ghc710/Control.hs to a file in the same directory called Control.hs.out
The first part of the .out file should be a verbatim copy of the original, if there is a diff then that is where the error lies.
The rest of the file is the AST, and annotations found.
Just took a look at Control.hs.out and the behavior is really weird. The part of the file between module GHC.Event.Control and #include "EventConfig.h" has lots of duplicated text mixed in with the correct text.
Here's the broken part:
module GHC.Event.Control
module(GHC.Event.Control
(-- * Managing the IO manager
-- * Managing the IO managerSignal
, SignalControlMessage(..)
, ControlMessageControl(..)
, ControlnewControl
, newControlcloseControl
,-- ** Control message reception
,-- ** Control message receptionreadControlMessage
,-- *** File descriptors
,-- *** File descriptorscontrolReadFd
, controlReadFdcontrolWriteFd
, controlWriteFdwakeupReadFd
,-- ** Control message sending
,-- ** Control message sendingsendWakeup
, sendWakeupsendDie
,-- * Utilities
,-- * UtilitiessetNonBlockingFD
, setNonBlockingFDwhere
) where
#include "EventConfig.h"
Here's what it's supposed to look like:
module GHC.Event.Control
(
-- * Managing the IO manager
Signal
, ControlMessage(..)
, Control
, newControl
, closeControl
-- ** Control message reception
, readControlMessage
-- *** File descriptors
, controlReadFd
, controlWriteFd
, wakeupReadFd
-- ** Control message sending
, sendWakeup
, sendDie
-- * Utilities
, setNonBlockingFD
) where
#include "EventConfig.h"
I had some rough theories before looking at this, but none of them are consistent with this behavior.
That's odd. Normally anything with a #include fails the tests. So I am not sure why it passes on linux, tbh.
Does it pass if you delete that line from the file? The #include.
CPP must die for haskell.
Commenting out or deleting the #include resulted in the whole module getting messed up in the same way. The version with an #include was only messed up until the #include, after which it was totally fine.
Off-hand, this looks a bit like two streams (ie STDOUT and STDERR) getting interleaved in the .out file. When the #include is there, it's possible one stream failed and the other didn't, leaving the rest of the module untouched. Is there anything in the test runner that could cause this?
Can you make the .out file that you get (together with the original if you have changed it) available via a gist or some such?
BTW, there is a function tt' in Tests.hs that I normally edit to run specific tests, then do it in the repl.
That way you can run a single test at a time.
Okay, here's a Gist with both outputs: https://gist.github.com/TikhonJelvis/fb2b4ec9dbc4e73eee060883c33f9fda
The only change I made to Control.hs was deleting the #include "EventConfig.h" line for one of the runs.
Oh yeah, here's the output for the one other failing test case (Internals.hs). I didn't make any changes to the original.
https://gist.github.com/TikhonJelvis/4f7049926cf8e984692f41540917cde6
@alanz I'm seeing the same test failures as @TikhonJelvis is, building with Nix, using either GHC 8.2 and 8.4.
I just re-read this thread, and it struck me that there is something about the Mac CPP that might be causing an issue. But I forget what it is.
I guess we need to set up some mac CI, however that is done.
Makes sense, thanks.
Is there some way to get GHC to use a different CPP implementation? I believe there's one written in Haskell. At the very least this would let me rerun the tests locally to see if it makes a difference.