Run test suites in azure ci

[jneira]

• This is the more promising option for running tests in macos and windows (thanks god and @alanz at least we have the circle ci for linux)

• [X] Linux jobs: all test suites are green

• [ ] Windows jobs

  • [ ] unit-test suite:
    • Failing tests for CabalHelper using stack-8.6.4.yaml: openend https://github.com/DanielG/cabal-helper/issues/91
  • [ ] dispatcher-test: flaky, it fails sometimes for all ghcs always with same errors (see below)
  • [X] plugin-dispatcher: green
  • [ ] wrapper-test: failing
  • [ ] functional-test: one failing test for ghc-8.6.4 (Hover.hover works)

• [ ] macos jobs

  • [X] ghc-8.6.*: all test suites green
  • ghc-8.4.*:
    • [ ] test hangs after error in dispatcher test (see below)
    • [ ] functional test cant be run (func-test: fd:28: hPutBuf: resource vanished (Broken pipe))

[fendor]

Some of these issues should be fixed by #1126, so maybe we can try again to enable azure-ci testing, once that pr has landed.

[jneira]

I am just running the pr in azure with tests enabled: https://dev.azure.com/jneira/haskell-ide-engine/_build/results?buildId=212

[jneira]

Hi, after applying the workaround to avoid the errors compiling cabal-helper at runtime the number of failed tests has decreased sensibly:

• master: https://dev.azure.com/jneira/haskell-ide-engine/_build/results?buildId=219
  • Quite builds have only errors related with liquid haskell and even a linux build succeded: https://dev.azure.com/jneira/haskell-ide-engine/_build/results?buildId=219&jobId=86dfe75d-acf3-5a7d-391c-10fa0ecb27bb

The functional tests suite with the hie-bios branch hasmore fails: https://dev.azure.com/jneira/haskell-ide-engine/_build/results?buildId=221. Most of them are like:

haskell-ide-engine>   1) Completion.completions works
haskell-ide-engine>        uncaught exception: ConduitParserException
haskell-ide-engine>        NamedParserException "NotificationMessage ServerMethod PublishDiagnosticsParams" (NamedParserException "NotificationMessage ServerMethod PublishDiagnosticsParams" (NamedParserException "NotificationMessage ServerMethod PublishDiagnosticsParams" (NamedParserException "Logging notification" (BothFailed (Unexpected "ConduitParser.empty") (NamedParserException "NotificationMessage ServerMethod PublishDiagnosticsParams" (NamedParserException "NotificationMessage ServerMethod PublishDiagnosticsParams" (NamedParserException "NotificationMessage ServerMethod PublishDiagnosticsParams" (NamedParserException "Logging notification" (BothFailed (Unexpected "ConduitParser.empty") (NamedParserException "NotificationMessage ServerMethod PublishDiagnosticsParams" (NamedParserException "NotificationMessage ServerMethod PublishDiagnosticsParams" (NamedParserException "NotificationMessage ServerMethod PublishDiagnosticsParams" (NamedParserException "Logging notification" (BothFailed (Unexpected "ConduitParser.empty") (Unexpected "ConduitParser.empty")))))))))))))))
haskell-ide-engine> 
haskell-ide-engine>   To rerun use: --match "/Completion/completions/works/"

[jneira]

I've reran tests in azure with actual master: https://dev.azure.com/jneira/haskell-ide-engine/_build/results?buildId=334&view=results

• The unit-test suite passes
• The dispatcher and plugin-dispatcher suites pass
• The functional-test fails due to different behaviour of received diagnostics (logs: https://gist.github.com/jneira/ae1968cc67b28d3f222ed3001cbbc052)
• The wrapper-test fails due to different output of the stack command

[fendor]

This would be great! maybe we should start poking members of the haskell organizations to enable azure for this project?

[jneira]

Absolutely, do we know who has the permissions to do it, @alanz, @bubba?

[alanz]

I think the haskell committee has control of this. Not sure who specifically.

[jneira]

Ok, maybe we can wait to have the azure setup complete, including green test suites and deploy.

EDIT: i get linux and macos to run the entire test suite succesfully: https://dev.azure.com/jneira/0fa2df97-1d6d-4a00-935a-03a77e7170b8/_build/results?buildId=344&view=logs&j=d2c7b64e-54d7-53d5-7f80-77915c458472

[jneira]

Update:

---

Windows wrapper-test errors:

haskell-ide-engine>   test\wrapper\HieWrapper.hs:18:9: 
haskell-ide-engine>   1) version checking picks up a stack.yaml with 8.2.1
haskell-ide-engine>        expected: "8.2.1"
haskell-ide-engine>         but got: ""
haskell-ide-engine> 
haskell-ide-engine>   To rerun use: --match "/version checking/picks up a stack.yaml with 8.2.1/"
haskell-ide-engine> 
haskell-ide-engine>   test\wrapper\HieWrapper.hs:23:9: 
haskell-ide-engine>   2) version checking picks up a stack.yaml with 8.2.2
haskell-ide-engine>        expected: "8.2.2"
haskell-ide-engine>         but got: "8.6.5"
haskell-ide-engine> 
haskell-ide-engine>   To rerun use: --match "/version checking/picks up a stack.yaml with 8.2.2/"

---

windows dispatcher-test errors:

Failures:

  test\dispatcher\Main.hs:157:5: 
  1) functional dispatch defers responses until module is loaded
       uncaught exception: ErrorCall
       Prelude.head: empty list

  To rerun use: --match "/functional dispatch/defers responses until module is loaded/"

  test\dispatcher\Main.hs:205:7: 
  2) functional dispatch instantly responds to deferred requests if cache is available
       expected: ("references",Just [DocumentHighlight {_range = Range {_start = Position {_line = 7, _character = 0}, _end = Position {_line = 7, _character = 2}}, _kind = Just HkWrite},DocumentHighlight {_range = Range {_start = Position {_line = 7, _character = 0}, _end = Position {_line = 7, _character = 2}}, _kind = Just HkWrite},DocumentHighlight {_range = Range {_start = Position {_line = 5, _character = 6}, _end = Position {_line = 5, _character = 8}}, _kind = Just HkRead},DocumentHighlight {_range = Range {_start = Position {_line = 7, _character = 0}, _end = Position {_line = 7, _character = 2}}, _kind = Just HkWrite},DocumentHighlight {_range = Range {_start = Position {_line = 7, _character = 0}, _end = Position {_line = 7, _character = 2}}, _kind = Just HkWrite},DocumentHighlight {_range = Range {_start = Position {_line = 5, _character = 6}, _end = Position {_line = 5, _character = 8}}, _kind = Just HkRead}])
        but got: ("req2",Nothing)

  To rerun use: --match "/functional dispatch/instantly responds to deferred requests if cache is available/"

  test\dispatcher\Main.hs:257:7: 
  3) functional dispatch returns hints as diagnostics
       expected: ("r5",Just (PublishDiagnosticsParams {_uri = Uri {getUri = "file:///D:/a/1/s/test/testdata/FuncTest.hs"}, _diagnostics = List [Diagnostic {_range = Range {_start = Position {_line = 9, _character = 6}, _end = Position {_line = 10, _character = 18}}, _severity = Just DsInfo, _code = Just (StringValue "Redundant do"), _source = Just "hlint", _message = "Redundant do\nFound:\n  do putStrLn \"hello\"\nWhy not:\n  putStrLn \"hello\"\n", _relatedInformation = Nothing}]}))
        but got: ("references",Nothing)

  To rerun use: --match "/functional dispatch/returns hints as diagnostics/"

  test\dispatcher\Main.hs:297:7: 
  4) functional dispatch instantly responds to failed modules with no cache with the default
       expected: ("req7",Just [])
        but got: ("r5",Nothing)

  To rerun use: --match "/functional dispatch/instantly responds to failed modules with no cache with the default/"

---

windows func-test error for ghc-8.6.4:

haskell-ide-engine>   1) Hover.hover works
haskell-ide-engine>        predicate failed on: HoverContents (MarkupContent {_kind = MkMarkdown, _value = "\n```haskell\nsum :: [Int] -> Int\n```\n```haskell\nsum :: forall (t :: * -> *) a. (Foldable t, Num a) => t a -> a\n```\n\n\n\nbase Data.Foldable"})
haskell-ide-engine> 
haskell-ide-engine>   To rerun use: --match "/Hover/hover/works/"

---

windows func-test error with cabal anf ghc-8.4.4

Failures:

  src\Language\Haskell\LSP\Test\Decoding.hs:121:32: 
  1) Completion.completions works
       uncaught exception: ErrorCall
       Couldn't decode response for the request type: Shutdown
       "{\"result\":{\"capabilities\":{\"typeDefinitionProvider\":true,\"foldingRangeProvider\":false,\"textDocumentSync\":{\"openClose\":true,\"change\":2,\"willSave\":false,\"willSaveWaitUntil\":false,\"save\":{\"includeText\":false}},\"workspace\":{},\"implementationProvider\":true,\"documentRangeFormattingProvider\":true,\"documentHighlightProvider\":true,\"executeCommandProvider\":{\"commands\":[\"1904:applyrefact:applyOne\",\"1904:applyrefact:applyAll\",\"1904:generic:type\",\"1904:ghcmod:check\",\"1904:ghcmod:type\",\"1904:ghcmod:casesplit\",\"1904:hsimport:import\",\"1904:package:add\",\"1904:pragmas:addPragma\"]},\"renameProvider\":false,\"colorProvider\":false,\"definitionProvider\":true,\"hoverProvider\":true,\"codeActionProvider\":true,\"completionProvider\":{\"triggerCharacters\":[\".\"],\"resolveProvider\":true},\"documentSymbolProvider\":true,\"documentFormattingProvider\":true,\"referencesProvider\":true}},\"jsonrpc\":\"2.0\",\"id\":0}"
       CallStack (from HasCallStack):
         error, called at src\Language\Haskell\LSP\Test\Decoding.hs:121:32 in lsp-test-0.10.0.0-2370d2b00c7c9eae10f8c415728453dde8ccc33a:Language.Haskell.LSP.Test.Decoding

  To rerun use: --match "/Completion/completions/works/"

Commenting the previous test triggers another functional errors, so i've deactivated func-test for win+cabal+ghc-8.4.4

---

macos error in dispatcher-test for ghc-8.4

haskell-ide-engine> dispatcher-test: 
haskell-ide-engine> lookupSymbol failed in relocateSection (RELOC_GOT)
haskell-ide-engine> /Users/runner/runners/2.163.1/work/1/s/.stack/programs/x86_64-osx/ghc-8.4.3/lib/ghc-8.4.3/integer-gmp-1.0.2.0/HSinteger-gmp-1.0.2.0.o: unknown symbol `___gmp_rands'

Errors in unis-test involving cabal helper reported here: https://github.com/DanielG/cabal-helper/issues/91

[jneira]

• Intermitent errors:
  • stack windows jobs fail with C:\path\to\ghc\strip.exe: unable to rename 'D:\sr\snapshots\add677e6\bin\happy.exe'; reason: File exists. I've hit it in local, and you have to delete the snapshot
  • ~~stack windows job fail installing the specific ghc for liquidhaskell: error decompressing tar~~
    • resolved installing ghc's in the same step, at the beginning

[jneira]

After #1601 there are a new failing unit test in windows:

• All windows jobs using ghc-8.6 (stack and cabal)

Failures:

  test/unit\\ApplyRefactPluginSpec.hs:42:5: 
  1) ApplyRefactPlugin, apply-refact plugin, apply-refact plugin commands, applies one hint only
       uncaught exception: GhcException
       unit-test.exe: could not detect mingw toolchain

  To rerun use: --match "/ApplyRefactPlugin/apply-refact plugin/apply-refact plugin commands/applies one hint only/"

• All windows ghc-8.4 jobs

Failures:

  test/unit\\ApplyRefactPluginSpec.hs:134:5: 
  1) ApplyRefactPlugin, apply-refact plugin, apply-refact plugin commands, respects hlint config files in project root dir
       uncaught exception: IOException of type NoSuchThing
       .\settings: openFile: does not exist (No such file or directory)

  To rerun use: --match "/ApplyRefactPlugin/apply-refact plugin/apply-refact plugin commands/respects hlint config files in project root dir/"

• I cant reproduce locally 😟

• The first error is thrown by ghc here: https://gitlab.haskell.org/ghc/ghc/blob/4898df1cc25132dc9e2599d4fa4e1bbc9423cda5/compiler/main/SysTools/BaseDir.hs