lifted-async icon indicating copy to clipboard operation
lifted-async copied to clipboard
verified publisher icon maoe

The test suite is non-deterministic

Open mrkkrp opened this issue 7 years ago • 3 comments

The test suite lifted-async is not deterministic and contributes to flakiness of Stackage HEAD project. For example, with the same build plan and the some docker image, sometimes we get this:

> /tmp/stackage-build2090/lifted-async-0.10.0.1$ ghc -clear-package-db -global-package-db -package-db=/home/circleci/project/builds/nightly/pkgdb Setup
> /tmp/stackage-build2090/lifted-async-0.10.0.1$ ./Setup configure --enable-tests --package-db=clear --package-db=global --package-db=/home/circleci/project/builds/nightly/pkgdb --libdir=/home/circleci/project/builds/nightly/lib --bindir=/home/circleci/project/builds/nightly/bin --datadir=/home/circleci/project/builds/nightly/share --libexecdir=/home/circleci/project/builds/nightly/libexec --sysconfdir=/home/circleci/project/builds/nightly/etc --docdir=/home/circleci/project/builds/nightly/doc/lifted-async-0.10.0.1 --htmldir=/home/circleci/project/builds/nightly/doc/lifted-async-0.10.0.1 --haddockdir=/home/circleci/project/builds/nightly/doc/lifted-async-0.10.0.1
Configuring lifted-async-0.10.0.1...
> /tmp/stackage-build2090/lifted-async-0.10.0.1$ ghc -clear-package-db -global-package-db -package-db=/home/circleci/project/builds/nightly/pkgdb Setup
> /tmp/stackage-build2090/lifted-async-0.10.0.1$ ./Setup build
Preprocessing library for lifted-async-0.10.0.1..
Building library for lifted-async-0.10.0.1..
[1 of 2] Compiling Control.Concurrent.Async.Lifted ( src/Control/Concurrent/Async/Lifted.hs, dist/build/Control/Concurrent/Async/Lifted.o )
[2 of 2] Compiling Control.Concurrent.Async.Lifted.Safe ( src/Control/Concurrent/Async/Lifted/Safe.hs, dist/build/Control/Concurrent/Async/Lifted/Safe.o )
Preprocessing test suite 'test-lifted-async' for lifted-async-0.10.0.1..
Building test suite 'test-lifted-async' for lifted-async-0.10.0.1..
[1 of 5] Compiling Test.Async.Common ( tests/Test/Async/Common.hs, dist/build/test-lifted-async/test-lifted-async-tmp/Test/Async/Common.o )
[2 of 5] Compiling Test.Async.IO    ( tests/Test/Async/IO.hs, dist/build/test-lifted-async/test-lifted-async-tmp/Test/Async/IO.o )
[3 of 5] Compiling Test.Async.Reader ( tests/Test/Async/Reader.hs, dist/build/test-lifted-async/test-lifted-async-tmp/Test/Async/Reader.o )
[4 of 5] Compiling Test.Async.State ( tests/Test/Async/State.hs, dist/build/test-lifted-async/test-lifted-async-tmp/Test/Async/State.o )
[5 of 5] Compiling Main             ( tests/TestSuite.hs, dist/build/test-lifted-async/test-lifted-async-tmp/Main.o )
Linking dist/build/test-lifted-async/test-lifted-async ...
Preprocessing test suite 'regression-tests' for lifted-async-0.10.0.1..
Building test suite 'regression-tests' for lifted-async-0.10.0.1..
[1 of 1] Compiling Main             ( tests/RegressionTests.hs, dist/build/regression-tests/regression-tests-tmp/Main.o )
Linking dist/build/regression-tests/regression-tests ...
> /tmp/stackage-build2090/lifted-async-0.10.0.1$ dist/build/test-lifted-async/test-lifted-async
lifted-async test suite
  Test.Async.IO
    async waitCatch:       OK
    async wait:            OK
    async exwaitCatch:     OK
    async exwait:          OK
    withAsync waitCatch:   OK
    withAsync wait2:       OK
    async cancel:          OK
    async poll:            OK
    async poll2:           OK
  Test.Async.State
    async waitCatch:       OK
    async wait:            OK
    async exwaitCatch:     OK
    async exwait:          OK
    withAsync waitCatch:   OK
    withAsync wait2:       OK
    async cancel:          OK
    async poll:            OK
    async poll2:           OK
    withAsync waitEither:  OK
    withAsync waitEither : OK
    withAsync waitBoth1:   OK
    withAsync waitBoth2:   OK
    link:                  OK
  Test.Async.Reader
    async waitCatch:       OK
    async wait:            OK
    async exwaitCatch:     OK
    async exwait:          OK
    withAsync waitCatch:   OK
    withAsync wait2:       OK
    async cancel:          OK
    async poll:            OK
    async poll2:           OK
    link:                  OK

All 33 tests passed (0.01s)
> /tmp/stackage-build2090/lifted-async-0.10.0.1$ dist/build/regression-tests/regression-tests
Main
  issue1: OK

All 1 tests passed (0.00s)

Other time we're less lucky:

> /tmp/stackage-build2090/lifted-async-0.10.0.1$ ghc -clear-package-db -global-package-db -package-db=/home/circleci/project/builds/nightly/pkgdb Setup
> /tmp/stackage-build2090/lifted-async-0.10.0.1$ ./Setup configure --enable-tests --package-db=clear --package-db=global --package-db=/home/circleci/project/builds/nightly/pkgdb --libdir=/home/circleci/project/builds/nightly/lib --bindir=/home/circleci/project/builds/nightly/bin --datadir=/home/circleci/project/builds/nightly/share --libexecdir=/home/circleci/project/builds/nightly/libexec --sysconfdir=/home/circleci/project/builds/nightly/etc --docdir=/home/circleci/project/builds/nightly/doc/lifted-async-0.10.0.1 --htmldir=/home/circleci/project/builds/nightly/doc/lifted-async-0.10.0.1 --haddockdir=/home/circleci/project/builds/nightly/doc/lifted-async-0.10.0.1
Configuring lifted-async-0.10.0.1...
> /tmp/stackage-build2090/lifted-async-0.10.0.1$ ghc -clear-package-db -global-package-db -package-db=/home/circleci/project/builds/nightly/pkgdb Setup
> /tmp/stackage-build2090/lifted-async-0.10.0.1$ ./Setup build
Preprocessing library for lifted-async-0.10.0.1..
Building library for lifted-async-0.10.0.1..
[1 of 2] Compiling Control.Concurrent.Async.Lifted ( src/Control/Concurrent/Async/Lifted.hs, dist/build/Control/Concurrent/Async/Lifted.o )
[2 of 2] Compiling Control.Concurrent.Async.Lifted.Safe ( src/Control/Concurrent/Async/Lifted/Safe.hs, dist/build/Control/Concurrent/Async/Lifted/Safe.o )
Preprocessing test suite 'test-lifted-async' for lifted-async-0.10.0.1..
Building test suite 'test-lifted-async' for lifted-async-0.10.0.1..
[1 of 5] Compiling Test.Async.Common ( tests/Test/Async/Common.hs, dist/build/test-lifted-async/test-lifted-async-tmp/Test/Async/Common.o )
[2 of 5] Compiling Test.Async.IO    ( tests/Test/Async/IO.hs, dist/build/test-lifted-async/test-lifted-async-tmp/Test/Async/IO.o )
[3 of 5] Compiling Test.Async.Reader ( tests/Test/Async/Reader.hs, dist/build/test-lifted-async/test-lifted-async-tmp/Test/Async/Reader.o )
[4 of 5] Compiling Test.Async.State ( tests/Test/Async/State.hs, dist/build/test-lifted-async/test-lifted-async-tmp/Test/Async/State.o )
[5 of 5] Compiling Main             ( tests/TestSuite.hs, dist/build/test-lifted-async/test-lifted-async-tmp/Main.o )
Linking dist/build/test-lifted-async/test-lifted-async ...
Preprocessing test suite 'regression-tests' for lifted-async-0.10.0.1..
Building test suite 'regression-tests' for lifted-async-0.10.0.1..
[1 of 1] Compiling Main             ( tests/RegressionTests.hs, dist/build/regression-tests/regression-tests-tmp/Main.o )
Linking dist/build/regression-tests/regression-tests ...
> /tmp/stackage-build2090/lifted-async-0.10.0.1$ dist/build/test-lifted-async/test-lifted-async
lifted-async test suite
  Test.Async.IO
    async waitCatch:       OK
    async wait:            OK
    async exwaitCatch:     OK
    async exwait:          OK
    withAsync waitCatch:   OK
    withAsync wait2:       OK
    async cancel:          OK
    async poll:            OK
    async poll2:           OK
  Test.Async.State
    async waitCatch:       OK
    async wait:            OK
    async exwaitCatch:     OK
    async exwait:          OK
    withAsync waitCatch:   OK
    withAsync wait2:       OK
    async cancel:          OK
    async poll:            OK
    async poll2:           OK
    withAsync waitEither:  OK
    withAsync waitEither : OK
    withAsync waitBoth1:   OK
    withAsync waitBoth2:   OK
    link:                  OK
  Test.Async.Reader
    async waitCatch:       OK
    async wait:            OK
    async exwaitCatch:     OK
    async exwait:          OK
    withAsync waitCatch:   OK
    withAsync wait2:       OK
    async cancel:          OK
    async poll:            OK
    async poll2:           OK
    link:                  FAIL
      tests/Test/Async/Reader.hs:125:
      expected ExceptionInLinkedThread _ TestException but got TestException

1 out of 33 tests failed (0.02s)

Look like this line is executed: https://github.com/maoe/lifted-async/blob/develop/tests/Test/Async/Reader.hs#L125.

mrkkrp avatar May 10 '18 15:05 mrkkrp

Thanks for reporting the issue. I was aware of it as the daily builds on Travis have been suffering from it for a while but haven’t gotten around to fixing it.

The issue looks very strange. The link in lifted-async is defined as liftBase . Async.link. Async.link is supposed to throw an exception wrapped with ExceptionInLinkedThread but we see an unwrapped TestException for some reason.

I think I can take a further look maybe in the weekend but in the meanwhile would you like me to disable this particular test until it’s fixed?

maoe avatar May 10 '18 22:05 maoe

Yes, that would be great.

mrkkrp avatar May 11 '18 05:05 mrkkrp

Uploaded https://hackage.haskell.org/package/lifted-async-0.10.0.2 that ignores the failure.

maoe avatar May 13 '18 05:05 maoe