enso icon indicating copy to clipboard operation
enso copied to clipboard

Published 20 hours ago • verified publisher icon enso-org

Transient issues in backend tests

Open Akirathan opened this issue 1 year ago • 11 comments

As of today, September 2023, we experience frequent Engine and Stdlib transient job failures. This issue serves as a list of all these issues so that we can keep track of them.

  • [x] RefactoringTests - rename project
    • https://github.com/enso-org/enso/actions/runs/6098519307/job/16548518988?pr=7751#step:10:7210
    • https://github.com/enso-org/enso/actions/runs/6081506750/job/16542318567?pr=7599#step:10:7207
  • [ ] InliningBuiltinsTest (NoSuchMethodError)
    • https://github.com/enso-org/enso/actions/runs/6147863209/job/16680272086?pr=7794#step:10:4583
    • https://github.com/enso-org/enso/actions/runs/6148637750/job/16682766510?pr=7756#step:10:4678
  • [ ] sbt: ServerAlreadyBootingException (java.io.IOException: Could not create lock for \\.\pipe\sbt-load-802998070017202719_lock, error 1336)
    • https://github.com/enso-org/enso/actions/runs/6098519307/job/16548520260?pr=7751#step:10:707
    • https://github.com/enso-org/enso/actions/runs/6238498583/job/16934446773?pr=7847#step:10:676
    • https://github.com/enso-org/enso/actions/runs/7888374779/job/21525749192?pr=9038#step:8:909
  • [x] Enso tests failures in Excel
    • https://github.com/enso-org/enso/actions/runs/6107071447/job/16573353177?pr=7718#step:10:10340
    • https://github.com/enso-org/enso/actions/runs/6149491753/job/16685489613?pr=7793#step:10:9902 (File not found exception)
  • [x] Native Image build failed with "Image generator watchdog detected no activity"
    • Reported as #7850
  • [ ] Error in Rust build script "HTTP status server error (503 Service Unavailable)"
    • https://github.com/enso-org/enso/actions/runs/6082740008/job/16501130768?pr=7726#step:10:560 LoggerInitializationException
    • https://github.com/enso-org/enso/actions/runs/6111204906/job/16585910923?pr=7751#step:10:7553
  • [ ] org.enso.interpreter.test.instrument.RuntimeErrorsTest: lllegalStateException: "The language did not complete all polyglot threads but should have"
    • https://github.com/enso-org/enso/actions/runs/6149491753/job/16720398230?pr=7793#step:10:6666
    • https://github.com/enso-org/enso/actions/runs/7714268820/job/21026123578?pr=8869
  • [x] org.enso.interpreter.test.instrument.RuntimeServerTest: should recompute expressions changing an execution environment
    • https://github.com/enso-org/enso/actions/runs/6380739479/job/17315711946?pr=7840#step:10:3778
  • [ ] Failed to upload file project-manager-windows.tar.gz
    • https://github.com/enso-org/enso/actions/runs/6186164135/job/16793186901?pr=7807
  • [ ] [org.apache.commons.io.IOIndexedException: IOException #0: Cannot delete file: C:\Users\ContainerAdministrator\AppData\Local\Temp\enso-test9984012585562224382\installed\log\enso-launcher-2023-09-19.log] (Cannot delete a temporary test directory)
    • https://github.com/enso-org/enso/actions/runs/6237105771/job/16929910573?pr=7825#step:10:6222
    • https://github.com/enso-org/enso/actions/runs/7903080664/job/21570185814?pr=8953#step:8:3326
  • [ ] [collect2: error: ld returned 1 exit status]
    • https://github.com/enso-org/enso/actions/runs/6340868084/job/17223234362?pr=7918
  • [ ] language server test "ReceivesTreeUpdateHandler" timeout:
    • https://github.com/enso-org/enso/actions/runs/6351538769/job/17252919996?pr=7883#step:10:4126
    • https://github.com/enso-org/enso/actions/runs/6380394667/job/17314693145#step:10:4479
    • https://github.com/enso-org/enso/actions/runs/6382365087/job/17320800859?pr=7929#step:10:4190
  • [x] "[live]" did not equal "[design]" (RuntimeServerTest.scala:4359)
    • https://github.com/enso-org/enso/actions/runs/6410804439/job/17420283503?pr=7953
  • [ ] Error: listen EADDRINUSE: address already in use :::47308
    • https://github.com/enso-org/enso/actions/runs/6547498818/job/17780102906?pr=8080
  • [x] ~~java.util.ServiceConfigurationError: org.enso.base.file_format.FileFormatSPI: Provider org.enso.database.EnsoConnectionSPI not found~~ (mixing artifacts from different builds https://github.com/enso-org/enso/pull/8006) #8129 ~~https://github.com/enso-org/enso/actions/runs/6549252500/job/17785756996?pr=8093~~
  • [ ] assertion failed: timeout (5 seconds) during expectMsgClass waiting for class java.lang.String in ReceivesTreeUpdatesHandlerTest
    • https://github.com/enso-org/enso/actions/runs/6642894609/job/18048711561?pr=8145
  • [ ] /Standard/Base/0.0.0-dev/src/Data.enso: error: The module Standard.Base.Data.Enso_Cloud does not exist.
    • https://github.com/enso-org/enso/actions/runs/6655553215/job/18086068902?pr=8146#step:10:3448
  • [ ] Another file not found in Excel_Spec.enso
    • https://github.com/enso-org/enso/actions/runs/6799384402/job/18485581916?pr=8254#step:10:5912
  • [ ] RuntimeServerTest - The language did not complete all polyglot threads
    • https://github.com/enso-org/enso/actions/runs/6960734466/job/18940838644?pr=8285#step:10:3379
  • [ ] Failed to decode a line from output: invalid utf-8 sequence of 1 bytes from index 5
    • Raw buffer: [67, 58, 92, 114, 32, 133, 13]. Decoded with placeholders: C:\r �
    • somewhere around startup benchmarks: at <enso> find_prefix<arg-2>(src\Startup\Startup.enso:55:13-27)
    • https://github.com/enso-org/enso/actions/runs/7262995437/job/19787353802?pr=8511#step:10:17357
    • Cannot find enso-engine-* in ... - https://github.com/enso-org/enso/actions/runs/7291182230/job/19869597501?pr=8620#step:10:16124
  • [ ] ProjectShutdownSpec engine not installed
    • file changed as we read it https://github.com/enso-org/enso/actions/runs/7260847544/job/19780840613?pr=8577#step:10:2139 reported as #8597
  • [ ] NullPointerException: Cannot invoke "com.oracle.graal.python.builtins.objects.frame.PFrame$Reference.getPyFrame()" because the return value of "com.oracle.graal.python.builtins.objects.function.PArguments.getCurrentFrameInfo(com.oracle.truffle.api.frame.Frame)" is null reported as #8632
  • [ ] Cannot read the package definition from [/tmp/11110293095345258873]
    • https://github.com/enso-org/enso/actions/runs/7759245233/job/21233882751?pr=8869

Akirathan avatar Sep 08 '23 15:09 Akirathan

Issues with the project rename tests (RefactoringTests suite) should be fixed by #7794

4e6 avatar Sep 11 '23 14:09 4e6

Added to the list: InliningBuiltinsTest (NoSuchMethodError) - https://github.com/enso-org/enso/actions/runs/6147863209/job/16680272086?pr=7794#step:10:4583

GitHub
Hybrid visual and textual functional programming. Contribute to enso-org/enso development by creating an account on GitHub.

4e6 avatar Sep 11 '23 16:09 4e6

"Enso test failures in Excel" transient failures - After discussion with @JaroslavTulach, we think that the issue might be cause by enso_project.data returning a different directory than expected. Probably caused by a GC issue in https://github.com/enso-org/enso/blob/develop/engine/runtime/src/main/java/org/enso/interpreter/node/expression/builtin/meta/EnsoProjectNode.java#L96C12-L96C12. Diagnostics added in https://github.com/enso-org/enso/pull/7793/commits/7f9c05d4d64575919cc3054447c5c77a7c798579

Akirathan avatar Sep 13 '23 10:09 Akirathan

Added to the list: java.io.IOException: Could not create lock for \\.\pipe\sbt-load-802998070017202719_lock, error 1336

  • https://github.com/enso-org/enso/actions/runs/6238498583/job/16934446773?pr=7847
GitHub
Hybrid visual and textual functional programming. Contribute to enso-org/enso development by creating an account on GitHub.

GregoryTravis avatar Sep 19 '23 18:09 GregoryTravis

Added to the list: java.io.IOException: Could not create lock for \\.\pipe\sbt-load-802998070017202719_lock, error 1336

* https://github.com/enso-org/enso/actions/runs/6238498583/job/16934446773?pr=7847

GitHub**Improve HTTP tests · enso-org/enso@fc8ad0e**Hybrid visual and textual functional programming. Contribute to enso-org/enso development by creating an account on GitHub.

This is actually the same error as under sbt: ServerAlreadyBootingException category - https://github.com/enso-org/enso/actions/runs/6238498583/job/16934446773?pr=7847#step:10:676. Moving it up in the list.

GitHub
Hybrid visual and textual functional programming. Contribute to enso-org/enso development by creating an account on GitHub.

Akirathan avatar Sep 20 '23 08:09 Akirathan

I added to the list an error I saw on my CI build:

 INFO ide_ci::program::command: sbtℹ️ /runner/_work/enso/enso/distribution/lib/Standard/Base/0.0.0-dev/src/Data.enso: error: The module Standard.Base.Data.Enso_Cloud does not exist.
 INFO ide_ci::program::command: sbtℹ️ [error] Test org.enso.interpreter.test.MetaIsATest.checkNumberIsNotInstanceOfNumber failed: org.graalvm.polyglot.PolyglotException: Compilation aborted due to errors., took 0.066 sec
 INFO ide_ci::program::command: sbtℹ️ [error]     at org.graalvm.polyglot.Value.invokeMember(Value.java:974)
 INFO ide_ci::program::command: sbtℹ️ [error]     at org.enso.interpreter.test.ValuesGenerator.v(ValuesGenerator.java:71)
 INFO ide_ci::program::command: sbtℹ️ [error]     at org.enso.interpreter.test.ValuesGenerator.v(ValuesGenerator.java:60)
 INFO ide_ci::program::command: sbtℹ️ [error]     at org.enso.interpreter.test.ValuesGenerator.typeNumber(ValuesGenerator.java:157)
 INFO ide_ci::program::command: sbtℹ️ [error]     at org.enso.interpreter.test.MetaIsATest.checkNumberIsNotInstanceOfNumber(MetaIsATest.java:150)
 INFO ide_ci::program::command: sbtℹ️ [error]     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 INFO ide_ci::program::command: sbtℹ️ [error]     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
 INFO ide_ci::program::command: sbtℹ️ [error]     at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 INFO ide_ci::program::command: sbtℹ️ [error]     at java.lang.reflect.Method.invoke(Method.java:568)
 INFO ide_ci::program::command: sbtℹ️ [error]     ...

So apparently it looks like an import Standard.Base.Data.Enso_Cloud that comes from a different PR must have somehow leaked to my branch.

radeusgd avatar Oct 26 '23 18:10 radeusgd

I added one more to the list - the startup time benchmark seems to be failing from time to time with

 INFO ide_ci::program::command: cmdℹ️ Found 2 cases to execute (ETA 50s)
 INFO ide_ci::program::command: cmdℹ️ Benchmarking 'Startup.empty_startup' with configuration: [warmup={2 iterations, 5 seconds each}, measurement={3 iterations, 5 seconds each}]
 INFO ide_ci::program::command: cmdℹ️ Execution finished with an error: Cannot find enso-engine-* in C:\runner\_work\enso\enso\built-distribution
ERROR ide_ci::program::command: cmdℹ️ Failed to decode a line from output: invalid utf-8 sequence of 1 bytes from index 5
 WARN ide_ci::program::command: cmdℹ️ Raw buffer: [67, 58, 92, 114, 32, 133, 13]. Decoded with placeholders: C:\r �
 INFO ide_ci::program::command: cmdℹ️         at <enso> Panic.throw(Internal)
 INFO ide_ci::program::command: cmdℹ️         at <enso> find_prefix<arg-2>(src\Startup\Startup.enso:55:13-27)
 INFO ide_ci::program::command: cmdℹ️         at <enso> find_prefix(src\Startup\Startup.enso:51-55)
 INFO ide_ci::program::command: cmdℹ️         at <enso> Startup.enso_bin(src\Startup\Startup.enso:60:19-63)
 INFO ide_ci::program::command: cmdℹ️         at <enso> suspended-block<arg-1>(src\Startup\Startup.enso:37:20-27)
 INFO ide_ci::program::command: cmdℹ️         at <enso> null(Internal)
 INFO ide_ci::program::command: cmdℹ️         at <enso> Data.enso_bin(Internal)
 INFO ide_ci::program::command: cmdℹ️         at <enso> Data.startup(src\Startup\Startup.enso:18:15-27)
 INFO ide_ci::program::command: cmdℹ️         at <enso> Data.bench_empty(src\Startup\Startup.enso:12:9-58)
 INFO ide_ci::program::command: cmdℹ️         at <enso> <anonymous><arg-2>(src\Startup\Startup.enso:40:47-62)
 INFO ide_ci::program::command: cmdℹ️         at <enso> <anonymous>(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Test\2023.2.1-dev\src\Bench.enso:74:54-62)
 INFO ide_ci::program::command: cmdℹ️         at <enso> <anonymous><arg-4>(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Test\2023.2.1-dev\src\Bench.enso:172:58-65)
 INFO ide_ci::program::command: cmdℹ️         at <enso> Runtime.no_inline(Internal)
 INFO ide_ci::program::command: cmdℹ️         at <enso> Bench.type.single_call(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Test\2023.2.1-dev\src\Bench.enso:214:13-33)
 INFO ide_ci::program::command: cmdℹ️         at <enso> case_branch(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Test\2023.2.1-dev\src\Bench.enso:196:31-51)
 INFO ide_ci::program::command: cmdℹ️         at <enso> Bench.type.measure(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Test\2023.2.1-dev\src\Bench.enso:194-207)
 INFO ide_ci::program::command: cmdℹ️         at <enso> <anonymous>(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Test\2023.2.1-dev\src\Bench.enso:172:13-66)
 INFO ide_ci::program::command: cmdℹ️         at <enso> <anonymous>(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Test\2023.2.1-dev\src\Bench.enso:135:60-70)
 INFO ide_ci::program::command: cmdℹ️         at <enso> f(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Base\2023.2.1-dev\src\Data\Vector.enso:258:26-50)
 INFO ide_ci::program::command: cmdℹ️         at <enso> go<arg-2>(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Base\2023.2.1-dev\src\Data\Range.enso:312:31-50)
 INFO ide_ci::program::command: cmdℹ️         at <enso> go(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Base\2023.2.1-dev\src\Data\Range.enso:311-313)
 INFO ide_ci::program::command: cmdℹ️         at <enso> Range.fold<arg-2>(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Base\2023.2.1-dev\src\Data\Range.enso:314:13-30)
 INFO ide_ci::program::command: cmdℹ️         at <enso> Range.fold(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Base\2023.2.1-dev\src\Data\Range.enso:308-314)
 INFO ide_ci::program::command: cmdℹ️         at <enso> Vector.fold(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Base\2023.2.1-dev\src\Data\Vector.enso:259:9-41)
 INFO ide_ci::program::command: cmdℹ️         at <enso> case_branch(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Test\2023.2.1-dev\src\Bench.enso:135:34-71)
 INFO ide_ci::program::command: cmdℹ️         at <enso> Bench.fold(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Test\2023.2.1-dev\src\Bench.enso:133-136)
 INFO ide_ci::program::command: cmdℹ️         at <enso> <anonymous>(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Test\2023.2.1-dev\src\Bench.enso:134:56-66)
 INFO ide_ci::program::command: cmdℹ️         at <enso> f(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Base\2023.2.1-dev\src\Data\Vector.enso:258:26-50)
 INFO ide_ci::program::command: cmdℹ️         at <enso> go<arg-2>(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Base\2023.2.1-dev\src\Data\Range.enso:312:31-50)
 INFO ide_ci::program::command: cmdℹ️         at <enso> go(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Base\2023.2.1-dev\src\Data\Range.enso:311-313)
 INFO ide_ci::program::command: cmdℹ️         at <enso> Range.fold<arg-2>(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Base\2023.2.1-dev\src\Data\Range.enso:314:13-30)
 INFO ide_ci::program::command: cmdℹ️         at <enso> Range.fold(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Base\2023.2.1-dev\src\Data\Range.enso:308-314)
 INFO ide_ci::program::command: cmdℹ️         at <enso> Vector.fold(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Base\2023.2.1-dev\src\Data\Vector.enso:259:9-41)
 INFO ide_ci::program::command: cmdℹ️         at <enso> case_branch(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Test\2023.2.1-dev\src\Bench.enso:134:29-67)
 INFO ide_ci::program::command: cmdℹ️         at <enso> Bench.fold(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Test\2023.2.1-dev\src\Bench.enso:133-136)
 INFO ide_ci::program::command: cmdℹ️         at <enso> Bench.run_main(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Test\2023.2.1-dev\src\Bench.enso:168-172)
 INFO ide_ci::program::command: cmdℹ️         at <enso> <anonymous>(src\Main.enso:118:9-22)
 INFO ide_ci::program::command: cmdℹ️         at <enso> <anonymous>(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Base\2023.2.1-dev\src\Data\Vector.enso:723:13-26)
 INFO ide_ci::program::command: cmdℹ️         at <enso> go<arg-2>(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Base\2023.2.1-dev\src\Data\Range.enso:243:21-36)
 INFO ide_ci::program::command: cmdℹ️         at <enso> go(C:\runner\_work\enso\enso\built-distribution\enso-engine-2023.2.1-dev-windows-amd64\enso-2023.2.1-dev\lib\Standard\Base\2023.2.1-dev\src\Data\Range.enso:242-244)

radeusgd avatar Dec 19 '23 16:12 radeusgd

Another case of RuntimeServerTest - The language did not complete all polyglot threads: https://github.com/enso-org/enso/actions/runs/7639249532/job/20811863728?pr=8844#step:10:3283 Log archive

GitHub
Hybrid visual and textual functional programming. Contribute to enso-org/enso development by creating an account on GitHub.

mwu-tow avatar Jan 25 '24 16:01 mwu-tow

A new one: Transient Engine Test Failures: RuntimeTextEditsTest and RuntimeSuggestionUpdatesTest

mwu-tow avatar Feb 05 '24 14:02 mwu-tow

I think I found one not yet listed above: should be able to use external libraries if they are needed by the visualization *** FAILED *** https://github.com/enso-org/enso/actions/runs/7786527964/job/21231559557?pr=8972#step:8:2204

GitHub
Hybrid visual and textual functional programming. Contribute to enso-org/enso development by creating an account on GitHub.

farmaazon avatar Feb 06 '24 08:02 farmaazon

Reported next one: Transient Engine Test Failure: S3.get_object "Read timed out"

mwu-tow avatar Feb 20 '24 15:02 mwu-tow

Another one: HttpDownloaderTest.downloadBigFileWithProgress

mwu-tow avatar Feb 26 '24 23:02 mwu-tow

A new one:

  • #9377

mwu-tow avatar Mar 12 '24 11:03 mwu-tow

  • https://github.com/enso-org/enso/issues/9457

mwu-tow avatar Mar 18 '24 14:03 mwu-tow