runtime icon indicating copy to clipboard operation
runtime copied to clipboard

[browser][MT] RuntimeError: memory access out of bounds

Open pavelsavara opened this issue 2 years ago • 20 comments

Error Blob

{
  "ErrorMessage": "RuntimeError: memory access out of bounds",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Reproduction Steps

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=515167 Log

[20:26:27] warn: worker sent an error! http://127.0.0.1:39745/_framework/dotnet.native.worker.js:1: Uncaught RuntimeError: memory access out of bounds
[20:26:27] fail: MONO_WASM: Error
                     at Ze (http://127.0.0.1:39745/_framework/dotnet.js:3:19735)
                     at e (http://127.0.0.1:39745/_framework/dotnet.js:3:33729)
                     at http://127.0.0.1:39745/_framework/dotnet.js:3:33874
[20:26:27] warn: MONO_WASM: mono_wasm_exit failed unwind

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=515167 Error message validated: RuntimeError: memory access out of bounds Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 1/5/2024 12:55:21 PM UTC

Report

Build Definition Test Pull Request
658853 dotnet/runtime WasmTestOnChrome-MT-System.Linq.Expressions.Tests.WorkItemExecution
658682 dotnet/runtime WasmTestOnChrome-MT-System.Runtime.InteropServices.Tests.WorkItemExecution dotnet/runtime#101330
655218 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101318
655451 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution
653807 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101442
654744 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101330
654556 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101478
654532 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#100775
653837 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101445
653677 dotnet/runtime WasmTestOnChrome-MT-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#100697
653288 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101329
653246 dotnet/runtime WasmTestOnChrome-System.Net.Http.Functional.Tests.WorkItemExecution dotnet/runtime#101408
653187 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101397
653143 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101342
652989 dotnet/runtime WasmTestOnChrome-System.Runtime.InteropServices.JavaScript.Tests.WorkItemExecution dotnet/runtime#101336
652546 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101312
652321 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101390
652318 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101392
652294 dotnet/runtime WasmTestOnV8-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101391
651705 dotnet/runtime WasmTestOnChrome-System.Net.Http.Functional.Tests.WorkItemExecution dotnet/runtime#101133
651501 dotnet/runtime WasmTestOnChrome-System.Data.Common.Tests.WorkItemExecution dotnet/runtime#101106
650709 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101339
650687 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101334
650580 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101329
650515 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#100497
650490 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution
649911 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution
649790 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101265
649425 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101179
649358 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101234
648959 dotnet/runtime WasmTestOnChrome-Microsoft.CSharp.Tests.WorkItemExecution dotnet/runtime#101263
648595 dotnet/runtime Workloads-NoWebcil-Wasm.Build.Tests.Blazor.SimpleRunTests.WorkItemExecution dotnet/runtime#101217
648560 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101263
648322 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101252
647882 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#100094
647786 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101226
647762 dotnet/runtime WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#100697
646615 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#100094
646414 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101168
646303 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101106
646378 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#101122
639588 dotnet/runtime WasmTestOnBrowser-System.IO.FileSystem.Tests.WorkItemExecution dotnet/runtime#100587
639593 dotnet/runtime WasmTestOnBrowser-System.IO.FileSystem.Tests.WorkItemExecution dotnet/runtime#100669
639431 dotnet/runtime normal-System.Text.Encodings.Web.Tests.WorkItemExecution dotnet/runtime#100812
639252 dotnet/runtime WasmTestOnBrowser-System.Text.Encodings.Web.Tests.WorkItemExecution dotnet/runtime#100877
638143 dotnet/runtime WasmTestOnBrowser-System.IO.FileSystem.Tests.WorkItemExecution dotnet/runtime#100877
638062 dotnet/runtime WasmTestOnBrowser-System.Data.Common.Tests.WorkItemExecution dotnet/runtime#100846
638079 dotnet/runtime WasmTestOnBrowser-System.IO.MemoryMappedFiles.Tests.WorkItemExecution dotnet/runtime#100873
637900 dotnet/runtime Methodical_d2.WorkItemExecution dotnet/runtime#100669
637717 dotnet/runtime normal-System.Text.Encodings.Web.Tests.WorkItemExecution dotnet/runtime#100520
637618 dotnet/runtime Methodical_d2.WorkItemExecution dotnet/runtime#100407
637425 dotnet/runtime Methodical_d2.WorkItemExecution dotnet/runtime#100855
636936 dotnet/runtime Methodical_r2.WorkItemExecution dotnet/runtime#100157
636875 dotnet/runtime Methodical_d2.WorkItemExecution dotnet/runtime#99829
636761 dotnet/runtime Methodical_r2.WorkItemExecution dotnet/runtime#100767
636681 dotnet/runtime Methodical_d2.WorkItemExecution dotnet/runtime#100048
636684 dotnet/runtime Methodical_d2.WorkItemExecution dotnet/runtime#100832
636653 dotnet/runtime Methodical_r2.WorkItemExecution dotnet/runtime#100819
636603 dotnet/runtime Methodical_d2.WorkItemExecution dotnet/runtime#100802
635807 dotnet/runtime Methodical_d2.WorkItemExecution dotnet/runtime#100801
636303 dotnet/runtime Methodical_d2.WorkItemExecution dotnet/runtime#100685
635977 dotnet/runtime Methodical_d2.WorkItemExecution
634881 dotnet/runtime Methodical_d2.WorkItemExecution dotnet/runtime#100775
634835 dotnet/runtime Methodical_d2.WorkItemExecution dotnet/runtime#100765
634832 dotnet/runtime Methodical_r2.WorkItemExecution dotnet/runtime#91042
634558 dotnet/runtime Methodical_d2.WorkItemExecution dotnet/runtime#100520
634482 dotnet/runtime Methodical_d2.WorkItemExecution dotnet/runtime#86638
634470 dotnet/runtime Methodical_d2.WorkItemExecution dotnet/runtime#100610
634402 dotnet/runtime Methodical_d2.WorkItemExecution dotnet/runtime#100702
634087 dotnet/runtime Methodical_r2.WorkItemExecution dotnet/runtime#99339
633984 dotnet/runtime Methodical_d2.WorkItemExecution dotnet/runtime#99339
633836 dotnet/runtime Methodical_d2.WorkItemExecution dotnet/runtime#100520
633815 dotnet/runtime Methodical_r2.WorkItemExecution dotnet/runtime#100726
633202 dotnet/runtime Methodical_d2.WorkItemExecution dotnet/runtime#99339
632984 dotnet/runtime Methodical_d2.WorkItemExecution dotnet/runtime#100520
632888 dotnet/runtime Methodical_d2.WorkItemExecution
632798 dotnet/runtime Methodical_d2.WorkItemExecution dotnet/runtime#100661
632701 dotnet/runtime Methodical_d2.WorkItemExecution dotnet/runtime#100718
632248 dotnet/runtime Methodical_r2.WorkItemExecution dotnet/runtime#100630
631682 dotnet/runtime WasmTestOnBrowser-System.Text.Encodings.Web.Tests.WorkItemExecution dotnet/runtime#100671
627292 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution dotnet/runtime#100415

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
2 21 81

pavelsavara avatar Jan 05 '24 12:01 pavelsavara

Tagging subscribers to 'arch-wasm': @lewing See info in area-owners.md if you want to be subscribed.

Issue Details

Error Blob

{
  "ErrorMessage": "RuntimeError: memory access out of bounds",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Reproduction Steps

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=515167 Log

[20:26:27] warn: worker sent an error! http://127.0.0.1:39745/_framework/dotnet.native.worker.js:1: Uncaught RuntimeError: memory access out of bounds
[20:26:27] fail: MONO_WASM: Error
                     at Ze (http://127.0.0.1:39745/_framework/dotnet.js:3:19735)
                     at e (http://127.0.0.1:39745/_framework/dotnet.js:3:33729)
                     at http://127.0.0.1:39745/_framework/dotnet.js:3:33874
[20:26:27] warn: MONO_WASM: mono_wasm_exit failed unwind```

<table>
  <tr>
    <th align="left">Author:</th>
    <td>pavelsavara</td>
  </tr>
  <tr>
    <th align="left">Assignees:</th>
    <td>pavelsavara</td>
  </tr>
  <tr>
    <th align="left">Labels:</th>
    <td>

`arch-wasm`, `blocking-clean-ci`, `area-VM-threading-mono`, `Known Build Error`, `os-browser`

</td>
  </tr>
  <tr>
    <th align="left">Milestone:</th>
    <td>9.0.0</td>
  </tr>
</table>
</details>

ghost avatar Jan 05 '24 12:01 ghost

Another Build Log

[23:11:59] info: [2024-01-10T23:11:59.397Z] [PASS] System.Threading.ThreadPools.Tests.ThreadPoolTests.RunProcessorCountItemsInParallel
[23:11:59] info: [2024-01-10T23:11:59.400Z] [SKIP] System.Threading.ThreadPools.Tests.ThreadPoolTests.SetMinMaxThreadsTest_ChangedInDotNetCore
[23:12:00] warn: worker sent an error! http://127.0.0.1:33633/_framework/dotnet.native.worker.js:1: Uncaught RuntimeError: memory access out of bounds
[23:12:00] fail: MONO_WASM: Error
                     at Ze (http://127.0.0.1:33633/_framework/dotnet.js:3:19735)
                     at e (http://127.0.0.1:33633/_framework/dotnet.js:3:33729)
                     at http://127.0.0.1:33633/_framework/dotnet.js:3:33874
[23:12:00] info: WASM EXIT 1
[23:12:00] info: Waiting to flush log messages with a timeout of 120 secs ..

pavelsavara avatar Jan 11 '24 10:01 pavelsavara

Another Log Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=549038&view=logs&jobId=63c2d0c8-fec2-5788-81c8-f3ac95e8841f

[11:44:47] fail: [0x066a4300-long 11:44:47.111] MONO_WASM: Unhandled error: {}
[11:44:47] fail: [0x066a4300-long 11:44:47.117] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[8732]:0x217b29
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[2731]:0xd2e2b
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[2815]:0xd597a
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[2077]:0xaed21
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[324]:0x286c7
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[253]:0x25e62
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[244]:0x18884
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[283]:0x26af0
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[3319]:0xf9db8
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[2587]:0xcbe17
[11:44:47] info: WASM EXIT 1

pavelsavara avatar Feb 01 '24 12:02 pavelsavara

Log

[08:55:30] info: Starting:    Microsoft.Extensions.Logging.Console.Tests.dll
[08:55:31] fail: [0x045f6788-emsc 08:55:31.650] MONO_WASM: preRunWorker() failed {}
[08:55:31] fail: [0x045f6788-emsc 08:55:31.652] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at http://127.0.0.1:44985/_framework/dotnet.native.wasm:wasm-function[649]:0x4a787
                     at http://127.0.0.1:44985/_framework/dotnet.native.wasm:wasm-function[650]:0x4a7d6
                     at http://127.0.0.1:44985/_framework/dotnet.native.wasm:wasm-function[652]:0x4a8ab
                     at http://127.0.0.1:44985/_framework/dotnet.native.wasm:wasm-function[530]:0x4793a
                     at e.<computed> (http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:226093)
                     at http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:101299
                     at Ga (http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:101426)
                     at Ya (http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:101713)
                     at http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:212904
                     at iu (http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:213061)
[08:55:31] info: [0x000dc84c-main 08:55:31.655] MONO_WASM: Dumping web worker info as seen by UI thread, it could be stale:
[08:55:31] info: 00 | 0x000dc84c-main: isRunning:false isAttached: true isEventLoop:false reuseCount:  1 - UI Thread
[08:55:31] info: 01 | 0x02ff8f90-norm: isRunning: true isAttached: true isEventLoop:false reuseCount:  1 - Finalizer
[08:55:31] info: 02 | 0x032a8030-pool: isRunning: true isAttached: true isEventLoop: true reuseCount:  1 - .NET TP Worker
[08:55:31] info: 03 | 0x035b0040-gate: isRunning: true isAttached: true isEventLoop:false reuseCount:  1 - .NET TP Gate
[08:55:31] info: 04 | 0x03dbea48-pool: isRunning: true isAttached: true isEventLoop: true reuseCount:  1 - .NET TP Worker
[08:55:31] info: 05 | 0x045f6788-emsc: isRunning: true isAttached:false isEventLoop:false reuseCount:  1 - pthread-assigned
[08:55:31] info: 06 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 07 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 08 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 09 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 10 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 11 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 12 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] fail: [0x000dc84c-main 08:55:31.657] MONO_WASM: Uncaught RuntimeError: memory access out of bounds
                 Error
                     at Object.Ge [as mono_exit] (http://127.0.0.1:44985/_framework/dotnet.js:3:20140)
                     at ft.config.exitOnUnhandledError.e.onerror (http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:38634)
[08:55:31] fail: [out of order message from the browser]: http://127.0.0.1:44985/_framework/dotnet.native.js 7:1295 Uncaught ErrorEvent: Uncaught RuntimeError: memory access out of bounds

pavelsavara avatar Feb 12 '24 08:02 pavelsavara

During thread creation Log Build

[18:39:28] fail: [--06290040-emsc 18:39:28.771] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at pthread_attr_init (http://127.0.0.1:34537/_framework/dotnet.native.wasm:wasm-function[8640]:0x2144c1)
                     at mono_thread_info_attach (http://127.0.0.1:34537/_framework/dotnet.native.wasm:wasm-function[786]:0x4ef10)
                     at start_wrapper (http://127.0.0.1:34537/_framework/dotnet.native.wasm:wasm-function[2762]:0xd6903)
                     at Object.invokeEntryPoint (http://127.0.0.1:34537/_framework/dotnet.native.js:8:69009)
                     at handleMessage (http://127.0.0.1:34537/_framework/dotnet.native.worker.js:1:2306)

pavelsavara avatar Feb 17 '24 15:02 pavelsavara

Related https://github.com/dotnet/runtime/issues/98630

pavelsavara avatar Feb 18 '24 15:02 pavelsavara

Log

[11:23:26] info: 001 | 0x03078030-pool: isRunning:false isAttached: true isEventLoop: true reuseCount:  1 - .NET TP Worker
[11:23:26] fail: [0x03078030-pool 11:23:26.696] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at dlmalloc (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[8788]:0x21d1e6)
                     at dlcalloc (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[8796]:0x21f96a)
                     at monoeg_g_calloc (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[598]:0x49521)
                     at monoeg_malloc0 (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[599]:0x49544)
                     at monoeg_g_array_new (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[569]:0x48b40)
                     at mono_interp_transform_method (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[438]:0x407b6)
                     at do_transform_method (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[255]:0x25481)
                     at mono_interp_exec_method (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[254]:0x251fb)
                     at interp_entry (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[327]:0x28391)
                     at interp_entry_static_2 (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[365]:0x29411)

pavelsavara avatar Feb 20 '24 23:02 pavelsavara

cc @vargaz @steveisok

lewing avatar Feb 21 '24 01:02 lewing

local run with nice stack trace

fail: [0x05a6fef0-jsww 15:29:09.596] MONO_WASM: memory access out of bounds
  RuntimeError: memory access out of bounds
      at dlfree (dlfree (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[17333]:0x34eda4))
      at bound_fn (http://127.0.0.1:61068/_framework/dotnet.runtime.js:4146:24)
      at sync_bound_fn (http://127.0.0.1:61068/_framework/dotnet.runtime.js:3980:13)
      at JSImport_INTERNAL_http_wasm_supports_streaming_request (https://dotnet/JSImport/INTERNAL_http_wasm_supports_streaming_request:4:86)
      at mono_wasm_invoke_jsimport (http://127.0.0.1:61068/_framework/dotnet.runtime.js:3893:5)
      at do_icall (do_icall (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[231]:0x29de6))
      at do_icall_wrapper (do_icall_wrapper (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[151]:0x26971))
      at mono_interp_exec_method (mono_interp_exec_method (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[142]:0x150ac))
      at interp_runtime_invoke (interp_runtime_invoke (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[185]:0x27d50))
      at mono_jit_runtime_invoke (mono_jit_runtime_invoke (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[9174]:0x200a97))

pavelsavara avatar Mar 10 '24 16:03 pavelsavara

another local with great log

MONO_WASM: memory access out of bounds
RuntimeError: memory access out of bounds
    at copy_object_no_checks.1 (copy_object_no_checks.1 (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[2325]:0xa15c1))
    at simple_nursery_serial_scan_object (simple_nursery_serial_scan_object (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[2321]:0x9f935))
    at simple_nursery_serial_drain_gray_stack (simple_nursery_serial_drain_gray_stack (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[2324]:0xa1454))
    at sgen_drain_gray_stack (sgen_drain_gray_stack (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1721]:0x7c5ca))
    at finish_gray_stack (finish_gray_stack (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1828]:0x812be))
    at collect_nursery (collect_nursery (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1744]:0x7d6ee))
    at sgen_perform_collection_inner (sgen_perform_collection_inner (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1741]:0x7d11f))
    at sgen_perform_collection (sgen_perform_collection (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1740]:0x7d001))
    at sgen_ensure_free_space (sgen_ensure_free_space (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1739]:0x7cfac))
    at sgen_alloc_obj_nolock (sgen_alloc_obj_nolock (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1669]:0x7af3e))```

pavelsavara avatar Mar 26 '24 16:03 pavelsavara

another local

RuntimeError: memory access out of bounds
    at dlfree (dlfree (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[17410]:0x34e69e))
    at bound_fn (https://localhost:65453/_framework/dotnet.runtime.js:4217:24)
    at async_bound_fn (https://localhost:65453/_framework/dotnet.runtime.js:4027:9)
    at JSImport_INTERNAL_dynamic_import (https://dotnet/JSImport/INTERNAL_dynamic_import:4:64)
    at mono_wasm_invoke_jsimport_MT (https://localhost:65453/_framework/dotnet.runtime.js:3948:5)
    at do_icall (do_icall (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[227]:0x29b34))
    at do_icall_wrapper (do_icall_wrapper (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[147]:0x266c6))
    at mono_interp_exec_method (mono_interp_exec_method (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[138]:0x150fd))
    at interp_runtime_invoke (interp_runtime_invoke (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[181]:0x27aa5))
    at mono_jit_runtime_invoke (mono_jit_runtime_invoke (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[9240]:0x2001ac))```

pavelsavara avatar Mar 26 '24 17:03 pavelsavara

Most of the problems are fixed by https://github.com/dotnet/runtime/pull/100304

There is new occurrence as fallout of OOM Log

[19:33:04] info: [FAIL] System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers(options: System.Text.Json.JsonWriterOptions, keyString: "mess><age")
[19:33:04] info: System.OutOfMemoryException : Out of memory
[19:33:04] info:    at System.GC.AllocateArray[Char](Int32 length, Boolean pinned)
[19:33:04] info:    at System.Text.StringBuilder.ExpandByABlock(Int32 minBlockCharCount)
[19:33:04] info:    at System.Text.StringBuilder.AppendWithExpansion(Char value)
[19:33:04] info:    at System.Text.StringBuilder.Append(Char value)
[19:33:04] info:    at System.IO.StringWriter.Write(Char value)
[19:33:04] info:    at Newtonsoft.Json.JsonTextWriter.WriteEnd(JsonToken token)
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.AutoCompleteClose(JsonContainerType type)
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.WriteEndArray()
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.WriteEnd(JsonContainerType type)
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.WriteEnd()
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.AutoCompleteAll()
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.Close()
[19:33:04] info:    at Newtonsoft.Json.JsonTextWriter.Close()
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.Dispose(Boolean disposing)
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.System.IDisposable.Dispose()
[19:33:04] info:    at System.Text.Json.Tests.Utf8JsonWriterTests.GetExpectedLargeArrayOfStrings(Int32 length)
[19:33:04] info:    at System.Text.Json.Tests.Utf8JsonWriterTests.WriteJsonOnlyWritesToStreamOnDemand_FlushAsync()
[19:33:04] info: --- End of stack trace from previous location ---
[19:33:04] info:    at System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers(JsonWriterOptions options, String keyString)
[19:33:04] info:    at System.Object.InvokeStub_Utf8JsonWriterTests.WriteNumbers(Object , Span`1 )
[19:33:04] info:    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
[19:33:04] fail: [0x020cdfc0-dpty 19:33:04.706] MONO_WASM: Unhandled error: RuntimeError: memory access out of bounds
[19:33:04] fail: [0x020cdfc0-dpty 19:33:04.707] MONO_WASM: Unhandled error: RuntimeError: memory access out of bounds
[19:33:04] info: [0x020cdfc0-dpty 19:33:04.710] MONO_WASM: Dumping web worker info as seen by UI thread, it could be stale:
[19:33:04] info: 001 | 0x020cdfc0-dpty: isRunning: true isAttached: true isEventLoop:false reuseCount:  1 -
[19:33:04] fail: [0x020cdfc0-dpty 19:33:04.748] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at copy_object_no_checks.1 (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[1115]:0x637d7)
                     at simple_nursery_serial_scan_object (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[1113]:0x63003)
                     at simple_nursery_serial_drain_gray_stack (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[1110]:0x626d8)
                     at finish_gray_stack (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[955]:0x56d6a)
                     at collect_nursery (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[927]:0x5481b)
                     at sgen_perform_collection (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[925]:0x540da)
                     at sgen_ensure_free_space (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[924]:0x53f75)
                     at sgen_alloc_obj_nolock (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[911]:0x533bc)
                     at mono_gc_alloc_string (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[3174]:0xf0e73)
                     at mono_string_new_size_checked (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[2645]:0xd1efe)
                     at mono_interp_exec_method (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[257]:0x208b2)
                     at interp_entry (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[330]:0x286f9)
                     at interp_entry_static_0 (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[366]:0x29702)
                     at wasm_native_to_interp_System_Runtime_InteropServices_JavaScript_JSSynchronizationContext_BackgroundJobHandler (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[232]:0x1676e)
                     at mono_background_exec (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[836]:0x50e85)
                     at Os (http://127.0.0.1:35135/_framework/dotnet.runtime.js:3:69709)
                     at callUserCallback (http://127.0.0.1:35135/_framework/dotnet.native.js:8:102608)
                     at http://127.0.0.1:35135/_framework/dotnet.native.js:8:111163
[19:33:04] info: WASM EXIT 1

pavelsavara avatar Apr 03 '24 06:04 pavelsavara

This incorrectly ate https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-100801-merge-8c5cd227f4844ef98c/Methodical_d2/1/console.e95f9cb2.log?helixlogtype=result from a non-MT lane, which looks like it could be a real issue.

kg avatar Apr 09 '24 20:04 kg

This incorrectly ate https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-100801-merge-8c5cd227f4844ef98c/Methodical_d2/1/console.e95f9cb2.log?helixlogtype=result from a non-MT lane, which looks like it could be a real issue.

I already opened https://github.com/dotnet/runtime/issues/100757 for it

pavelsavara avatar Apr 10 '24 06:04 pavelsavara

this appears to have been fixed with https://github.com/dotnet/runtime/pull/100907 closing so that new failures are caught

lewing avatar Apr 13 '24 22:04 lewing

hit again in https://github.com/dotnet/runtime/pull/101179

lewing avatar Apr 18 '24 00:04 lewing

There are 4 more this morning, 3 of them are after OOM. The 4th is in GC (probably on the edge of OOM too) https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-101122-merge-f513e5fd0d4a45d08e/WasmTestOnBrowser-System.Text.Json.Tests/1/console.93cb60d2.log?helixlogtype=result

Perhaps some alloc failed and returned NULL and we are not checking it ?

pavelsavara avatar Apr 18 '24 07:04 pavelsavara

Alternatively, are we missing some memory fence which would synchronize the memory between WASM threads ?

We do issue .grow(0) in mono_wasm_synchronization_context_pump but sgen_perform_collection could be doing lot of work and the memory could grow on another thread in the meantime.

[09:03:07] fail: [0x020ff9e8-dpty 09:03:07.808] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at copy_object_no_checks.1 (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[1119]:0x677c9)
                     at simple_nursery_serial_scan_object (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[1117]:0x66fe9)
                     at simple_nursery_serial_drain_gray_stack (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[1114]:0x666be)
                     at finish_gray_stack (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[957]:0x5a74b)
                     at collect_nursery (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[930]:0x5813f)
                     at sgen_perform_collection (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[928]:0x579f4)
                     at sgen_ensure_free_space (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[927]:0x5788d)
                     at sgen_alloc_obj_nolock (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[914]:0x56d40)
                     at mono_gc_alloc_vector (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[3167]:0xfab94)
                     at mono_array_new_specific_internal (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[2710]:0xddfeb)
                     at mono_array_new_specific_checked (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[2711]:0xde022)
                     at mono_interp_exec_method (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[257]:0x21463)
                     at interp_entry (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[330]:0x297e8)
                     at interp_entry_static_0 (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[366]:0x2a830)
                     at wasm_native_to_interp_System_Runtime_InteropServices_JavaScript_JSSynchronizationContext_PumpHandler (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[232]:0x171f1)
                     at mono_wasm_synchronization_context_pump (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[206]:0x1556e)

pavelsavara avatar Apr 18 '24 07:04 pavelsavara

I think under normal circumstances we shouldn't need to .grow(0), the theory was that we specifically needed to do it if the debugger paused thread(s) while a grow operation happened, wasn't it?

Unchecked null pointer feels very plausible. It will fail silently and also corrupt the zero page (which will cause jiterp traces to break.)

kg avatar Apr 18 '24 07:04 kg

I saw Log

[13:54:11] fail: Error in mono_download_assets: RangeError: Start offset -1 is outside the bounds of the buffer
[13:54:11] fail: [--000eafbc-emsc 13:54:11.333] MONO_WASM: Start offset -1 is outside the bounds of the buffer
                 RangeError: Start offset -1 is outside the bounds of the buffer
                     at new Uint8Array (<anonymous>)
                     at http://127.0.0.1:41625/_framework/dotnet.runtime.js:3:77638
                     at Object.pa [as instantiate_asset] (http://127.0.0.1:41625/_framework/dotnet.runtime.js:3:77686)
                     at n (http://127.0.0.1:41625/_framework/dotnet.js:3:8629)
                     at async Promise.all (index 129)
[13:54:11] fail: [out of order message from the browser]: http://127.0.0.1:41625/_framework/dotnet.runtime.js 2:77637 Uncaught RangeError: Start offset -1 is outside the bounds of the buffer
[13:54:11] info: WASM EXIT 1

This translates to ICU asset data being loaded into memory via mono_wasm_load_bytes_into_heap_persistent on MT build in UI thread, while the mono is starting on deputy thread.

It seems that Module._sbrk returned -1 🤯

It seems that sbrk could be racing with other threads and loose. https://github.com/emscripten-core/emscripten/blob/1ca2f3141ec99110d6c5c5690dcd1c65d08d97e5/system/lib/libc/sbrk.c#L75

Also I'm not sure on what version of emscripten we are now W.R.T. https://github.com/emscripten-core/emscripten/pull/20793/files

We are just lucky it was called from JS and that we got nice JS exception. I bet the same could happen to our C code as well. Which would mean we could get RuntimeError: memory access out of bounds unless all our code is handling -1 and NULL returns from sbrk and malloc

pavelsavara avatar Apr 19 '24 07:04 pavelsavara

It looks like this was originally intended to match MT failures, but based on the recent matches in the automated list, it's matching ST failures. That's not good, memory access out of bounds on ST is more likely to be real bugs that we introduced in PRs and now it will be green.

kg avatar Jul 17 '24 17:07 kg

They are JS error, it will not capture the same issue if it happens on Windows target for example.

When access out of bounds happens in ST it only makes troubleshooting MT more difficult.

pavelsavara avatar Jul 22 '24 10:07 pavelsavara