Loading CPU profile in JIT doesn't work

[derekxu16]

The following is a copy of the bug report that @jensjoha had initially filed in the SDK repo:

Say I run

out/ReleaseX64/dart-sdk/bin/dart --enable-vm-service --pause_isolates_on_exit --profiler pkg/_fe_analyzer_shared/test/scanner_benchmark.dart pkg/_fe_analyzer_shared/lib/src/parser/parser_impl.dart --bytes

and launch devtools. I go to the "CPU Profiler" tab, click "Load all CPU samples". Then it just sits there, "Fetching CPU samples" forever. In the javascript console I can see this stuff:

completePageInitializationIfReady_  {message: 'The message port closed before a response was received.'}

Attempting to load DevTools with canvaskit renderer.

Version: 2.43.0, Renderer: canvaskit

WebSocket connection to 'ws://127.0.0.1:46349/uqClOtdj8j8=' failed: 

[Dart Tooling Daemon connection failed.]: WebSocketChannelException: Instance of 'minified:OT'

DevTools log level changed to INFO

Could not find release notes for DevTools version 2.43.0.

Attempted to call extension 'ext.flutter.inspector.structuredErrors', but no service with that name exists

Could not time async operation "cpuProfileProcessingTime" because an exception was thrown:
Null check operator used on a null value
TypeError: Cannot read properties of undefined (reading 'd')
    at bNN.$2$current$parent (http://127.0.0.1:8181/EY_ENZZmO58=/devtools/main.dart.js:89924:44)
    at bNN.$2$current$parent (http://127.0.0.1:8181/EY_ENZZmO58=/devtools/main.dart.js:89928:83)
    at Object.cAi (http://127.0.0.1:8181/EY_ENZZmO58=/devtools/main.dart.js:14693:32)
    at http://127.0.0.1:8181/EY_ENZZmO58=/devtools/main.dart.js:14601:129
    at bOY.a (http://127.0.0.1:8181/EY_ENZZmO58=/devtools/main.dart.js:5131:63)
    at bOY.$2 (http://127.0.0.1:8181/EY_ENZZmO58=/devtools/main.dart.js:62218:14)
    at Object.k (http://127.0.0.1:8181/EY_ENZZmO58=/devtools/main.dart.js:5117:10)
    at Object.To (http://127.0.0.1:8181/EY_ENZZmO58=/devtools/main.dart.js:14631:10)
    at http://127.0.0.1:8181/EY_ENZZmO58=/devtools/main.dart.js:50796:14
    at bOY.a (http://127.0.0.1:8181/EY_ENZZmO58=/devtools/main.dart.js:5131:63)

Uncaught Error: Null check operator used on a null value
    at bNN.$2$current$parent (cpu_profile_model.dart:1261:39)
    at bNN.$2$current$parent (cpu_profile_model.dart:1372:9)
    at Object.cAi (cpu_profile_model.dart:1380:5)
    at cpu_profile_model.dart:562:50
    at bOY.a (async_patch.dart:311:19)
    at bOY.$2 (async_patch.dart:336:23)
    at Object.k (async_patch.dart:241:3)
    at Object.To (cpu_profile_model.dart:556:11)
    at cpu_profile_service.dart:64:42
    at bOY.a (async_patch.dart:311:19)

[zoneGuarded]: Null check operator used on a null value
../../../lib/src/screens/profiler/cpu_profile_model.dart 1261:39                   _extension#0|generateStackFramesJson.processStackFrame
../../../lib/src/screens/profiler/cpu_profile_model.dart 1372:9                    _extension#0|generateStackFramesJson.processStackFrame
../../../lib/src/screens/profiler/cpu_profile_model.dart 1380:5                    CpuSamples.<anonymous extension>.generateStackFramesJson
../../../lib/src/screens/profiler/cpu_profile_model.dart 562:50                    CpuProfileData.generateFromCpuSamples
org-dartlang-sdk:///dart-sdk/lib/_internal/js_runtime/lib/async_patch.dart 311:19  _wrapJsFunctionForAsync
org-dartlang-sdk:///dart-sdk/lib/_internal/js_runtime/lib/async_patch.dart 336:23  _wrapJsFunctionForAsync.<anonymous function>
org-dartlang-sdk:///dart-sdk/lib/_internal/js_runtime/lib/async_patch.dart 241:3   _asyncStartSync
../../../lib/src/screens/profiler/cpu_profile_model.dart 556:11                    CpuProfileData.generateFromCpuSamples
../../../lib/src/screens/profiler/cpu_profile_service.dart 64:42                   CpuProfilerExtension.getCpuProfile
org-dartlang-sdk:///dart-sdk/lib/_internal/js_runtime/lib/async_patch.dart 311:19  _wrapJsFunctionForAsync

I am running through a ssh tunnel with port 8181 open btw.

Worth noting though is that if I instead do

pkg/vm/tool/precompiler2 pkg/_fe_analyzer_shared/test/scanner_benchmark.dart test.aot
out/ReleaseX64/dartaotruntime --enable-vm-service --pause_isolates_on_exit --profiler test.aot pkg/_fe_analyzer_shared/lib/src/parser/parser_impl.dart --bytes

it works. It loads the profile and there are no errors in the console.

[elliette]

@jakemac53 and another one!

[jakemac53]

@derekxu16 This looks related to the fact that you are running through SSH - note that web socket URL that it is trying to connect to WebSocket connection to 'ws://127.0.0.1:46349/uqClOtdj8j8=' failed: - this is not the port you have open (it is a different random one).

Using VsCode and the proxy server its creates I was able to load a profile (even when running devtools on a remote machine through ssh). I had to do some manual munging of the connection URL to use the vscode proxy URL, prior to that I saw the same errors you see.

I did however still see several errors in the JavaScript console, but was ultimately able to load the profile.

[jakemac53]

It looks like when running via the AOT command you provided, it always uses port 8181. So that is why it works.

[jakemac53]

I believe you could work around this by passing explicitly --enable-vm-service=8181 to specify that port.

[jakemac53]

The remaining errors I do see are related to the dart tooling daemon connection which fails (but gracefully). It is the same core issue, it tries to connect to dtd using a local host/port which isn't being forwarded. I think it probably gets the connection URI from a file or something. I am not sure there is much we can do there.

[derekxu16]

This issue was reported by @jensjoha, I just transferred it from the SDK repo to here.

[jakemac53]

cc @jensjoha can you confirm things work if you provide an explicit port that you have forwarded?

I do expect you will see the error connecting to the dart tooling daemon still though.

[jensjoha]

Specifying a port doesn't change things. --enable-vm-service already implies 8181:

--enable-vm-service[=<port>[/<bind-address>]]
  Enables the VM service and listens on specified port for connections
  (default port number is 8181, default bind address is localhost).

You'd have to specify --enable-vm-service=0 to get a random port.

It's weird that it's trying to connect to a different port, the output is

The Dart VM service is listening on http://127.0.0.1:8181/jBuyQl3qlK0=/
The Dart DevTools debugger and profiler is available at: http://127.0.0.1:8181/jBuyQl3qlK0=/devtools/?uri=ws://127.0.0.1:8181/jBuyQl3qlK0=/ws

so it's seemingly asking webtools to connect to port 8181.

That being said, it does stop complaining about not being able to connect if chromoting in, but it still doesn't load and I still get null errors:

Successfully connected to DTD at: ws://127.0.0.1:42383/uojO5f1dPJI=

DevTools log level changed to INFO

[GroupMarkerNotSet(crbug.com/242999)!:A0E01200F4070000]Automatic fallback to software WebGL has been deprecated. Please use the --enable-unsafe-swiftshader flag to opt in to lower security guarantees for trusted content.

[GroupMarkerNotSet(crbug.com/242999)!:A0C06606F4070000]Automatic fallback to software WebGL has been deprecated. Please use the --enable-unsafe-swiftshader flag to opt in to lower security guarantees for trusted content.

Attempted to call extension 'ext.flutter.inspector.structuredErrors', but no service with that name exists

Could not find release notes for DevTools version 2.43.0.

Could not time async operation "cpuProfileProcessingTime" because an exception was thrown:
Null check operator used on a null value
TypeError: Cannot read properties of undefined (reading 'd')
    at bNN.$2$current$parent (http://127.0.0.1:8181/jBuyQl3qlK0=/devtools/main.dart.js:89924:44)
    at bNN.$2$current$parent (http://127.0.0.1:8181/jBuyQl3qlK0=/devtools/main.dart.js:89928:83)
    at bNN.$2$current$parent (http://127.0.0.1:8181/jBuyQl3qlK0=/devtools/main.dart.js:89928:83)
    at bNN.$2$current$parent (http://127.0.0.1:8181/jBuyQl3qlK0=/devtools/main.dart.js:89928:83)
    at bNN.$2$current$parent (http://127.0.0.1:8181/jBuyQl3qlK0=/devtools/main.dart.js:89928:83)
    at bNN.$2$current$parent (http://127.0.0.1:8181/jBuyQl3qlK0=/devtools/main.dart.js:89928:83)
    at bNN.$2$current$parent (http://127.0.0.1:8181/jBuyQl3qlK0=/devtools/main.dart.js:89928:83)
    at bNN.$2$current$parent (http://127.0.0.1:8181/jBuyQl3qlK0=/devtools/main.dart.js:89928:83)
    at bNN.$2$current$parent (http://127.0.0.1:8181/jBuyQl3qlK0=/devtools/main.dart.js:89928:83)
    at bNN.$2$current$parent (http://127.0.0.1:8181/jBuyQl3qlK0=/devtools/main.dart.js:89928:83)

Uncaught Error: Null check operator used on a null value
    at bNN.$2$current$parent (cpu_profile_model.dart:1261:39)
    at bNN.$2$current$parent (cpu_profile_model.dart:1372:9)
    at bNN.$2$current$parent (cpu_profile_model.dart:1372:9)
    at bNN.$2$current$parent (cpu_profile_model.dart:1372:9)
    at bNN.$2$current$parent (cpu_profile_model.dart:1372:9)
    at bNN.$2$current$parent (cpu_profile_model.dart:1372:9)
    at bNN.$2$current$parent (cpu_profile_model.dart:1372:9)
    at bNN.$2$current$parent (cpu_profile_model.dart:1372:9)
    at bNN.$2$current$parent (cpu_profile_model.dart:1372:9)
    at bNN.$2$current$parent (cpu_profile_model.dart:1372:9)

[zoneGuarded]: Null check operator used on a null value
../../../lib/src/screens/profiler/cpu_profile_model.dart 1261:39  _extension#0|generateStackFramesJson.processStackFrame
../../../lib/src/screens/profiler/cpu_profile_model.dart 1372:9   _extension#0|generateStackFramesJson.processStackFrame
../../../lib/src/screens/profiler/cpu_profile_model.dart 1372:9   _extension#0|generateStackFramesJson.processStackFrame
../../../lib/src/screens/profiler/cpu_profile_model.dart 1372:9   _extension#0|generateStackFramesJson.processStackFrame
../../../lib/src/screens/profiler/cpu_profile_model.dart 1372:9   _extension#0|generateStackFramesJson.processStackFrame
../../../lib/src/screens/profiler/cpu_profile_model.dart 1372:9   _extension#0|generateStackFramesJson.processStackFrame
../../../lib/src/screens/profiler/cpu_profile_model.dart 1372:9   _extension#0|generateStackFramesJson.processStackFrame
../../../lib/src/screens/profiler/cpu_profile_model.dart 1372:9   _extension#0|generateStackFramesJson.processStackFrame
../../../lib/src/screens/profiler/cpu_profile_model.dart 1372:9   _extension#0|generateStackFramesJson.processStackFrame
../../../lib/src/screens/profiler/cpu_profile_model.dart 1372:9   _extension#0|generateStackFramesJson.processStackFrame

[jakemac53]

I did send out a fix for the null check exception fwiw, https://github.com/flutter/devtools/pull/8940. But, I doubt its really going to help your situation, as this indicates a not useful response anyways. But, it might get you a different or better error.

[kenzieschmoll]

@jensjoha would you be able to try again with the latest DevTools code and see if you get a better error now that https://github.com/flutter/devtools/pull/8940 is fixed?

[jensjoha]

I still get null-errors in the console and the actual page is just standing there saying "Fetching CPU samples".

[jakemac53]

@jensjoha can you provide a CPU profile which fails to load?

[jensjoha]

I don't understand. The reproduction is in the first message?

[abitofevrything]

Here is a CPU profile that causes the error on the latest dev SDK release.

I don't know how to get a "raw" profile from the VM itself, so this is actually the JSON-RPC message that was sent to devtools containing the samples. Let me know if/how I need to get the raw samples.

samples.json