JuliaLang
Heap snapshot does not seem to work anymore on 1.11 (in chrome profile loader)
julia> using Profile
julia> Profile.take_heap_snapshot("1.11.heapsnapshot")
"1.11.heapsnapshot"
Trying to load this in Chrome leads to
An error occurred when a call to method 'buildSnapshot' was requested
TypeError: Cannot read properties of undefined (reading '1')
at F.name (devtools://devtools/bundled/devtools-frontend/front_end/entrypoints/heap_snapshot_worker/heap_snapshot_worker.js:1:8216)
at F.rawName (devtools://devtools/bundled/devtools-frontend/front_end/entrypoints/heap_snapshot_worker/heap_snapshot_worker.js:1:38825)
at F.name (devtools://devtools/bundled/devtools-frontend/front_end/entrypoints/heap_snapshot_worker/heap_snapshot_worker.js:1:39038)
at F.isDocumentDOMTreesRoot (devtools://devtools/bundled/devtools-frontend/front_end/entrypoints/heap_snapshot_worker/heap_snapshot_worker.js:1:40431)
at b.isUserRoot (devtools://devtools/bundled/devtools-frontend/front_end/entrypoints/heap_snapshot_worker/heap_snapshot_worker.js:1:35720)
at b.calculateDistances (devtools://devtools/bundled/devtools-frontend/front_end/entrypoints/heap_snapshot_worker/heap_snapshot_worker.js:1:18912)
at b.calculateDistances (devtools://devtools/bundled/devtools-frontend/front_end/entrypoints/heap_snapshot_worker/heap_snapshot_worker.js:1:35160)
at b.initialize (devtools://devtools/bundled/devtools-frontend/front_end/entrypoints/heap_snapshot_worker/heap_snapshot_worker.js:1:14141)
at new b (devtools://devtools/bundled/devtools-frontend/front_end/entrypoints/heap_snapshot_worker/heap_snapshot_worker.js:1:33684)
at A.buildSnapshot (devtools://devtools/bundled/devtools-frontend/front_end/entrypoints/heap_snapshot_worker/heap_snapshot_worker.js:1:43578)
On 1.10 the same procedure works fine.
Yeah these aren't backported to 1.10 yet https://github.com/JuliaLang/julia/pull/53833 https://github.com/JuliaLang/julia/pull/53984
AFAIK the VS Code heap snapshot viewer doesn't allow diffing two heap snapshots, unlike the Chromium Dev Tools? So I guess there's currently no way to diff heap snapshots :face_with_diagonal_mouth:.
Any help figuring out why would be appreciated. They seem to expect different metadata. See the PRs that have changed it
Trying to get to the bottom of this.
Supposedly this is a valid V8 structure
{
"snapshot": {
"title": "Heap Snapshot",
"uid": 1,
"meta": {
"node_fields": ["type", "name", "id", "self_size", "edge_count"],
"node_types": [
["object", "closure", "regexp", "string", "number", "native", "synthetic", "concatenated string", "sliced string"],
"string",
"number",
"number",
"number"
],
"edge_fields": ["type", "name_or_index", "to_node"],
"edge_types": [
["context", "element", "property", "internal", "hidden", "shortcut", "weak"],
"string_or_number",
"node"
]
},
"node_count": 1000,
"edge_count": 5000
},
"strings": [
"",
"(root)",
"Object",
"Array",
"foo",
"bar",
/* ... other strings ... */
],
"nodes": [
0, 1, 2, 3, 4, /* ... node data ... */
],
"edges": [
0, 1, 2, /* ... edge data ... */
],
/* ... other sections ... */
}
A julia example I just collected
{
"snapshot": {
"meta": {
"node_fields": ["type", "name", "id", "self_size", "edge_count", "trace_node_id", "detachedness"],
"node_types": [
["synthetic", "jl_task_t", "jl_module_t", "jl_array_t", "object", "jl_datatype_t", "String", "jl_svec_t", "jl_sym_t"],
"string",
"number",
"number",
"number",
"number",
"number"
],
"edge_fields": ["type", "name_or_index", "to_node"],
"edge_types": [
["internal", "element", "hidden", "property", "binding"],
"string_or_number",
"from_node"
],
"trace_function_info_fields": ["function_id", "name", "script_name", "script_id", "line", "column"],
"trace_node_fields": ["id", "function_info_index", "count", "size", "children"],
"sample_fields": ["timestamp_us", "last_assigned_id"],
"location_fields": ["object_index", "script_id", "line", "column"]
},
"node_count": 1759206,
"edge_count": 5447107,
"trace_function_count": 0
},
"trace_function_infos": [],
"trace_tree": [],
"samples": [],
"locations": [],
"nodes": [
0, 0, 0, 0, 4, 0, 0,
0, 1, 1, 0, 6, 0, 0,
...
"edges":[
0,1,7,
0,2,14,
...
"strings":[
"",
"GC roots",
...
}
https://github.com/JuliaLang/julia/pull/53833 which fixed opening in VSCode's viewer, and likely broke chrome devtools made these changes
References: https://learn.microsoft.com/en-us/microsoft-edge/devtools-guide-chromium/memory-problems/heap-snapshot-schema#schema-of-the-heapsnapshot-data https://v8docs.nodesource.com/node-0.8/d8/deb/classv8_1_1_heap_snapshot.html#a159532596beaf59473bf3189703c39a1
The problem appears to be the inclusion of
"sample_fields":["timestamp_us","last_assigned_id"],
"location_fields":["object_index","script_id","line","column"]
without it vscode cannot load the heapsnapshot, but chrome devtools can. with it chrome devtools cannot, but vscode can.
Dev tools parsing for those fields (which fails if they are there, but the line numbers don't correspond to here) https://github.com/ChromeDevTools/devtools-frontend/blob/75970e0fc2a9be009fa4d946cedc0e0018c16a88/front_end/entrypoints/heap_snapshot_worker/HeapSnapshotLoader.ts#L241-L251
vscode https://github.com/microsoft/vscode-v8-heap-tools/blob/c5b34396392397925ecbb4ecb904a27a2754f2c1/v8-heap-parser/src/decoder.rs#L48-L50
It'd be helpful to find the exact files & lines these are coming from in devtools, but I guess these js files were minified
TypeError: Cannot read properties of undefined (reading 'length')
at new O (devtools://devtools/bundled/devtools-frontend/front_end/entrypoints/heap_snapshot_worker/heap_snapshot_worker.js:1:13338)
at new A (devtools://devtools/bundled/devtools-frontend/front_end/entrypoints/heap_snapshot_worker/heap_snapshot_worker.js:1:39902)
at R.buildSnapshot (devtools://devtools/bundled/devtools-frontend/front_end/entrypoints/heap_snapshot_worker/heap_snapshot_worker.js:1:50296)
at HeapSnapshotWorkerDispatcher.dispatchMessage (devtools://devtools/bundled/devtools-frontend/front_end/entrypoints/heap_snapshot_worker/heap_snapshot_worker.js:1:54089)
The error above is to this (in an unminified version of heap_snapshot_worker,js) https://gist.github.com/IanButterworth/9fb742180ee6e682e04d34819fd9d500#file-heap_snapshot_worker-js-L693
So somehow the inclusion of
"sample_fields":["timestamp_us","last_assigned_id"],
"location_fields":["object_index","script_id","line","column"]
is preventing strings from being populated.
It's this raw line https://github.com/ChromeDevTools/devtools-frontend/blob/2709fa744749f851721829cc218698729f74bd24/front_end/entrypoints/heap_snapshot_worker/HeapSnapshot.ts#L828
https://learn.microsoft.com/en-us/microsoft-edge/devtools-guide-chromium/memory-problems/heap-snapshot-schema not sure if this is helpful but I always drop it.
That's linked above. I think this is a bug in what each expects to be there.
FWIW it feels like the line in your gist corresponds to this snippet from the DevTools TypeScript source:
https://github.com/ChromeDevTools/devtools-frontend/blob/2709fa744749f851721829cc218698729f74bd24/front_end/entrypoints/heap_snapshot_worker/HeapSnapshot.ts#L826-L830
I see that a vscode-generated heapsnapshot opens in chrome dev tools, and visa-versa. It's just julia ones that don't open in dev tools.
Some summaries
Julia
julia> json_string = read("/Users/ian/Documents/GitHub/julia/96449_180407234248750.heapsnapshot", String);
julia> js = JSON3.read(json_string)
JSON3.Object{Base.CodeUnits{UInt8, String}, Vector{UInt64}} with 8 entries:
:snapshot => {…
:trace_function_infos => Union{}[]
:trace_tree => Union{}[]
:samples => Union{}[]
:locations => Union{}[]
:nodes => [0, 0, 0, 0, 4, 0, 0, 0, 1, 1 … 0, 0, 0, 4, 1747986, 4660449792, 16, 1, 0, 0]
:edges => Union{Float64, Int64}[0, 1, 7, 0, 2, 14, 0, 4, 21, 0 … 43428, 3, 29907, 43428, 3, 539187, 43428, 3, 1528671, 10617334]
:strings => ["", "GC roots", "GC finalizer list roots", "Task", "root task", "current task", "Main", "main_module", "Array{Any, 1}", "…
julia> js.snapshot
JSON3.Object{Base.CodeUnits{UInt8, String}, SubArray{UInt64, 1, Vector{UInt64}, Tuple{UnitRange{Int64}}, true}} with 4 entries:
:meta => {…
:node_count => 1747684
:edge_count => 5408593
:trace_function_count => 0
julia> js.snapshot.meta
JSON3.Object{Base.CodeUnits{UInt8, String}, SubArray{UInt64, 1, Vector{UInt64}, Tuple{UnitRange{Int64}}, true}} with 8 entries:
:node_fields => ["type", "name", "id", "self_size", "edge_count", "trace_node_id", "detachedness"]
:node_types => Any[["synthetic", "jl_task_t", "jl_module_t", "jl_array_t", "object", "jl_datatype_t", "String", "jl_svec_t", "jl_sy…
:edge_fields => ["type", "name_or_index", "to_node"]
:edge_types => Any[["internal", "element", "hidden", "property", "binding"], "string_or_number", "from_node"]
:trace_function_info_fields => ["function_id", "name", "script_name", "script_id", "line", "column"]
:trace_node_fields => ["id", "function_info_index", "count", "size", "children"]
:sample_fields => ["timestamp_us", "last_assigned_id"]
:location_fields => ["object_index", "script_id", "line", "column"]
julia> for (k,v) in js
@show k,length(v)
end
(k, length(v)) = (:snapshot, 4)
(k, length(v)) = (:trace_function_infos, 0)
(k, length(v)) = (:trace_tree, 0)
(k, length(v)) = (:samples, 0)
(k, length(v)) = (:locations, 0)
(k, length(v)) = (:nodes, 12233788)
(k, length(v)) = (:edges, 16225779)
(k, length(v)) = (:strings, 1748010)
chrome
julia> json_string = read("/Users/ian/Documents/GitHub/julia/chrome_Heap-20240926T080701.heapsnapshot", String);
julia> js = JSON3.read(json_string)
JSON3.Object{Base.CodeUnits{UInt8, String}, Vector{UInt64}} with 8 entries:
:snapshot => {…
:nodes => [9, 1, 1, 0, 14, 0, 0, 9, 2, 3 … 87, 0, 0, 9, 73578, 252940, 0, 0, 0, 0]
:edges => [1, 1, 7, 5, 3695, 23807, 5, 3696, 24038, 5 … 5824616, 1, 85, 5667606, 1, 86, 5296452, 1, 87, 5583277]
:trace_function_infos => Union{}[]
:trace_tree => Union{}[]
:samples => Union{}[]
:locations => [522795, 513, 55, 461709, 522760, 513, 55, 461709, 525546, 513 … 55, 894511, 4174961, 513, 55, 1402055, 4174968, 513, 55, 1402…
:strings => ["<dummy>", "", "(GC roots)", "(Bootstrapper)", "(Builtins)", "(Client heap)", "(Code flusher)", "(Compilation cache)", "(Debugg…
julia> js.snapshot
JSON3.Object{Base.CodeUnits{UInt8, String}, SubArray{UInt64, 1, Vector{UInt64}, Tuple{UnitRange{Int64}}, true}} with 4 entries:
:meta => {…
:node_count => 841614
:edge_count => 3659995
:trace_function_count => 0
julia> js.snapshot.meta
JSON3.Object{Base.CodeUnits{UInt8, String}, SubArray{UInt64, 1, Vector{UInt64}, Tuple{UnitRange{Int64}}, true}} with 8 entries:
:node_fields => ["type", "name", "id", "self_size", "edge_count", "trace_node_id", "detachedness"]
:node_types => Any[["hidden", "array", "string", "object", "code", "closure", "regexp", "number", "native", "synthetic", "concatenated st…
:edge_fields => ["type", "name_or_index", "to_node"]
:edge_types => Any[["context", "element", "property", "internal", "hidden", "shortcut", "weak"], "string_or_number", "node"]
:trace_function_info_fields => ["function_id", "name", "script_name", "script_id", "line", "column"]
:trace_node_fields => ["id", "function_info_index", "count", "size", "children"]
:sample_fields => ["timestamp_us", "last_assigned_id"]
:location_fields => ["object_index", "script_id", "line", "column"]
julia> for (k,v) in js
@show k,length(v)
end
(k, length(v)) = (:snapshot, 4)
(k, length(v)) = (:nodes, 5891298)
(k, length(v)) = (:edges, 10979985)
(k, length(v)) = (:trace_function_infos, 0)
(k, length(v)) = (:trace_tree, 0)
(k, length(v)) = (:samples, 0)
(k, length(v)) = (:locations, 241380)
(k, length(v)) = (:strings, 136268)
vscode
julia> json_string = read("/Users/ian/Documents/GitHub/julia/vscode-profile-2024-09-25-21-49-23.heapsnapshot", String);
julia> js = JSON3.read(json_string)
JSON3.Object{Base.CodeUnits{UInt8, String}, Vector{UInt64}} with 8 entries:
:snapshot => {…
:nodes => [9, 1, 1, 0, 8, 0, 0, 9, 2, 3 … 0, 0, 0, 8, 15102, 316, 14, 0, 0, 0]
:edges => [1, 1, 7, 5, 3432, 22477, 5, 3433, 22540, 1 … 427007, 1, 109, 427014, 1, 110, 427021, 1, 111, 427028]
:trace_function_infos => Union{}[]
:trace_tree => Union{}[]
:samples => Union{}[]
:locations => [134162, 7, 261, 28, 134204, 7, 273, 36, 134225, 7 … 330, 24, 418257, 7, 346, 14, 418264, 7, 370, 13]
:strings => ["<dummy>", "", "(GC roots)", "(Bootstrapper)", "(Builtins)", "(Client heap)", "(Code flusher)", "(Compilation cache)", "(Debugg…
julia> js.snapshot
JSON3.Object{Base.CodeUnits{UInt8, String}, SubArray{UInt64, 1, Vector{UInt64}, Tuple{UnitRange{Int64}}, true}} with 4 entries:
:meta => {…
:node_count => 61005
:edge_count => 257826
:trace_function_count => 0
julia> js.snapshot.meta
JSON3.Object{Base.CodeUnits{UInt8, String}, SubArray{UInt64, 1, Vector{UInt64}, Tuple{UnitRange{Int64}}, true}} with 8 entries:
:node_fields => ["type", "name", "id", "self_size", "edge_count", "trace_node_id", "detachedness"]
:node_types => Any[["hidden", "array", "string", "object", "code", "closure", "regexp", "number", "native", "synthetic", "concatenated st…
:edge_fields => ["type", "name_or_index", "to_node"]
:edge_types => Any[["context", "element", "property", "internal", "hidden", "shortcut", "weak"], "string_or_number", "node"]
:trace_function_info_fields => ["function_id", "name", "script_name", "script_id", "line", "column"]
:trace_node_fields => ["id", "function_info_index", "count", "size", "children"]
:sample_fields => ["timestamp_us", "last_assigned_id"]
:location_fields => ["object_index", "script_id", "line", "column"]
julia> for (k,v) in js
@show k,length(v)
end
(k, length(v)) = (:snapshot, 4)
(k, length(v)) = (:nodes, 427035)
(k, length(v)) = (:edges, 773478)
(k, length(v)) = (:trace_function_infos, 0)
(k, length(v)) = (:trace_tree, 0)
(k, length(v)) = (:samples, 0)
(k, length(v)) = (:locations, 22168)
(k, length(v)) = (:strings, 29769)
We don't populate "locations".
According to https://learn.microsoft.com/en-us/microsoft-edge/devtools-guide-chromium/memory-problems/heap-snapshot-schema
"locations" : Contains information about the script location of nodes. To parse this data, use snapshot.meta.location_fields with the nodes array. | Array
But if you take the number of node entries in the vscode one as an example: 427035 divide by the length of node_fields (7) gives 61005 nodes.
Multiply that by the length of location_fields (4) gives 244020 which is much bigger than the actual 22168.
If we can figure out the right size to zero pad to, it'd be worth trying I think.
Found the issue! The order of the fields in the json object is critical for dev tools! I guess their JSON parser doesn't just load the whole json object in then process it, but does some processing on the fly..