Can't load profile due to OOM error

[tehmatt]

Similar to https://github.com/firefox-devtools/profiler/issues/2534, but occurs when trying to load a large profile (anything over 1GB can reliably trigger this for me).

https://github.com/firefox-devtools/profiler/issues/2534#issuecomment-1331395361 shows a good example of this.

┆Issue is synchronized with this Jira Task

[tehmatt]

This profile size is pretty easy to hit with FunctionTrace (yeah, we should add options to prune those sizes...).

My quick look makes me think the Profiler is decoding to an ArrayBuffer too soon and that it might be possible to stream more things to avoid multiple copies of the data in memory before the json.parse() step. I don't write much js, but it also seems like it's maybe possible to use Response(blob).json() as a hacky form of streaming json parsing. I tried playing around a bit but didn't get anything working before running into Gitpod issues and giving up for now.

[julienw]

Are you loading as a file or as an url?

Your idea of using Response(blob).json() is an interesting idea! It's worth trying.

I was also wondering about using the new streams capabilities of the web platform. To achieve this we'll probably need to look at our ungzip operation, which uses a quite old asm-build of zlib, where we currently use only the gzcompress and gzdecompress entry points, and where we could use a more streaming operation. It could make sense to switch to pako if that works in browsers (pako is already transitively installed in the project due to jszip) or fflate that promises good performance and supports zip too (so it could replace jszip as well).

Note that Response can take a Stream as an input, so this would work with your idea as well.

Some food for thought!

[tehmatt]

With a test 1.4GB profile it'll fail when loaded from a file or url. I believe with both it fails sending the ArrayBuffer through the TextDecoder.

[julienw]

Can you give us a good way to generate such a profile with functiontrace? Especially do you have the python script or app or scenario that, when you use functiontrace with it, would produce such a file? We don't know functiontrace much ourselves :-)

[tehmatt]

I have a modified tensorflow tutorial that's easy to tweak to generate arbitrarily large profiles:

big_profile.py

#!/usr/bin/env python3                                                                                                                                                                                                                                         
# https://github.com/tensorflow/docs/blob/master/site/en/tutorials/quickstart/beginner.ipynb
import tensorflow as tf

def main():
    mnist = tf.keras.datasets.mnist
    (x_train, y_train), (x_test, y_test) = mnist.load_data()
    x_train, x_test = x_train / 255.0, x_test / 255.0

    model = tf.keras.models.Sequential([
        tf.keras.layers.Flatten(input_shape=(28, 28)),
        tf.keras.layers.Dense(128, activation='relu'),
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Dense(10, activation='softmax')
    ])

    model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

    model.fit(x_train, y_train, epochs=1)
    model.fit(x_train, y_train, epochs=1)  # <---- THESE GENERATE TONS OF DATA
    model.fit(x_train, y_train, epochs=1)
    model.evaluate(x_test, y_test, verbose=2)

if __name__ == "__main__":
    main()

Run with functiontrace --trace-memory big_profile.py.

You can pretty much keep adding model.fit calls until you get a large enough profile. For me, three fit calls generates ~1.3GB raw, 200MB compressed, which triggers the issue on my machine.

[julienw]

Worth mentioning is this new API: https://developer.mozilla.org/en-US/docs/Web/API/TextDecoderStream It's not in Firefox ESR yet, so if we use it it would still be good to keep the old way of working as well with some feature detection code.

This could help in having only a small part of the data being duplicated at one moment. Only the JSON.parse needs to have the full data at this moment.

BTW I had a look at how Response's json call is implemented: this fetches the full text before parsing json on it. See https://searchfox.org/mozilla-central/rev/6aa9eac63c2025306b184fe593e8a1d5c4635938/dom/base/BodyConsumer.cpp#733-746 It's also not clear to me if we consume the text in a streaming fashion.

[Delengowski]

@julienw

Hi, I am having the same issue. I recently built profiler (within last month) and am hosting it on an air gapped system. Having this SAME exact issue with a ~270MB profile generated by functiontrace. Are you saying if I upgrade the firefox we are using to 105 then profile will be able to load without memory error?

[julienw]

A newish project I just found while working on something else: https://www.npmjs.com/package/json-stream-es

Also https://github.com/worker-tools/json-stream