[Storage] java.lang.OutOfMemoryError after repeated use of putData()/putFile()

[CloudMountain]

[REQUIRED] Step 2: Describe your environment

• Android Studio version:

Android Studio Hedgehog | 2023.1.1 Patch 2 Build #AI-231.9392.1.2311.11330709, built on January 18, 2024 Runtime version: 17.0.7+0-17.0.7b1000.6-10550314 aarch64 VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o. macOS 14.4.1 GC: G1 Young Generation, G1 Old Generation Memory: 2048M Cores: 8 Metal Rendering is ON Registry: external.system.auto.import.disabled=true debugger.watches.in.variables=false ide.text.editor.with.preview.show.floating.toolbar=false

Non-Bundled Plugins: Dart (231.9411) io.flutter (77.2.1)

• Firebase Component: Storage

• Component version: 21.0.0 (I am using the flutter firebase_core plugin version 3.3.0 which uses firebase-bom version 33.1.0)

[REQUIRED] Step 3: Describe the problem

I originally posted about a problem I was having in the flutterfire repo here: https://github.com/firebase/flutterfire/issues/13385, and was told this was an problem with the native sdk and to raise a new issue here. Apologies in advance as I have limited experience in writing native code, and so can only really present the issue using Flutter terminology.

As can be seen in the original issue, the problem I am having is that I am trying to upload byte buffers of ~5MB on the order of a few hundred times during the course of a user session. Some users are experiencing OOM crashes of the following form: Fatal Exception: java.lang.OutOfMemoryError: Failed to allocate a 2576 byte allocation with 1020864 free bytes and 996KB until OOM, target footprint 268435456, growth limit 268435456; giving up on allocation because <1% of heap free after GC.

After doing some memory profiling with Android Studio, I was able to narrow down the issue to calls of the ref().putData()and/or ref().putFile()functions from the firebase_storage flutter plugin.

Steps to reproduce:

Make a button in a flutter app which simply creates a Uint8List of length 5,000,000 and then uploads it using either putData() or putFile(). Recording Java/Kotlin allocations whilst tapping that button shows the following byte buffers are allocated and then never deallocated:

Relevant Code:

Uint8List buffer = Uint8List(5000000);
Reference ref =  FirebaseStorage.instance.ref();
UploadTask? uploadTask = await ref.child(path/to/file.pcm).putData(buffer);

[lehcar09]

Hi @CloudMountain, thank you for reaching out. The Firebase Storage putBytes() or putData() (in Flutter) and putFile() are asynchronous methods. It returns a Taskobject that you can use to monitor the upload progress and handle its completion or failure.

Calling multiple putBytes methods in Firebase Storage without proper management can potentially lead to Out-of-Memory (OOM) exceptions, especially on devices with limited RAM.

Are you calling the putBytes() concurrently? If there are 50 concurrent calls with 5MB file size to upload, it's possible that the memory allocation used for this operation is more than the memory available on your device

I tried reproducing the issue and I was able to encounter the OOM exception on the 32nd concurrent call. Is there any chance you can share a bit more code snippet on how you did the “upload byte buffers of ~5MB on the order of a few hundred times during the course of a user session”? Thanks!

[google-oss-bot]

Hey @CloudMountain. We need more information to resolve this issue but there hasn't been an update in 5 weekdays. I'm marking the issue as stale and if there are no new updates in the next 5 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

[avishayw]

Hi @lehcar09

This crash reproduced for me as well. To your question:

Calling multiple putBytes methods in Firebase Storage without proper management can potentially lead to Out-of-Memory (OOM) exceptions, especially on devices with limited RAM.

Are you calling the putBytes() concurrently? If there are 50 concurrent calls with 5MB file size to upload, it's possible that the memory allocation used for this operation is more than the memory available on your device

I tried reproducing the issue and I was able to encounter the OOM exception on the 32nd concurrent call. Is there any chance you can share a bit more code snippet on how you did the “upload byte buffers of ~5MB on the order of a few hundred times during the course of a user session”? Thanks!

There are no concurrent upload trials in my case as far as my understanding goes (see upload method snippet below).

Stacktrace

java.lang.OutOfMemoryError: Failed to allocate a 3038016 byte allocation with 1288744 free bytes and 1258KB until OOM, target footprint 268435456, growth limit 268435456
    at io.flutter.plugin.common.StandardMessageCodec.readBytes(StandardMessageCodec.java:320)
    at io.flutter.plugin.common.StandardMessageCodec.readValueOfType(StandardMessageCodec.java:386)
    at io.flutter.plugins.firebase.storage.GeneratedAndroidFirebaseStorage$FirebaseStorageHostApiCodec.readValueOfType(GeneratedAndroidFirebaseStorage.java:692)
    at io.flutter.plugin.common.StandardMessageCodec.readValue(StandardMessageCodec.java:340)
    at io.flutter.plugin.common.StandardMessageCodec.readValueOfType(StandardMessageCodec.java:424)
    at io.flutter.plugins.firebase.storage.GeneratedAndroidFirebaseStorage$FirebaseStorageHostApiCodec.readValueOfType(GeneratedAndroidFirebaseStorage.java:692)
    at io.flutter.plugin.common.StandardMessageCodec.readValue(StandardMessageCodec.java:340)
    at io.flutter.plugin.common.StandardMessageCodec.decodeMessage(StandardMessageCodec.java:89)
    at io.flutter.plugin.common.BasicMessageChannel$IncomingMessageHandler.onMessage(BasicMessageChannel.java:262)
    at io.flutter.embedding.engine.dart.DartMessenger.invokeHandler(DartMessenger.java:292)
    at io.flutter.embedding.engine.dart.DartMessenger.lambda$dispatchMessageToQueue$0$io-flutter-embedding-engine-dart-DartMessenger(DartMessenger.java:319)
    at io.flutter.embedding.engine.dart.DartMessenger$$ExternalSyntheticLambda0.run
    at android.os.Handler.handleCallback(Handler.java:942)
    at android.os.Handler.dispatchMessage(Handler.java:99)
    at android.os.Looper.loopOnce(Looper.java:226)
    at android.os.Looper.loop(Looper.java:313)
    at android.app.ActivityThread.main(ActivityThread.java:8762)
    at java.lang.reflect.Method.invoke(Method.java)
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:604)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1067)

Environment

IDE:

Android Studio Flamingo | 2022.2.1 Patch 2 Build #AI-222.4459.24.2221.10121639, built on May 12, 2023 Runtime version: 17.0.6+0-b2043.56-9586694 amd64 VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o. Flutter 3.22.2 • channel stable • https://github.com/flutter/flutter.git Framework • revision 761747bfc5 (4 months ago) • 2024-06-05 22:15:13 +0200 Engine • revision edd8546116 Tools • Dart 3.4.3 • DevTools 2.34.3

Running Device Specifications (Gathered from Sentry) Device 1 - Physical

Kernel Version: 4.19.111-27127798 Name: Android Rooted: no Version: 13(TP1A.220624.014.M127GXXS8DXE4) Battery Level: 100% battery_temperature: 30.2 Boot Time: 2024-09-15T18:35:16.684Z(22 days before this event) Brand: samsung Charging: true connection_type: wifi Family: SM-M127G Free Memory: 1012.7 MiB Free Storage: 34.7 GiB Low Memory: false Manufacturer: samsung Memory Size: 3.6 GiB Model: SM-M127G (TP1A.220624.014) Model Id: TP1A.220624.014 Name: Galaxy M12 Online: true Simulator: false Storage Size: 51.3 GiB

Device 2 - Emulator

Kernel Version: 5.15.41-android13-8-00055-g4f5025129fe8-ab8949913 Name: Android Rooted: no Version: 13(sdk_gphone64_x86_64-userdebug 13 TE1A.220922.012 9302419 dev-keys) Battery Level: 100% battery_temperature: 25 Boot Time: 2024-10-01T20:39:18.681Z(17 hours before this event) Brand: google Charging: false connection_type: wifi External Free Storage: 509.9 MiB External Storage Size: 510.0 MiB Family: sdk_gphone64_x86_64 Free Storage: 3.8 GiB Manufacturer: Google Memory Size: 1.9 GiB Model: sdk_gphone64_x86_64 (TE1A.220922.012) Model Id: TE1A.220922.012 Online: true Simulator: true Storage Size: 5.8 GiB

PC specifications:

Processor 12th Gen Intel(R) Core(TM) i7-12700H 2.30 GHz Installed RAM 16.0 GB (15.6 GB usable) System type 64-bit operating system, x64-based processor

Windows specifications

Edition Windows 11 Home Version 23H2 OS build 22631.4169 Experience Windows Feature Experience Pack 1000.22700.1034.0

Packages:

firebase_core: ^3.6.0 firebase_storage: ^12.3.2 firebase_app_check: ^0.3.1+3

Steps to reproduce

1. Have some file to upload (mine is a few KBs in size)
2. Upload at a rate of once per second, but only if not uploading already (no concurrent uploads):

//...
_uploadTimer = Timer.periodic(const Duration(seconds: 1), (_) {
      _uploadLogsToStorage();
    });
//...
Future<void> _uploadLogsToStorage() async {
    if (_uploading) {
      return;
    }
    _uploading = true;
    bool firstLoop = true;
    // List logs in path
    final logsDir = await _getLogsDir();
    try {
      List<String> logs = path_utils.listFilesInDir(logsDir);
      if (logs.isNotEmpty) {
        final storageRef = MsGlobals().storage.ref();
        for (String log in logs) {
          // Verify again that not uploading already due to the loop
          if (!firstLoop ) {
            break;
          }
          firstLoop  = false;
          final filename = path.basename(log);
          Reference logRef =
              storageRef.child('logs/${MsGlobals().userUuid}/$filename');
          final byteData = await File(log).readAsBytes();
          logRef
              .putData(byteData, SettableMetadata(contentType: 'text/plain'))
              .snapshotEvents
              .listen((taskSnapshot) async {
            switch (taskSnapshot.state) {
              case TaskState.running:
                break;
              case TaskState.paused:
                break;
              case TaskState.success:
                // Check if log is older than 2 hours
                final String dateString = filename.split('.')[0];
                final DateTime logDateTime = parseFormattedDateTime(dateString);
                if (DateTime.now()
                    .toUtc()
                    .isAfter(logDateTime.add(const Duration(hours: 2)))) {
                  await path_utils.deleteFile(path: log);
                }
                _uploading = false;
                break;
              case TaskState.canceled:
                _uploading = false;
                break;
              case TaskState.error:
                _uploading = false;
                break;
            }
          });
        }
      } else {
        _uploading = false;
      }
    } catch (e) {
      debugPrint(e.toString());
      _uploading = false;
    }
  }
//...

Steps to reproduce FASTER

Just increase the size of the file to upload:

//...
_uploadTimer = Timer.periodic(const Duration(seconds: 1), (_) {
      _uploadLogsToStorage();
    });
//...
Future<void> _uploadLogsToStorage() async {
    if (_uploading) {
      return;
    }
    _uploading = true;
    bool firstLoop = true;
    // List logs in path
    final logsDir = await _getLogsDir();
    try {
      List<String> logs = path_utils.listFilesInDir(logsDir);
      if (logs.isNotEmpty) {
        final storageRef = MsGlobals().storage.ref();
        for (String log in logs) {
          // Verify again that not uploading already due to the loop
          if (!firstLoop ) {
            break;
          }
          firstLoop  = false;
          final filename = path.basename(log);
          Reference logRef =
              storageRef.child('logs/${MsGlobals().userUuid}/$filename');
          Uint8List byteData = await File(log).readAsBytes();
		  if (const String.fromEnvironment('MEMORY_TEST',
                  defaultValue: 'false') ==
              'true') {
            /// Because there is OutOfMemory crash after about an hour or two,
            /// I want to make the app crash quicker by multiplying the byte
            /// data, and uploading to a separate folder in bucket.
            logRef = storageRef.child(
                'logs/${MsGlobals().userUuid}-MEMORY_TEST-$_sessionTime/$filename');
            const byteDataMultiplier = 1000;
            final repeatedBytes =
                List.generate(byteDataMultiplier, (_) => byteData)
                    .expand((e) => e)
                    .toList();
            byteData = Uint8List.fromList(repeatedBytes);
          }
          logRef
              .putData(byteData, SettableMetadata(contentType: 'text/plain'))
              .snapshotEvents
              .listen((taskSnapshot) async {
            switch (taskSnapshot.state) {
              case TaskState.running:
                break;
              case TaskState.paused:
                break;
              case TaskState.success:
                // Check if log is older than 2 hours
                final String dateString = filename.split('.')[0];
                final DateTime logDateTime = parseFormattedDateTime(dateString);
                if (DateTime.now()
                    .toUtc()
                    .isAfter(logDateTime.add(const Duration(hours: 2)))) {
                  await path_utils.deleteFile(path: log);
                }
                _uploading = false;
                break;
              case TaskState.canceled:
                _uploading = false;
                break;
              case TaskState.error:
                _uploading = false;
                break;
            }
          });
        }
      } else {
        _uploading = false;
      }
    } catch (e) {
      debugPrint(e.toString());
      _uploading = false;
    }
  }
//...

Additional information

From MemoryView, it can be seen that RSS is constantly increasing, while Dart/Flutter VM & Native stay pretty consistent and managed to be cleaned by garabage collector:

Analyzing Heap Dump with "Eclipse Memory Analyzer" found TaskStateChannelStreamHandler as leak suspect

I used the profiler and did heap dump (classes are sorted by shallow size, descending):

Then saved the file and loaded into Eclipse memory analyzer to find leak suspects:

---

Help please. I'm out of ideas on how to debug this.

Thanks in advance, let me know if additional details are needed.

[google-oss-bot]

Hey @CloudMountain. We need more information to resolve this issue but there hasn't been an update in 5 weekdays. I'm marking the issue as stale and if there are no new updates in the next 5 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

[google-oss-bot]

Since there haven't been any recent updates here, I am going to close this issue.

@CloudMountain if you're still experiencing this problem and want to continue the discussion just leave a comment here and we are happy to re-open this.