java.io.FileNotFoundException: open failed: ENOSPC (No space left on device)

[mrblrrd]

[READ] Step 1: Are you in the right place?

Issues filed here should be about bugs in the code in this repository. If you have a general question, need help debugging, or fall into some other category use one of these other channels:

• For general technical questions, post a question on StackOverflow with the firebase tag.
• For general Firebase discussion, use the firebase-talk google group.
• For help troubleshooting your application that does not fall under one of the above categories, reach out to the personalized Firebase support channel.

[REQUIRED] Step 2: Describe your environment

• Android Studio version: Iguana | 2023.2.1 Patch 1
• Firebase Component: Crashlytics
• Component version: 18.6.2

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

Fatal Exception: java.io.FileNotFoundException: /data/user/0/<package>/files/datastore/firebase_session_Y29tLnZrb250YWt0ZS5hbmRyb2lk_data.preferences_pb.tmp: open failed: ENOSPC (No space left on device)
       at libcore.io.IoBridge.open(IoBridge.java:574)
       at java.io.FileOutputStream.<init>(FileOutputStream.java:236)
       at java.io.FileOutputStream.<init>(FileOutputStream.java:186)
       at androidx.datastore.core.SingleProcessDataStore.writeData$datastore_core(SingleProcessDataStore.kt:425)
       at androidx.datastore.core.SingleProcessDataStore.transformAndWrite(SingleProcessDataStore.kt:410)
       at androidx.datastore.core.SingleProcessDataStore.access$getFile(SingleProcessDataStore.kt:76)
       at androidx.datastore.core.SingleProcessDataStore.access$transformAndWrite(SingleProcessDataStore.kt:76)
       at androidx.datastore.core.SingleProcessDataStore$transformAndWrite$1.invokeSuspend(SingleProcessDataStore.kt:13)
       at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
       at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
       at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)
       at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:103)
       at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)

Relevant Code:

No specific code.

[argzdev]

Hi @mrblrrd, thanks for reaching out. It looks like the issue is due to the device not having enough memory left in the internal storage. Does this happen even with fair amount of internal memory within the device? Also is this occurring on multiple devices?

[mrblrrd]

Yes, this is due to lack of disk space. I believe this should not lead to a crash of the application. We see a noticeable increase in such crashes in our prod application on Crashlytics Dashboard after updating to version 18.6.2. Different devices: Redmi 9T, Honor X7a, Redmi Note 10S, Galaxy A12, etc.

[argzdev]

Thanks for the extra details, @mrblrrd. What was the previous version of Crashlytics did you use prior to encounter this issue? Any chance you could share the full stacktrace? I'm having a hard time pin pointing in which part of Crashlytics is triggering the issue.

[mrblrrd]

The previous version was 18.3.3. The stack trace above is full. I can give you another recent example:

Fatal Exception: java.io.FileNotFoundException
/data/user/0/<package>/files/datastore/firebase_session_Y29tLnZrb250YWt0ZS5hbmRyb2lk_data.preferences_pb.tmp: open failed: ENOSPC (No space left on device)

Fatal Exception: java.io.FileNotFoundException: /data/user/0/<package>/files/datastore/firebase_session_Y29tLnZrb250YWt0ZS5hbmRyb2lk_data.preferences_pb.tmp: open failed: ENOSPC (No space left on device)
       at libcore.io.IoBridge.open(IoBridge.java:492)
       at java.io.FileOutputStream.<init>(FileOutputStream.java:236)
       at java.io.FileOutputStream.<init>(FileOutputStream.java:186)
       at androidx.datastore.core.SingleProcessDataStore.writeData$datastore_core(SingleProcessDataStore.kt:425)
       at androidx.datastore.core.SingleProcessDataStore.transformAndWrite(SingleProcessDataStore.kt:410)
       at androidx.datastore.core.SingleProcessDataStore.access$getFile(SingleProcessDataStore.kt:76)
       at androidx.datastore.core.SingleProcessDataStore.access$transformAndWrite(SingleProcessDataStore.kt:76)
       at androidx.datastore.core.SingleProcessDataStore$transformAndWrite$1.invokeSuspend(SingleProcessDataStore.kt:13)
       at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
       at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
       at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)
       at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:103)
       at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)

Caused by android.system.ErrnoException
open failed: ENOSPC (No space left on device)

Caused by android.system.ErrnoException: open failed: ENOSPC (No space left on device)
       at libcore.io.Linux.open(Linux.java)
       at libcore.io.ForwardingOs.open(ForwardingOs.java:166)
       at libcore.io.BlockGuardOs.open(BlockGuardOs.java:254)
       at libcore.io.ForwardingOs.open(ForwardingOs.java:166)
       at android.app.ActivityThread$AndroidOs.open(ActivityThread.java:7980)
       at libcore.io.IoBridge.open(IoBridge.java:478)
       at java.io.FileOutputStream.<init>(FileOutputStream.java:236)
       at java.io.FileOutputStream.<init>(FileOutputStream.java:186)
       at androidx.datastore.core.SingleProcessDataStore.writeData$datastore_core(SingleProcessDataStore.kt:425)
       at androidx.datastore.core.SingleProcessDataStore.transformAndWrite(SingleProcessDataStore.kt:410)
       at androidx.datastore.core.SingleProcessDataStore.access$getFile(SingleProcessDataStore.kt:76)
       at androidx.datastore.core.SingleProcessDataStore.access$transformAndWrite(SingleProcessDataStore.kt:76)
       at androidx.datastore.core.SingleProcessDataStore$transformAndWrite$1.invokeSuspend(SingleProcessDataStore.kt:13)
       at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
       at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
       at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)
       at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:103)
       at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)

Most likely the error occurs when accessing dataStore without catching FileNotFoundException.

[argzdev]

Hey @mrblrrd, thanks for pointing me in the right direction. I was able to kind of simulate the behavior using a datastore separately without Firebase and then testing it on a emulator with full memory. It's technically the same with the issue here. I might have a solution in mind. I'll reach out to our engineers and see if we can add this fix.

Notes: https://github.com/firebase/firebase-android-sdk/blob/30b10ab0448a9729bfcca55480f92a00b3cedcac/firebase-sessions/src/main/kotlin/com/google/firebase/sessions/SessionDatastore.kt#L103-L112

Upon testing, initialization of datastore does not cause any crash. I'm suspecting because the created File() from datastore is being stored as cache in the RAM and not in the internal storage, as a result, even if the internal storage is full, the app would not crash.

On the other hand, when calling the edit from the datastore, this will now write in the internal storage. https://github.com/firebase/firebase-android-sdk/blob/30b10ab0448a9729bfcca55480f92a00b3cedcac/firebase-sessions/src/main/kotlin/com/google/firebase/sessions/SessionDatastore.kt#L87-L89

which will cause the java.io.FileNotFoundException: open failed: ENOSPC failure.

Possible solution would to be add a conditional checker before writing in the datastore, or actually I think a simple try catch would also work.

fun hasEnoughStorage(context: Context): Boolean {
        val stat = context.applicationContext.filesDir.usableSpace
        val threshold =  1L * 1024 * 1024
        Log.d(TAG, "hasEnoughStorage: $stat > $threshold")
        return stat > threshold
}

[mrober]

This was fixed, please update to the latest version.