sentry-cocoa icon indicating copy to clipboard operation
sentry-cocoa copied to clipboard
verified publisher icon getsentry

Lots of OutOfMemory

Open markuswahl-2 opened this issue 3 years ago • 25 comments

Platform

iOS

Installed

CocoaPods

Version

7.11.0

Steps to Reproduce

We do not know. These are the only breadcrumbs from sentry.io:

CATEGORY DESCRIPTION LEVEL TIME
sentry.event A value set to the context or sdk is not serializable. Dropping context and sdk. Error 21:07:28
exception OutOfMemory: The OS most likely terminated your app because it overused RAM. Error 21:07:28

We are seeing this with sentry.cocoa 7.9.0, 7.10.2 and 7.11.0. One difference we see when we upgraded from 7.9.0 to 7.11.0 is that we no longer get the information about APP Start Time. With 7.9.0 we see that all such OOM events occur in the same second that the app was started. After having upgraded to 7.10.2 or 7.11.0 we no longer get that piece of information.

Expected Result

No crash

Actual Result

Tags

environment production

event.environment native

event.origin ios

handled no

level fatal

mechanism out_of_memory

markuswahl-2 avatar May 25 '22 07:05 markuswahl-2

I don't fully get your problem, @markuswahl-2. Can you please answer the following questions to clarify this issue:

  1. As you linked to the dart repo, I assume your app is built with Flutter and uses the Flutter SDK?
  2. Your expected result states no crash. Is the Cocoa SDK causing crashes in your app?
  3. Your title states lots of OutOfMemroy. So is the problem now, that your app crashes, or do you get a lot of OutOfMemory errors?
  4. What exactly do you mean by that we no longer get the information about APP Start Time?
  5. These are the only breadcrumbs from sentry.io: where did you get these breadcrumbs from? The OutOfMemory event?

In 7.11.0, we released a fix for detecting ANRs as false OOMs #1695. That's why you see a change in behavior.

philipphofmann avatar May 27 '22 07:05 philipphofmann

Hi!

  1. Yes
  2. Well, we do not know what causes it. We logged this bug report since we would be glad if we could get some more information out of sentry about these crashes. And since you have said that you have fixed the false OutOfMemory-reports in 7.11.0, we wanted to let you know that that is perhaps not the case. Ie that the problem is still there. I can see in https://github.com/getsentry/sentry-dart/issues/814 that we are not the only ones that still see the OutOfMemory-reports even after updating to 7.11.0.
  3. We do not know. We have not observed the issue ourselves, only through the error reports from sentry.io. Do you know of a way for us to spot the difference? We get the tag fatal. Given that there was a problem (at least prior to 7.11.0) with false OutOfMemory-reports, can we infer from the tag fatal that these reports are for real, ie about real crashes and not just somehow false OutOfMemory-reports?
  4. The sentry.io section called APP (with sub headings App Build, ID, Build ID, Build Name and Start Time) are present for apps that use versions prior to 7.11.0 but that section is not present for apps that use version 7.11.0.
  5. Yes, on sentry.io. I gather that sentry.io renders this from the json-file attached to each report. I attach one file of each here (with-context-information-prior-to-7.11.0-redacted.json is the one where we can see the start time, without-context-information-using-7.11.0-redacted.json is the one were that information is missing). I am guessing that the context section in with-context-information-prior-to-7.11.0-redacted.json is the context that the error message "sentry.event - A value set to the context or sdk is not serializable. Dropping context and sdk." is all about...

We only see these OutOfMemory-reports from iOS devices.

Best regards, Markus

markuswahl-2 avatar May 30 '22 10:05 markuswahl-2

with-context-information-prior-to-7.11.0-redacted.json

{
  "event_id": "REDACTED",
  "project": REDACTED,
  "release": "REDACTED",
  "dist": "580",
  "platform": "cocoa",
  "message": "",
  "datetime": "2022-05-23T08:12:56.779649Z",
  "tags": [
    [
      "app.device",
      "REDACTED"
    ],
    [
      "device",
      "iPhone12,8"
    ],
    [
      "device.family",
      "iOS"
    ],
    [
      "environment",
      "production"
    ],
    [
      "event.environment",
      "native"
    ],
    [
      "event.origin",
      "ios"
    ],
    [
      "handled",
      "no"
    ],
    [
      "level",
      "fatal"
    ],
    [
      "mechanism",
      "out_of_memory"
    ],
    [
      "os",
      "iOS 15.4.1"
    ],
    [
      "os.name",
      "iOS"
    ],
    [
      "os.rooted",
      "no"
    ],
    [
      "dist",
      "580"
    ],
    [
      "release",
      "REDACTED"
    ],
    [
      "user",
      "id:REDACTED"
    ]
  ],
  "_metrics": {
    "bytes.ingested.event": 1410,
    "bytes.stored.event": 2623
  },
  "contexts": {
    "app": {
      "app_start_time": "2022-05-23T08:12:56Z",
      "device_app_hash": "REDACTED",
      "build_type": "app store",
      "app_identifier": "com.org.org",
      "app_name": "Organization Name App 2.0",
      "app_version": "1.0.10",
      "app_build": "580",
      "app_id": "REDACTED",
      "type": "app"
    },
    "device": {
      "family": "iOS",
      "model": "iPhone12,8",
      "model_id": "D79AP",
      "memory_size": 3100426240,
      "usable_memory": 1877671936,
      "storage_size": 63966400512,
      "boot_time": "2022-05-07T21:36:47Z",
      "timezone": "CEST",
      "locale": "en_SE",
      "type": "device"
    },
    "os": {
      "name": "iOS",
      "version": "15.4.1",
      "build": "19E258",
      "kernel_version": "Darwin Kernel Version 21.4.0: Mon Feb 21 21:27:53 PST 2022; root:xnu-8020.102.3~1/RELEASE_ARM64_T8030",
      "rooted": false,
      "type": "os"
    }
  },
  "culprit": "",
  "environment": "production",
  "exception": {
    "values": [
      {
        "type": "OutOfMemory",
        "value": "The OS most likely terminated your app because it overused RAM.",
        "mechanism": {
          "type": "out_of_memory",
          "handled": false
        }
      }
    ]
  },
  "fingerprint": [
    "{{ default }}"
  ],
  "grouping_config": {
    "enhancements": "eJybzDRxY3J-bm5-npWRgaGlroGxrpHxBABcYgcZ",
    "id": "newstyle:2019-05-08"
  },
  "hashes": [
    "REDACTED"
  ],
  "key_id": "REDACTED",
  "level": "fatal",
  "logger": "",
  "metadata": {
    "display_title_with_tree_label": true,
    "type": "OutOfMemory",
    "value": "The OS most likely terminated your app because it overused RAM."
  },
  "nodestore_insert": 1653293579.902627,
  "received": 1653293577.576998,
  "sdk": {
    "name": "sentry.cocoa",
    "version": "7.9.0",
    "integrations": [
      "isolateErrorIntegration",
      "runZonedGuardedIntegration",
      "widgetsFlutterBindingIntegration",
      "flutterErrorIntegration",
      "widgetsBindingIntegration"
    ]
  },
  "timestamp": 1653293576.779649,
  "title": "OutOfMemory: The OS most likely terminated your app because it overused RAM.",
  "type": "error",
  "user": {
    "id": "REDACTED",
    "ip_address": "REDACTED",
    "geo": {
      "country_code": "SE",
      "region": "Sweden"
    }
  },
  "version": "7",
  "location": null
}

markuswahl-2 avatar May 30 '22 10:05 markuswahl-2

without-context-information-using-7.11.0-redacted.json

{
  "event_id": "REDACTED",
  "project": REDACTED,
  "release": "REDACTED",
  "dist": "805",
  "platform": "cocoa",
  "message": "",
  "datetime": "2022-05-30T08:51:06.743513Z",
  "tags": [
    [
      "environment",
      "production"
    ],
    [
      "event.environment",
      "native"
    ],
    [
      "event.origin",
      "ios"
    ],
    [
      "handled",
      "no"
    ],
    [
      "level",
      "fatal"
    ],
    [
      "mechanism",
      "out_of_memory"
    ],
    [
      "dist",
      "805"
    ],
    [
      "release",
      "REDACTED"
    ],
    [
      "user",
      "id:REDACTED"
    ]
  ],
  "_metrics": {
    "bytes.ingested.event": 701,
    "bytes.stored.event": 1796
  },
  "breadcrumbs": {
    "values": [
      {
        "timestamp": 1653900666.744,
        "type": "error",
        "category": "sentry.event",
        "level": "error",
        "message": "A value set to the context or sdk is not serializable. Dropping context and sdk."
      }
    ]
  },
  "culprit": "",
  "environment": "production",
  "exception": {
    "values": [
      {
        "type": "OutOfMemory",
        "value": "The OS most likely terminated your app because it overused RAM.",
        "mechanism": {
          "type": "out_of_memory",
          "handled": false
        }
      }
    ]
  },
  "fingerprint": [
    "{{ default }}"
  ],
  "grouping_config": {
    "enhancements": "REDACTED",
    "id": "newstyle:2019-05-08"
  },
  "hashes": [
    "REDACTED"
  ],
  "key_id": "REDACTED",
  "level": "fatal",
  "logger": "",
  "metadata": {
    "display_title_with_tree_label": true,
    "type": "OutOfMemory",
    "value": "The OS most likely terminated your app because it overused RAM."
  },
  "nodestore_insert": 1653900670.293765,
  "received": 1653900667.589878,
  "sdk": {
    "name": "sentry.cocoa",
    "version": "7.11.0"
  },
  "timestamp": 1653900666.743513,
  "title": "OutOfMemory: The OS most likely terminated your app because it overused RAM.",
  "type": "error",
  "user": {
    "id": "REDACTED",
    "ip_address": "REDACTED",
    "geo": {
      "country_code": "SE",
      "city": "Gothenburg",
      "region": "Sweden"
    }
  },
  "version": "7",
  "location": null
}

markuswahl-2 avatar May 30 '22 10:05 markuswahl-2

FYI, the Cocoa SDK creates the OOM errors on heuristics. There is no 100% guarantee that your app really crashed. Especially on hybrid platforms such as Flutter and RN, we had issues in the past that we created OOMs by mistake. You are not the first person to be confused by these OOM errors.

Can you provide a repro for an OOM event maybe?

I don't think that the Cocoa SDK causes the missing APP data. I think that one of the recent changes to the dart SDK causes this:

  • https://github.com/getsentry/sentry-dart/pull/838
  • https://github.com/getsentry/sentry-dart/pull/761

Is the APP data only missing for OOM errors or other events as well?

philipphofmann avatar May 31 '22 07:05 philipphofmann

I see. Thanks for your feedback.

We can see reports in sentry for other events where the APP data section is present.

In regards to the APP data section and the field "Start Time": After reading the heuristics section you mentioned, I understand that the OOM is reported the next time the app starts. If so, what does the "Start Time" field in the APP section indicate? Is it the start time of the app for the session that had an OOM or is it the start time of the app that reports the OOM (i e the next time the app starts)? If it is the latter, I understand that we can not infer how long the app had been live when (if) it experienced OOM, from the start time field in the APP section. Right?

With repro, you mean if we can reproduce it, right? I am afraid not.

markuswahl-2 avatar Jun 02 '22 13:06 markuswahl-2

This could be related to https://github.com/getsentry/sentry-cocoa/issues/1877

philipphofmann avatar Jun 10 '22 08:06 philipphofmann

Hey @markuswahl-2, can you try to update to 7.18.0, because #1877 might have fixed the issue.

philipphofmann avatar Jun 22 '22 13:06 philipphofmann

Will do, thanks!

markuswahl-2 avatar Jun 27 '22 08:06 markuswahl-2

I am afraid we still get those errors even after bumping to 7.18.1 of sentry.cocoa. Also we see a sudden other error (but in android of NoClassDefFoundError: io.sentry.flutter.f) after having bumped to 7.18.1.

markuswahl-2 avatar Jun 29 '22 06:06 markuswahl-2

I'm sorry to hear that, @markuswahl-2. Does your app do any background work? Could it be that it runs in the background and then goes into the foreground when the user actively opens it?

philipphofmann avatar Jun 29 '22 07:06 philipphofmann

I think that is possible. (That's normal for most apps, no?)

markuswahl-2 avatar Jun 29 '22 07:06 markuswahl-2

Yes, that's normal for sure, but I'm just asking because it could be that we don't detect this properly.

Do you maybe have enableAutoSessionTracking disabled, @markuswahl-2? If so, https://github.com/getsentry/sentry-dart/pull/905 would fix the issue.

philipphofmann avatar Jun 29 '22 07:06 philipphofmann

OK, thanks.

It is this one, right? https://pub.dev/documentation/sentry_flutter/latest/sentry_flutter/SentryFlutterOptions/enableAutoSessionTracking.html

It is no entirely clear to me from the docs what enableAutoSessionTracking does.

Do I understand it correctly that setting enableAutoSessionTracking to false (while keeping the default value of enableOutOfMemoryTracking of true), and waiting for https://github.com/getsentry/sentry-dart/pull/905 to become available, I might expect improvements?

It goes without saying that we are very much still interested in getting reports about true OOM, it is just the possibly false OOM reports that we want to skip. 😃

markuswahl-2 avatar Jun 29 '22 11:06 markuswahl-2

(And to answer your question, no we did not have enableAutoSessionTracking disabled, but we are considering to do it, especially once https://github.com/getsentry/sentry-dart/pull/905 becomes available).

markuswahl-2 avatar Jun 29 '22 11:06 markuswahl-2

Do I understand it correctly that setting enableAutoSessionTracking to false (while keeping the default value of enableOutOfMemoryTracking of true), and waiting for https://github.com/getsentry/sentry-dart/pull/905 to become available, I might expect improvements?

No, if you set enableAutoSessionTracking to false, it might get worse. That's why I asked. The default is true. If you didn't touch the option https://github.com/getsentry/sentry-dart/pull/905 should not have an impact on you.

(And to answer your question, no we did not have enableAutoSessionTracking disabled, but we are considering to do it, especially once getsentry/sentry-dart#905 becomes available).

Why do you plan on disabling enableAutoSessionTracking? If you had it disabled in the past it could lead to wrongly reported OOMs. https://github.com/getsentry/sentry-dart/pull/905 fixes this problem.

philipphofmann avatar Jun 29 '22 14:06 philipphofmann

@markuswahl-2, I'm out of ideas. If you want, can you maybe send me the link to your project to [email protected] so that I can look at your data? Maybe I spot something.

philipphofmann avatar Jun 29 '22 14:06 philipphofmann

No, I meant that we were eager to try setting enableAutoSessionTracking to false because I, incorrectly, understood from you that that might correct it. The setting enableAutoSessionTracking is and has always been unmodified in our source code. And now that I understand you, we will of course not disable that setting.

markuswahl-2 avatar Jun 30 '22 11:06 markuswahl-2

Here are the five files in our app that touches upon Sentry. (I added suffix .txt because github did not allow them otherwise): Podfile.lock.txt leading_icon.dart.txt main.dart.txt pubspec.yaml.txt pubspec.lock.txt

markuswahl-2 avatar Jul 01 '22 10:07 markuswahl-2

Thanks a lot for your update, @markuswahl-2. I didn't have time yet to look at your setup. I will get back to you later.

philipphofmann avatar Jul 05 '22 15:07 philipphofmann

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

github-actions[bot] avatar Jul 27 '22 00:07 github-actions[bot]

Hi Philipp! I was away from keyboard during the summer. I am back now. Have you had some insight into this?

Best regards

markuswahl-2 avatar Aug 15 '22 06:08 markuswahl-2

Hello @markuswahl-2 Philipp is in PTO. Sorry but we need to wait for him to came back to look into this. We don't have enough bandwidth right now.

brustolin avatar Aug 16 '22 08:08 brustolin

Hello @philipphofmann Have you had time to look at the files? Best regards, Markus

markuswahl-2 avatar Sep 06 '22 05:09 markuswahl-2

Hey, @markuswahl-2,

sorry, no didn't get to that yet. Try to do it this week.

philipphofmann avatar Sep 07 '22 13:09 philipphofmann

Hi @kevinrenskers Have you have had a chance to look into this?

best regards, Markus

markuswahl-2 avatar Sep 28 '22 05:09 markuswahl-2

I am looking into this now.

kevinrenskers avatar Oct 03 '22 14:10 kevinrenskers

Hey, @markuswahl-2,

Could it be that some parts of your app use exit(code)? We have an issue on React-Native for creating wrong OOMs when some code calls exit(code). https://github.com/getsentry/sentry-docs/issues/4350.

We don't have any other ideas what could cause your OOMs at the moment. Do you by any chance have a way to reproduce it now?

philipphofmann avatar Oct 11 '22 13:10 philipphofmann

@philipphofmann I can also see that if I just stop the debugger and open it again, but this is likely an exit(0) example, will try a release mode.

marandaneto avatar Oct 13 '22 08:10 marandaneto

@markuswahl-2 I could not reproduce it using a release build, can you provide a minimal reproducible example and the steps to reproduce it?

marandaneto avatar Oct 13 '22 11:10 marandaneto