Adobe-Runtime-Support icon indicating copy to clipboard operation
Adobe-Runtime-Support copied to clipboard

Increased ANR rate with 51.2.2.4

Open yvant opened this issue 2 months ago • 36 comments

I've noticed a visible increase of my ANR over the previous weeks. After some digging, I came to the conclusion that the issue comes from AIR 51.2.2.4.

Here's my global ANR rate from July 4th. It's already pretty high, but we can see an increase at the beginning of september. Image

Here are my ANR rate for each version of my game and their release dates

Image
  • 4.22.24 uses a modified version of the 51.2.2.2 from this thread

Additionally, here are some screens from Crashlytics. I highlighted only the new ANR types.

  • 4.22.25 Image

  • 4.22.26 Image

  • 4.22.27 Image

  • 4.22.28 Image

I've just seen that the AIR 51.2.2.5 has been released. Can it fix those ANR? Downgrading to 51.1.4.1 apparently doesn't fix the ANR rate too much. So maybe some issue has been introduced and merged into that version as well. In the meantime, I'm going back to the modified 51.2.2.2 since I had the lowest ANR rate with that one.

yvant avatar Oct 02 '25 15:10 yvant

I checked another issue that seems linked: the Frozen frames has increased at the same time approximatively:

Image

The 4.22.25 (AIR 51.2.2.4) has higher frozen frames than the 4.22.24 (AIR 51.2.2.2 modified)

yvant avatar Oct 02 '25 16:10 yvant

Thanks for the details - can I just check what I think is a typo, at the end of the initial comment you say "I'm going back to the modified 51.2.2.4 since I had the lowest ANR rate with that one" - I think you mean "modified 51.2.2.2" based on the table and note you'd written above that?

And also -> between the modified 51.2.2.2 and the 51.2.2.4 version, did you change anything else e.g. updating ANEs, since I can see mention of com.google.android in a number of the ANRs. There seem to be a number of fairly generic ones too e.g. the file descriptor reading, I've seen that a couple of times recently now - what version of AIR SDK is within 4.22.268, is your game version in the table essentially meaning "4.22.26" covers all "4.22.26*"?

thanks

ajwfrost avatar Oct 02 '25 21:10 ajwfrost

Right, that's a typo: I meant that I'll go back to the modified 51.2.2.2.

The 4.22.24 version (2025.08.27) of the game included a bunch of ANE updates. Here's a screenshot of the mail updates from Distriqt: Image

Regarding the versionning, I use 8 for ARM8 and 7 for ARM7. So 4.22.268 and 4.22.267 are both 4.22.26. There's no actual 4.22.26.

yvant avatar Oct 02 '25 22:10 yvant

Can you send through the details on some of those ANR's related to the extensions? They look to be internal to the google libs (which isn't uncommon) but I haven't seen one around UMP init before.

marchbold avatar Oct 02 '25 22:10 marchbold

Sure, which ones? You can mention the version + the required lines if it's more convenient.

yvant avatar Oct 02 '25 22:10 yvant

The GoogleAdMobUserMessagingPlatform init one under 4.22.27

marchbold avatar Oct 02 '25 22:10 marchbold

      main (native):tid=1 systid=16111 
   at com.distriqt.extension.adverts.platforms.admob.GoogleAdMobUserMessagingPlatform.<init>(GoogleAdMobUserMessagingPlatform.java:48)
   at com.distriqt.extension.adverts.platforms.admob.GoogleAdMob.ump(GoogleAdMob.java:741)
   at com.distriqt.extension.adverts.functions.ump.RequestConsentInfoUpdateFunction.call(RequestConsentInfoUpdateFunction.java:45)
   at com.adobe.air.customHandler.callTimeoutFunction(Native method)
   at com.adobe.air.customHandler.handleMessage(customHandler.java:35)
   at android.os.Handler.dispatchMessage(Handler.java:106)
   at android.os.Looper.loopOnce(Looper.java:346)
   at android.os.Looper.loop(Looper.java:475)
   at android.app.ActivityThread.main(ActivityThread.java:7889)
   at java.lang.reflect.Method.invoke(Native method)
   at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548)
   at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1009)

Like that?

yvant avatar Oct 02 '25 22:10 yvant

Interesting. I think I may be able to do something for this one. Are you using UMP early in your app?

marchbold avatar Oct 03 '25 05:10 marchbold

Regarding the loading of the ANEs, I’m using a sequence that allows 200 ms for each of them to load. For example, I launch Facebook initialisation then 200ms later, I'll launch IAP initialisation then Firebase etc. I don't know if it's the right way to load ANEs, but I've been doing this for a while now. This sequence is launched in the first frame but the UMP comes after all other initialisations, so after 1.5s - 2s.

yvant avatar Oct 03 '25 08:10 yvant

Have you set the OPTIMIZE_INITIALIZATION flag at all?

<meta-data
          android:name="com.google.android.gms.ads.flag.OPTIMIZE_INITIALIZATION"
          android:value="true"/>

marchbold avatar Oct 05 '25 05:10 marchbold

No I haven't, I just have <meta-dataandroid:name="com.google.android.gms.ads.flag.OPTIMIZE_AD_LOADING"android:value="true"/> from APM. Should I add it beneath that line?

yvant avatar Oct 05 '25 09:10 yvant

I'm seeing a similar thread also. interestingly the crash rates are decreasing but ANR is going up and up as soon as I updated to AIRSDK_51.2.2. The speed of ANRs increasing is looking very scary.

Image

most errors are from here:

      at com.adobe.air.customHandler.callTimeoutFunction (Native method)
      at com.adobe.air.customHandler.handleMessage (customHandler.java:35)
      at android.os.Handler.dispatchMessage (Handler.java:106)
      at android.os.Looper.loopOnce (Looper.java:201)
      at android.os.Looper.loop (Looper.java:288)
      at android.app.ActivityThread.main (ActivityThread.java:7941)
      at java.lang.reflect.Method.invoke (Native method)
      at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run (RuntimeInit.java:569)  
      at com.android.internal.os.ZygoteInit.main (ZygoteInit.java:1015)

and here:

      at com.adobe.air.Entrypoints.EntryMainWrapper (Native method)
      at com.adobe.air.Entrypoints.EntryMain (Entrypoints.java:267)
      at com.adobe.air.AndroidActivityWrapper.LaunchApplication (AndroidActivityWrapper.java:1307)
      at com.adobe.air.AndroidActivityWrapper.launchApplication (AndroidActivityWrapper.java:1663)
      at com.adobe.air.AndroidActivityWrapper$3.run (AndroidActivityWrapper.java:1615)
      at android.os.Handler.handleCallback (Handler.java:938)
      at android.os.Handler.dispatchMessage (Handler.java:99)
      at android.os.Looper.loopOnce (Looper.java:211)
      at android.os.Looper.loop (Looper.java:300)
      at android.app.ActivityThread.main (ActivityThread.java:8279)
      at java.lang.reflect.Method.invoke (Native method)
      at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run (RuntimeInit.java:576)
      at com.android.internal.os.ZygoteInit.main (ZygoteInit.java:1074

myflashlab avatar Oct 05 '25 21:10 myflashlab

For me also anr increased dramatically...

hadisn avatar Oct 06 '25 09:10 hadisn

@hadisn @myflashlab can I please check which exact version of the AIR SDK you saw the ANR start to increase? There were some issues in earlier 51.2.2.x variants which could be responsible for this particularly in the EntryMain call stack, but the 51.2.2.4 version (and .5) should have resolved that..

thanks

ajwfrost avatar Oct 06 '25 10:10 ajwfrost

I've used 51.2.2.4 and not the earlier ones. And two days ago updated to .5 and seems like the same issues are there also.

myflashlab avatar Oct 06 '25 11:10 myflashlab

That's not good .. is it possible to get our diagnostics ANE included in this and see if that provides any useful information on the ANRs - currently it's a little blunt but it may give a bit of data that points towards an area to investigate further...

Please only trigger the diagnostics session when it's on an ARMv8 device though, we had a report of instability from ARMv7..

ajwfrost avatar Oct 06 '25 11:10 ajwfrost

That's not good .. is it possible to get our diagnostics ANE included in this and see if that provides any useful information on the ANRs - currently it's a little blunt but it may give a bit of data that points towards an area to investigate further...

Please only trigger the diagnostics session when it's on an ARMv8 device though, we had a report of instability from ARMv7..

@ajwfrost ah! I didn't know there is a diagnostics ANE around. Where can I get it? Update: It's here: https://github.com/airsdk/Adobe-Runtime-Support/tree/master/samples/air_diagnostics

myflashlab avatar Oct 06 '25 14:10 myflashlab

In my case com.adobe.air.Entrypoints.EntryMainWrapper not present in current sdk (51.2.2.x) for now... but anr rate increased from 0.17 to 0.73...

I uploaded versoin with <meta-data android:name="com.google.android.gms.ads.flag.OPTIMIZE_INITIALIZATION" android:value="true"/> to test.

Before I had enabled runtimeInBackgroundThread but was forced to disable it in previous versions of sdk because of anr's. Don't know what to do now...

hadisn avatar Oct 06 '25 15:10 hadisn

@ajwfrost is there something like listener for splash screen so we can do stuff after air splash screen is removed...

hadisn avatar Oct 06 '25 17:10 hadisn

Have you set the OPTIMIZE_INITIALIZATION flag at all?

<meta-data
          android:name="com.google.android.gms.ads.flag.OPTIMIZE_INITIALIZATION"
          android:value="true"/>

Is this a new feature? Can it be combined with the following setting?

httpwebmedia avatar Oct 07 '25 08:10 httpwebmedia

It's a bit early, but here's my ANR rate switching back to AIR 51.2.2.2 (modified):

Image

There's a really high spike at the beginning (few number of installs are impacted by even 1 or 2 ANR) but then it seems to reach its standard value. I will post some more in a few days.

yvant avatar Oct 07 '25 15:10 yvant

Have you set the OPTIMIZE_INITIALIZATION flag at all?

<meta-data
          android:name="com.google.android.gms.ads.flag.OPTIMIZE_INITIALIZATION"
          android:value="true"/>

Is this a new feature? Can it be combined with the following setting?

Yes, was added by Admob recently.

marchbold avatar Oct 08 '25 01:10 marchbold

Here's an update of my ANR rate:

Image

So it seems that the AIR 51.2.2.2 (modified) leads to a lower ANR rate. I can try AIR 51.2.2.3 if I get the same modifications (splash screen and ABI when using apk and not aab), if you think it can help.

yvant avatar Oct 14 '25 23:10 yvant

That's not good .. is it possible to get our diagnostics ANE included in this and see if that provides any useful information on the ANRs - currently it's a little blunt but it may give a bit of data that points towards an area to investigate further...

Please only trigger the diagnostics session when it's on an ARMv8 device though, we had a report of instability from ARMv7..

@ajwfrost Ok, I added the diagnostic ANE - please let me know if i have to change anything in the implementation?

// set up diagnostics
		if(Capabilities.supports64BitProcesses)
		{
			_diagInfo = DiagnosticInfo.instance;
			_diagInfo.debugFunction = debugFunc;
			checkForOldDiagnostics();
			setupDiagnostics();
			_diagInfo.start(
					AIRDiagnostics.DiagCheckLongFunc |
					AIRDiagnostics.DiagLogErrorCallstacks |
					AIRDiagnostics.DiagLogWorker
			);
		}

and am releasing to users slowly and gradually, here are the first logs i have just extracted: I didn't find anything useful in them, but maybe you can? let me know if i can somehow provide more info?

diagnostics.zip

myflashlab avatar Oct 19 '25 08:10 myflashlab

Hi

The only thing I found in that set of diagnostics was that I think it's not cleaning up on application exit .. if you're able to add that option in, it may help reduce some of the unnecessary ones.

From all the various logs, we could only find one instance where things seem to go really wrong. There are actually a lot of the long-running functions - and all the ones we looked at were just "process frame" i.e. it's ActionScript that's taking a long time. There was then one instance where a function took 10+ seconds (you can see in the log the reports of a long-running frame where we output the time it's taken, every second, and it just keeps increasing). Anything that gets to 5s or so is going to be a candidate for ANR. Really not sure why might be causing a function to take over 5s, but it should mean it's AS3 itself (I'm pretty sure if it's a call out to an ANE, we'd log that separately, and the same goes for GPU activities such as uploads).

Next steps I think on our side are to add some more diagnostic output for this area so that we can check what's going on within the AS3 code when we hit e.g. 3 or 4 seconds. Which could be two methods calling each other back and forth etc, or a tight loop, or recursion (although I'd expect that to hit a stack overflow before 5s is up) etc.

thanks

ajwfrost avatar Oct 20 '25 09:10 ajwfrost

it's not cleaning up on application exit

Correct. in docs I did see where it was suggesting to exit the app when the app goes to background. but unfortunately that is not an option for my app and cannot add that logic as it is important that the app is still running some processes while still in background.

myflashlab avatar Oct 22 '25 12:10 myflashlab

ANR is not still looking good but at least it is stable and not increasing anymore.

Image

Really looking forward to a possible fix as it seems like it is starting to hit the business. Here I am attaching the logs for the past few weeks... and turning off the diagnostics generator for now. If any hint is supposed to be found, it should be in these logs.

diagnostics.zip

myflashlab avatar Nov 03 '25 10:11 myflashlab

Image

It's getting better on my side. I've been using AIR 51.2.2.2 (modified) and also removed the Facebook ANE since I don't really use it atm.

yvant avatar Nov 03 '25 16:11 yvant

@ajwfrost I hope you are finding a solution soon. I have another news which might be somehow related: When I run the app in Emulator, very rarely, I see the freezing popup. even though the app works perfectly fine, I have to click on the "wait" button.

This never happened on my a real device though and it might have nothing to do with this case. but thought to mention it anyway.

Image

myflashlab avatar Nov 19 '25 17:11 myflashlab

same for 51.2.2.5

gchitaladze avatar Nov 25 '25 11:11 gchitaladze