Waiting for GPU is killing performance. SUPER CRITICAL BUG!!!

[hardcoremore]

trafficstars

Hi @ajwfrost,

This is a huge issue Adobe never fixed that is destroying performance for NO GOOD reason. This seams to be happening on iOS, Windows and Android. I was not able to reproduce it on Mac OS.

Can you please put this on a high priority, because it is destroying everything I worked for?

Here you can find Adobe Scout profile that I have just recorded: N/A

If you look at the profile you can see that my game is running smooth except when this Waiting for GPU kicks in for no good reason. You can observe this on the frames at the end of scout profile.

What it looks like is that AIR suddenly locks application to 30 FPS for no good reason.

Also you can notice how AIR is pretty bad with keeping each frame at exact FPS even if application takes way less time to process and render than entire frame time.

ONE BIG NOTE: This was tested with filters removed. So this bad timing IS NOT caused by this issue #20

Please take a look at this, it is a really big issue for games. This is still happening with latest AIR 33.1.1.98.

Regards,

Caslav

[itlancer]

Related issues: https://github.com/Gamua/Adobe-Runtime-Support/issues/85 https://tracker.adobe.com/#/view/AIR-4198654 https://tracker.adobe.com/#/view/AIR-3651843

[ajwfrost]

Hi

As far as I know, "waiting for GPU" is where we've submitted the requests (normally asynchronous) into the GPU and are then waiting for them to complete i.e. once we've completed a frame and are waiting before we can start the next frame rendering. But if this sort of thing is happening across platforms and mechanisms - Android/iOS using OpenGL ES, Windows using Direct3D 11 - then it may imply there's something that the runtime is doing here which is wrong.

Could we get a copy of this game to run this locally and see if we can reproduce it on the development machines - are there any special steps to trigger this (there are lots of mouse events then key presses happening.. or do we just play the game!)

thanks

[hardcoremore]

Hi @ajwfrost,

Where I can send you the game?

I have a .exe file of my game which is not released yet and has test levels with which I can reproduce this bug every time. It always happen in under a minute of gameplay, it doesn't take much.

[ajwfrost]

Can you try uploading with the below please? https://transfer.harman.com/requests/aQ4rFkPgIETfoiPrOuqOxU

[hardcoremore]

Hi @ajwfrost ,

I have sent you release build of the game. Please confirm if you receive all the files and if you can run it and debug it.

Also I have discovered something very strange. This bug can be reproduced on my lap top 100% of the time when I plug in my charger, but when my lap top is running only on battery this bug is very rare. Could this be connected to power state of the device. I remember some other users could trigger this behaviour by changing screen brightness of the mobile device.

Anyway, still even when this bug does not happen key frames of Adobe AIR seams to be very toothy even when very little time of entire frame time is used, so it would be also great idea to check if timings are calculated properly and with enough precision.

Regards,

Caslav

Regards,

Caslav

[ajwfrost]

Thanks @hardcoremore, yes we've got the files and will look into it. Curious about this not happening when on battery, as far as I'm aware AIR wouldn't be doing anything differently based on the power state!

But yes, we can try to see what's actually happening regarding all the timings, and what's making it wait..

[hardcoremore]

Hi @ajwfrost ,

Any news on this. Where you able to reproduce the issue with the app I sent you?

Thanks, Caslav

[hardcoremore]

Hi @ajwfrost,

Any news on this one? Where you able to reproduce issue with the game I have sent you.

Regards, Caslav

[ajwfrost]

Hi - sorry, missed that earlier request! Will check on this one, I have a recollection that we did reproduce it with your app but hadn't got further with it.. but I'll need to double-check with the team next week.

thanks

[hardcoremore]

Thanks @ajwfrost,

Its great that you were able to reproduce this issue with the game I sent you.

Looking forward to fixing this ugly bug because it is a really big blocker for games.

Regards, Caslav

[hardcoremore]

Hi @ajwfrost,

Was there any progress on this issue?

Regards, Caslav

[ajwfrost]

@hardcoremore not got anywhere with this as yet, aiming to understand that GPU memory issue first as the two may be linked (typically if we settled into a period where there's a need to do extra GC clean-up, this sort of drop in performance could be a result...)

[ajwfrost]

@hardcoremore sorry we can't find the game that you sent us for some reason .. is it possible to get that again as the online cache has expired? I'd like to check it against the runtime here that's fixed the GPU memory leak.. https://transfer.harman.com/requests/lAvvcpQgLDixUe1A90EZ5B

thanks

[hardcoremore]

Hi @ajwfrost,

Awesome, I have just sent the new version. Hope to see this bug fixed as well.

[hardcoremore]

Also @ajwfrost, I have one more request.

Can you please load level 42 "Polygon Collision Test" from the game I have sent you and play it.

It will be slow because there are a lot of calculations and function calls (physics simulation) but can you please do some quick profiling just to check if there is something that can improve performance that Adobe didn't implement right or bother to do.

Just to check when you run that level if there is something that stands out that can be implemented in much more efficient way. It would mean a lot and I think that can be very good real scenario to stress test AIR.

Thanks

[hardcoremore]

Hi @ajwfrost,

Any news about this issue?

[hardcoremore]

@ajwfrost,

I have tested this with latest AIR 33.1.1.217 and I can still reproduce it always. It destroys performance on PC completely by locking it to 30FPS.

This is a HUGE problem because my game is too complex to create levels with debug player (debug player is too slow and I can not use it to create and test levels).

I can only create levels for my game when I export production build to captive run time for Windows and there performance are great except for this which is practically destroying everything.

Literally I am in position where I have been working on a game for 5 years and now I can not create and test levels because of this bug which is several years old.

[Adolio]

Hi everyone,

I think I'm now facing this issue as well. My game is usually running smoothly at 60 fps but frequently the frame rate falls down to 30 fps for no apparent reasons and gets stuck to this value until I do some (big?) changes in the rendering (like showing the pause overlay panel or resetting the current level). That's a bit weird and unfortunately I don't have anything on my side to easily reproduce the issue...

I'm using Starling 2.6 & the latest Adobe AIR SDK from Harman (31.1.1.217) under Windows 10 (with a 64-bit build).

Best, Aurélien

[Seitei]

I can confirm same happens to me, after a Windows update last night, the debugger version runs at a capped ~32 FPS.

[tudal]

I have the same issue. Only in debug mode. Weirdly when I have open Scout with ActionScript Sampler enabled then I have 60 fps back. Please fix this because testing on debug is pain right now.

[OliverJoyce]

I can confirm that since the latest Windows 10 update, this is happening to me too. AIR apps that ran at 60fps are now capped at around 32fps. This is a real showstopping bug for AIR, anyone found a solution yet?

[hardcoremore]

Hi @ajwfrost, is there any progress with this issue?

[teotigraphix]

Same, the Windows Update just capped by frame rate to 30 as well. Guess I will try and go back to an older SDK so I can test for "real".

[teotigraphix]

@ajwfrost I just rolled back to an earlier SDK that did not have this problem and the problem still exists. So this definitely is something on a hardware/software level that changed.

Debug mode stuck at <=30 FPS.

[teotigraphix]

UPDATE: I have managed to hack fix this;

I have a feeling it's the Power Management mode which was at Optimum, I set it back to max performance.

I also changed the max FPS but I don't think that was it, the same issue happens with USB and performance audio, if the settings get messed up, you get crackling, here we get clamped frame rate.

I have run the debugger multiple times and I finally get 60 FPS back on Windows 10.

[BMGMobile]

Sharing a workaround from another issue. It appears that this bug only occurs if the frame rate is set near or lower than the display refresh rate. Doing the following seems to work around the issue: stage.frameRate = 1000.0;

If you're using the ENTER_FRAME callback you will be called after each vsync, so 60 FPS on a typical monitor, but more frequently on 120/144/240/320Hz monitors. You'll have to manage your own logic to handle faster refresh rates.

[ajwfrost]

Belatedly picking up this thread.. it's looking possible that the fact that things slow down after a while could be due to the GPU getting hot and therefore getting throttled, which would perhaps tie in with some of the observations (and hack for changing the power management mode). But also if it's something that would be based on some clock skews based on framerates, then we may want to change how the enter-frame thing works a little.

Currently you set the desired frame rate e.g. via the authoring tools, which is then embedded into the SWF file header. Then there's some logic in the core rendering loop that decides whether or not it's time to draw a new frame, based on the time since the last one (it tries to catch up if the actual frame processing/rendering takes too long). The trigger for this check, at least on Windows, is a timer that's going off at a higher rate (10x) where we're also checking for other activity such as input events and network updates etc.

So, if we come out of a rendering loop and then expect to have a frame delay of 1000/60 = 16ms, it might be that by the time we've done the next bit of rendering we have then missed that next vsync slot and we just get delayed due to the way in which the GPU works. i.e. Time 1000 -> Frame 1 start processing Time 1010 -> Frame 1 complete processing Time 1016 -> Flip GPU so frame appears Time 1022 perhaps -> Frame 2 start processing Time 1032 -> Flip GPU but nothing provided yet Time 1032 -> Frame 2 complete processing Time 1048 -> Flip GPU so frame appears

This is a bit of conjecture but we'll do some checking to see if this might be what's happening. We've been trying to work out where Scout gets its numbers from, but it appears this is doing some calculations based on the data that's getting dispatched from the AIR runtime, there's no "waiting for GPU" value being passed across which is making it a little harder to see where this is happening!

thanks

[BMGMobile]

Glad you're looking into it! From what we've seen, this is not a GPU throttling issue, especially since a workaround is to ask Adobe Air to render at 1000FPS. Our first reports of this issue were from players using 3rd party utilities to monitor frame rate, and we've been able to verify that our game stutters in this state, so it's not just a Adobe Scout issue.

[hardcoremore]

Thanks for looking at this @ajwfrost !

[ajwfrost]

Hi all. There's some curious behaviour here ... but essentially it appears that:

1. enabling the advanced telemetry, or getting the player to do some other work, is helping in some way to keep the runtime working and running more smoothly - we're not sure why this it yet!
2. if you don't have much going on then after a while the Windows system seems to just call back our application less frequently which means we're then skipping frames...
3. ... and that's mostly down to how the main run loop is designed. It looks fine in theory, but in practice it doesn't seem to work how we would expect.

Actions we're looking at: a) investigate further to see if there is something we can learn from the advanced telemetry setting or similar, to try to keep the frame rate up without having to do extra work! b) reworking the main event loop -> could be a relatively tricky thing to do especially given how AIR can end up with multiple Windows all being controlled by one instance of the AVM - does anyone here actually use multiple NativeWindow objects as we could perhaps create a new mode for high-performance requirements at the expense of losing multi-window support? c) actually have the internal timing logic change based on the Stage.vsyncEnabled setting (currently turning it off just means that D3D11 would do the 'present' and not wait anywhere, so framerate would be ignored?) d) adding some other options into the AIR descriptor file so that you can choose some more control over things such as Direct3D level

Comments/thoughts welcome..

thanks