Perceivable jank in the team tab

[filiph]

I'm pretty sure the jank is there even in production, even on high end phones.

If it's not just me holding it wrong, I think this is a blocker, and we should do everything in our power to fix it.

AFAIK, this is mostly about the Flare widgets being shown for the first time. An initial idea, if that's the case, is to scatter the showing of the flare files. A more drastic measure is to use a bitmap instead, and only use Flare when we must.

But first, @efortuna — can you please try and run the app in --production on some real world device? I've tried the two (Android and iOS) devices available to me and I'm pretty sure I see jank on both.

[filiph]

https://photos.app.goo.gl/MwFW9NAr5zu6YuWf7, the actual walkthrough starts at about 0:20. This is driven by an Acer chromebox, one of the fastest ones out there.

[luigi-rosso]

I'm seeing some really high GPU times, which is really surprising given that we're mostly using raster images in these Flare files.

What's even more surprising is that it's sort of random as to when it kicks in. Take a look at 16 seconds, it's playing the same file it has been playing with the same animation and all the sudden the GPU goes from 5.7 ms per frame to close to 16.4 (which is practically vsync). Would be good to see if someone can further profile what's going on with the GPU.

https://drive.google.com/file/d/1c2rWFx7iTyDbc57awoTHmv5DccmZdkl9/view?usp=sharing

[luigi-rosso]

Here's a walkthrough on the iPhone X, even with GPU time getting close to the max, it still appears jank free: https://drive.google.com/open?id=1F5d72aJWY68M739amAp7jwaeGNa7gRLv

I'm going to test a little with an inexpensive Nokia...

[luigi-rosso]

I think fixing the UI on those screens will definitely help some. I wonder if the combination of the color filter (to make the images B&W) and the really large fill is causing it to jank. It does really seem like it struggles the first time the file is loaded.

It definitely janks on the Nokia and the scrolling of that list is practically impossible even in release mode. I'll do some investigation...

[efortuna]

I tested on a couple of different phones. iPhone 8 slight jank moving from first screen to the team, but scrolling seems okay. Nexus 5 jank moving from first screen to the rest, also scrolling is bad.

I tested on iPhone X and Pixel 3 are both are good.

The problem is our display is using iPhone 8s (and Pixels and the big screens which Filip mentioned) so this is a pretty high priority. Also we want other people downloading the app to have it feel responsive.

[luigi-rosso]

One significant drain on the GPU of the Nokia is the oval+rect clipping around the character's bust.

We can help it and keep most of the look and feel by killing the rectangular portion which we compute in screen space to sort of let the artwork bleed out the top and not get confined by the oval.. There's something about this changing size that seems to affect performance.

Performance is still poor on the Nokia, but you can actually use the list (went from an avg 62 ms/frame to 34ms/frame) by commenting out the clip.addRect lines: https://github.com/2d-inc/dev_rpg/blob/101bd8232fdd91c71e46a3ecc157cfed1d2d5d1d/lib/src/widgets/flare/hiring_bust.dart#L180

I'd be curious to see if that helps other devices too.

[efortuna]

I thiiiiink it fixed the problem going from the first page to the team page on the iPhone 8.

Still seeing some jank on the Nexus 5 going from the first page to the team. Team scrolling is possibly slightly better? there's still a little jank but maybe a little les.

[luigi-rosso]

So the jank is definitely related to loading the characters. On the iPhoneX in debug:

flutter: Took 462 ms for assets/flare/Sourcerer.flr
flutter: Took 190 ms for assets/flare/Users.flr
flutter: Took 202 ms for assets/flare/Joy.flr
flutter: Took 211 ms for assets/flare/Coin.flr
flutter: Took 300 ms for assets/flare/TheJack.flr
flutter: Took 532 ms for assets/flare/TheRefactorer.flr
flutter: Took 1303 ms for assets/flare/TheArchitect.flr
flutter: Took 1892 ms for assets/flare/TheHacker.flr
flutter: Took 2035 ms for assets/flare/ProgramManager.flr
flutter: Took 2249 ms for assets/flare/UXResearcher.flr
flutter: Took 604 ms for assets/flare/Designer.flr
flutter: Took 728 ms for assets/flare/Tester.flr

Even in release it takes way too long:

flutter: Took 83 ms for assets/flare/Designer.flr
flutter: Took 60 ms for assets/flare/TheRefactorer.flr
flutter: Took 33 ms for assets/flare/Users.flr
flutter: Took 33 ms for assets/flare/Joy.flr
flutter: Took 33 ms for assets/flare/Coin.flr
flutter: Took 41 ms for assets/flare/TheJack.flr
flutter: Took 71 ms for assets/flare/Sourcerer.flr
flutter: Took 146 ms for assets/flare/TheArchitect.flr
flutter: Took 199 ms for assets/flare/TheHacker.flr
flutter: Took 210 ms for assets/flare/ProgramManager.flr
flutter: Took 234 ms for assets/flare/UXResearcher.flr
flutter: Took 16 ms for assets/flare/Tester.flr

It's interesting how some files are so much slower than others. This is without images, btw.

I'm still investigating... but in the meantime I also did a test with preloading all the Flare files in the cache, which removes the jank, but delays boot (not proposing we do this, but helps test): https://github.com/luigi-rosso/dev_rpg/commit/154a421f7d8d4c69ecbc9497b9b4e10f2ae3bcaf

[efortuna]

Hmm... is it trying to load all of those even before we've scrolled them into view (jank moving from first screen to the team screen) ? or is the scrolling jank because it's trying to load them, but not loading them fast enough?

[luigi-rosso]

It's the latter, scrolling causes some Flare file to be requested by the UI, the load occurs, takes too long...jank. Once you've scrolled the whole list, it shouldn't jank anymore. In fact if you sit a long time on the home screen and let it cycle through all the characters, you'll see much less jank when starting the game.

The warmup_flare branch I linked above does the first thing you mentioned, it tries to preload everything and it prevents the jank.

[luigi-rosso]

All that said, I'm still getting to the root cause of what is making the load so slow.

[efortuna]

you can also increase the amount of stuff that it draws offscreen to prepare for scrolling... I believe with cacheExtent in the GridView? https://docs.flutter.io/flutter/widgets/ScrollView/cacheExtent.html I haven't played with this extensively but that may help "warm up" the files ahead of scrolling them.

[luigi-rosso]

That'll help with scrolling, but the initial jank when navigating to that page will still be there.

I've improved it quite a bit by changing some of the byte reading to generating new ByteData views. Unfortunately this doesn't work for unaligned multibyte types (Float/Uint/etc) so those still get read in a tight loop. I'm doing some more profiling still, but it should already be quite a bit better.

I haven't messed around with Isolates yet but is that basically the only way to do loading in something akin to a background thread?

Debug:

flutter: Took 219 ms for assets/flare/Users.flr
flutter: Took 231 ms for assets/flare/Joy.flr
flutter: Took 241 ms for assets/flare/Coin.flr
flutter: Took 195 ms for assets/flare/TheJack.flr
flutter: Took 227 ms for assets/flare/Sourcerer.flr
flutter: Took 264 ms for assets/flare/TheRefactorer.flr
flutter: Took 339 ms for assets/flare/TheArchitect.flr
flutter: Took 343 ms for assets/flare/ProgramManager.flr
flutter: Took 373 ms for assets/flare/UXResearcher.flr

Release:

flutter: Took 9 ms for assets/flare/ProgramManager.flr
flutter: Took 31 ms for assets/flare/Users.flr
flutter: Took 30 ms for assets/flare/Joy.flr
flutter: Took 28 ms for assets/flare/Coin.flr
flutter: Took 25 ms for assets/flare/TheJack.flr
flutter: Took 31 ms for assets/flare/Sourcerer.flr
flutter: Took 42 ms for assets/flare/TheRefactorer.flr
flutter: Took 68 ms for assets/flare/TheArchitect.flr
flutter: Took 92 ms for assets/flare/TheHacker.flr
flutter: Took 98 ms for assets/flare/UXResearcher.flr
flutter: Took 36 ms for assets/flare/Designer.flr
flutter: Took 41 ms for assets/flare/Tester.flr

[filiph]

Thanks for looking into it, Luigi. These are some really good results in a very short span.

Isolate is a background thread, yes, but isolates cannot share memory, so I'm not sure if it's helpful for you. Basically, whatever the background thread does, it needs to serialize the result and send it back to the main isolate. If all you do here is deserialization, then an Isolate might not help you.

(For completeness: you can send primitive data types without serialization. And you can also send the deserialized file in parts so that the UI thread never needs to spend too much time dealing with the flr file at once.)

[luigi-rosso]

I think I figured out how to make it work with a full object from a custom class. Notice the special circumstances text here: https://api.dartlang.org/stable/2.2.0/dart-isolate/SendPort/send.html

There are some restrictions, I had to rework how we instance certain objects that are not available in the Isolate (like painting/canvas objects) but provided it's only user defined classes and primitives, it seems to work ok. Still need to do some cleanup (and will need to push a new Flare package with breaking changes).

Flare animations now pop in once they load, but this should be better than the jank and we could hide it with a quick fade (although sometimes that feels superfluous).

The biggest problem now is that there is still some jank which I suspect is from the images decoding. I cannot move that to an isolate. I wonder if someone on the Flutter team has a suggestion for how to load these more efficiently. We do this right now:

// List<Uint8List> data = ... loaded from Flare file ...
List<ui.Codec> codecs = await Future.wait(data.map(ui.instantiateImageCodec));
List<ui.FrameInfo> frames = await Future.wait(codecs.map((ui.Codec codec) => codec.getNextFrame()));
_images = frames.map((ui.FrameInfo frame) => frame.image).toList(growable: false);

I suspect reducing the image resolution will help with this. Overall it does already feel much better, however.

These are the timings for just the image decode on the lower end nokia:

decode: 321 ms
decode: 172 ms
decode: 416 ms
decode: 898 ms
decode: 912 ms
decode: 1585 ms
decode: 233 ms
decode: 697 ms

I'll re-export everything at half resolution and see what it's like.

[efortuna]

FYI -- just ran on Pixel 3 and am seeing lag between the "start game" and displaying the team tab

[luigi-rosso]

Is that in release/debug? Did it get worse?

[efortuna]

ugh I'm so sorry yes it was in debug. I will stop talking about this now (hangs head)

[luigi-rosso]

Np, I think it'll get better in debug with the warmup stuff too.

[luigi-rosso]

I caught a condition where in some cases we'd load Flare files twice. This is fixed in the latest Flare which my chompy branch uses. The chompy PR will include this fix which should further help janks from the welcome screen to the game.