Improve Notification GIFs

[Rogue136198]

Summary

Improve Notification GIFs in the following ways:

• [x] Scale to a fixed width with height based on the original aspect ratio
• [x] Decrease the time between frame generation from 2 seconds to 0.75 seconds
• [x] Implement dynamic frame count to output as many frames as possible
• [x] Increase frame playback to 4 FPS (0.25 seconds) in notification shade

Overall this should result in smoother notification animations thus improving the UX.

The goal is to have notification GIFs feel more fluid (comparable to nest cameras)

Notifications will playback at 3x realtime

Comparing a few cameras:

Original Resolution	Aspect Ratio	Number of Output Frames	Notification Real-time Duration	Notification Playback Duration
1600x1200	1.33 (4:3)	14	10.5s	3.5s
1920x1080	1.77 (16:9)	19	14.25s	4.75s
2560x1920	1.33 (4:3)	14	10.5s	3.5s

NOTE Due to a bug in Android 13 the size of the notification is cut in half. Once bug is resolved in Android this can be updated in the app.

Screenshots

Link to pull request in Documentation repository

Documentation: home-assistant/companion.home-assistant#

Any other notes

[homeassistant]

Hi @Rogue136198,

It seems you haven't yet signed a CLA. Please do so here.

Once you do that we will be able to review and accept this pull request.

Thanks!

[Rogue136198]

It would appear there is a limitation on size(?) for videos in notifications. Currently unclear if this is an android or HA limitation. I did note that this issue is present on the mainline app but increasing the frames of the generated GIF made it more prevalent.

[Rogue136198]

This has been an interesting learning experience. I now understand why certain choices were made.

I think I am going to need to page @NickM-27 for assistance/guidance on how or even if I should proceed with this. Basically I've increased the framerate and number of frames but now I've run into the same situation as found in #2353

[NickM-27]

Yes, android 12 sets a max limit that is 2.5x the number of pixels in the screen in mb. I chose the lower rate to make the odds higher that more parts of the video is included.

[Rogue136198]

Interesting, so If I took the same notification on a 1080p phone vs a 1440p phone there is a chance it would play on the later but not the former?

I wonder how google has made notifications seem fluid with nest cameras. Probably has to do with a. first party integration and b. a known resolution on their cameras.

Have you ever considered changing the compression logic in such a way where video is compressed down to a fixed resolution while preserving aspect ratio? With this you would have a more predictable resolution. I've been trying to figure out the math/code required to implement this. I figure it should be possible to get at least 10 frames using this method.

[NickM-27]

Interesting, so If I took the same notification on a 1080p phone vs a 1440p phone there is a chance it would play on the later but not the former?

Correct

I wonder how google has made notifications seem fluid with nest cameras. Probably has to do with a. first party integration and b. a known resolution on their cameras.

They use a solution very similar to this one. Main thing is the video is spliced before it's put into the media metadata retriever.

Have you ever considered changing the compression logic in such a way where video is compressed down to a fixed resolution while preserving aspect ratio?

Could be done, the existing just quarters it I believe. I don't think it'd be too difficult with the current implementation to have a set width and calculate the respective height.

[NickM-27]

Makes sense to me

[Rogue136198]

Makes sense to me

Lets see how well it works.

Also, if you don't mind, could you explain what this means?

the video is spliced before it's put into the media metadata retriever.

I don't know what you mean the video is spliced.

[NickM-27]

Google takes their standard mp4 recording and lowers the frame rate / resolution before it's sent through this similar logic on Android

[Rogue136198]

PR should be in it's final form now. Ends up being the only change is increasing the frame interval from 2 seconds to 1 second. In my testing this has resulted in smoother animations. This could be subjective so others should probably test before this is merged.

I ended up dropping the ratio based scaling and reverted back to 1/4 scaling because of an odd bug. When using ratio scaling it would cut off the top and bottom of the image in the notification. I have no idea why.

Additionally I was trying to see if there was a way to compress the frames (e.g. JPEG/WEBP) in order to have more overall frames while staying within the size limit. I was unable to figure out how to do this and I'm not even sure if its possible.

[NickM-27]

I think there's still something that could be done to improve the sizing. Maybe instead of just dividing by 4, the division factor could be calculated based on the original size

[Rogue136198]

I think there's still something that could be done to improve the sizing. Maybe instead of just dividing by 4, the division factor could be calculated based on the original size

That's why this is still draft, I'm not satisfied with it yet.

You have me thinking. Basically take the original width and compute a division factor rounded to the nearest whole number then divide by that. e.g. 1080p would have a division factor of 4 and 4K would have a division factor of 8. Interesting Idea.

While I try that out, would you have any idea why my original plan resulted in the image being cut off? This was the code:

val ratio = width / height
val newHeight = 480 / ratio
val newWidth = 480

[NickM-27]

While I try that out, would you have any idea why my original plan resulted in the image being cut off? This was the code:

val ratio = width / height

val newHeight = 480 / ratio

val newWidth = 480

Maybe 480 was too small? I think the math makes sense, the image views have a set size / cropping so perhaps that small size led to that. I'll have to refresh on the image views for that

[Rogue136198]

While I try that out, would you have any idea why my original plan resulted in the image being cut off? This was the code:

val ratio = width / height

val newHeight = 480 / ratio

val newWidth = 480

Maybe 480 was too small? I think the math makes sense, the image views have a set size / cropping so perhaps that small size led to that. I'll have to refresh on the image views for that

Figured it out! Turns out I needed to change the Integers to Floats before dividing them. Without doing this it was causing the ratio to be rounded to the nearest whole number. Noob mistake but I'm learning.

[NickM-27]

Oh yeah that would do it

[Rogue136198]

Increasing the number of frames to 12 with frames generated every 0.5 seconds.

In my testing this results in a smooth animation with minimal cases of the video being too large resulting in the animation not being added to the notification. In my testing this only happened once in about 100 notifications.

[NickM-27]

Nice!

[Rogue136198]

On which device did you test this? I'm unable to get the notification to display on an emulator (Nexus 5X preset in Android Studio, on Android 12) and a Pixel 4a with a 1.7MB 240p mp4 test video.

I am testing against a Pixel 5 emulator and my own Pixel 4a 5G

2022-07-16 23:53:48.532 1764-4712/? W/NotificationService: Removed too large RemoteViews (6220800 bytes) on pkg: io.homeassistant.companion.android.debug tag: null id: 151044254

Not sure why the entire view is so much larger than the original video, but I've tested it with two other, similar videos and the same thing happens. The current limit seems to be 5MB, not screen size dependent (although it could be overriden on individual devices, this was just wat came up in a quick search). Perhaps this information could be used to determine the size of the RemoteViews, and remove frames if necessary?

I also do not know why the view is larger than the original, for now I am going to figure its because you were using a video smaller than 480p? For the 5MB limit, thank you for finding that, I had a hunch that it was not screen size dependant but I had no information to confirm that. I like the idea of determining the RemoteViews limit and dynamically adjusting the frame count. If you (or anyone) has any advice on how to go about implementing that I'd be up to giving it a shot.

Also another suggestion: the ViewFlipper in the remote view wasn't updated and is still flipping every 0.4s, with every frame now representing 0.5s this is very close to original speed. Would it be a good idea to also decrease it to retain the faster speed?

Yes, I like this. Now I just need to figure out where I change this...

[NickM-27]

This is what the android docs said last I read them / worked on this feature:

The total Bitmap memory used by the RemoteViews object cannot exceed that required to fill the screen 1.5 times, ie. (screen width x screen height x 4 x 1.5) bytes.

https://stackoverflow.com/questions/13494898/remoteviews-for-widget-update-exceeds-max-bitmap-memory-usage-error

Edit: seems that particular limitation is for a different class https://android.googlesource.com/platform/frameworks/base/+/27f592d/core/java/android/appwidget/AppWidgetManager.java

[Rogue136198]

I've gone ahead and implemented the logic to prevent scaling up if the initial video width is less than 480px. I hope this solves the issue you were running into @jpelgrom

Additionally I've set the ViewFlipper so the GIF should playback at 2x real-time. Personally I'm a fan of real-time playback with this many frames but I'd like to hear each of your opinions on it.

[NickM-27]

Personally, I don't think real-time playback makes sense, that's for watching the whole clip. I want to quickly digest what happened from the notification.

Worst case scenario an option for this through a different parameter could be added in a separate PR

[jpelgrom]

I've gone ahead and implemented the logic to prevent scaling up if the initial video width is less than 480px. I hope this solves the issue you were running into @jpelgrom

It seems to be working now with the first test video I linked! There's still a warning in Logcat about a too large RemoteViews but that'll probably continue to happen until Android natively supports adding video.

2022-07-17 14:24:43.786 1764-4966/? W/NotificationService: RemoteViews too large on pkg: io.homeassistant.companion.android.debug tag: null id: 203300198 this might be stripped in a future release

If I add another video that is a little larger though, I get the same error as I posted in my previous message, with the same amount of bytes (6220800). It makes sense when considering that a Bitmap is in-memory, no compression added, but when we're both using Google devices with the same resolution to test you'd expect it to work or fail in the same places...

Using frames.sumOf { frame -> frame.allocationByteCount } I get exactly half the amount of bytes of the error message (3110400), ~so I guess this has something to do with display density?.~

Edit: it doesn't seem to be related to display density as the provided Bitmap objects already have the same density as the screen. I'm not entirely sure how to proceed, whether my case is the exception or something more users will encounter. @Rogue136198 could you perhaps check what total size is for you for the Bitmaps using a 16:9 video that is >=480px high and long enough to get 12 frames? Maybe temporarily increase the number of frames to see if you also get an error for 2x the size?

I did manage to get it to work with larger files by saving the bitmaps and adding them using an URI instead, even with tons more frames (62! Limited by the video length and my patience for the decoder. Bitmap objects size 16.8MB, on disk 1.3MB due to compression), maybe some limits don't apply in that case? That would require a lot more changes and we still don't know what the limit is.

Additionally I've set the ViewFlipper so the GIF should playback at 2x real-time. Personally I'm a fan of real-time playback with this many frames but I'd like to hear each of your opinions on it.

I don't really have a preference, I mentioned it to make sure keeping the same flipper speed was intentional. However if the same flipper speed is used the framerate of the notification won't change like the title suggests 😉

[Rogue136198]

Personally, I don't think real-time playback makes sense, that's for watching the whole clip. I want to quickly digest what happened from the notification.

I'm leaning towards one frame every 0.75 seconds with playback of a frame every 0.25 seconds resulting in a 3x real-time playback. The result is 9 seconds of action played back in 3 seconds. I have this running right now and it looks pretty good to me. P.S. I've been testing live against my front door security came so I get the randomness of traffic passing by.

Worst case scenario an option for this through a different parameter could be added in a separate PR

I like this idea but I also agree, separate PR

It seems to be working now with the first test video I linked! There's still a warning in Logcat about a too large RemoteViews but that'll probably continue to happen until Android natively supports adding video.

2022-07-17 14:24:43.786 1764-4966/? W/NotificationService: RemoteViews too large on pkg: io.homeassistant.companion.android.debug tag: null id: 203300198 this might be stripped in a future release

Excellent. I agree about the warning, I've seen it and have been ignoring it.

If I add another video that is a little larger though, I get the same error as I posted in my previous message, with the same amount of bytes (6220800). It makes sense when considering that a Bitmap is in-memory, no compression added, but when we're both using Google devices with the same resolution to test you'd expect it to work or fail in the same places...

Using frames.sumOf { frame -> frame.allocationByteCount } I get exactly half the amount of bytes of the error message (3110400), so I guess this has something to do with display density? I'm doing some additonal tests and will update this comment later.

This is admittedly confusing me. Granted, its very strange that you are getting exactly half.

Additionally I've set the ViewFlipper so the GIF should playback at 2x real-time. Personally I'm a fan of real-time playback with this many frames but I'd like to hear each of your opinions on it.

I don't really have a preference, I mentioned it to make sure keeping the same flipper speed was intentional. However if the same flipper speed is used the framerate of the notification won't change like the title suggests 😉

Ah, makes sense. When I started this I was less looking to change the framerate and more looking to change the number of frames and the speed at which they are grabbed. So less so the displayed framerate and more so the extracted framerate.

[Rogue136198]

Edit: it doesn't seem to be related to display density as the provided Bitmap objects already have the same density as the screen. I'm not entirely sure how to proceed, whether my case is the exception or something more users will encounter.

I have been experimenting. I think it may have something to do with source bitrate, framerate or aspect ratio. The numbers only make sense for aspect ratio. When testing with my doorbell or test camera I cannot get as many frames as when testing with an episode of ATLA.

Here are the results and info on the source files.

frames / resolution / aspect ratio / bitrate / fps / file size
Doorbell: 14 Frames / 1600x1300 / 1.23 / 294kbps / 14.91fps / 434KB
ATLA: 19 Frames / 1920x1080 / 1.77 / 10598kbps / 23.98fps / 1.77GB
Test Cam (1): 14 Frames / 2560x1920 / 1.33 / 8399kbps / 30.05fps / 50MB
Test Cam (2): 19 Frames / 2650x1440 / 1.77 / 8493kbps / 30.05fps / 52.3MB

As you can see there appears to be a strong coloration between aspect ratio and max number of frames. For Test Cam 1 vs 2 this is the same camera just the recording resolution is changed. If I go above the frame counts for each file the GIF will not load in the notification. I don't think this is something exceptional, rather I think this is something many could encounter based on my testing.

@Rogue136198 could you perhaps check what total size is for you for the Bitmaps using a 16:9 video that is >=480px high and long enough to get 12 frames? Maybe temporarily increase the number of frames to see if you also get an error for 2x the size?

I'll admit, I don't know how to check this. I am very much a novice at this. Can you explain where I need to look or what I need to do?

I did manage to get it to work with larger files by saving the bitmaps and adding them using an URI instead, even with tons more frames (62! Limited by the video length and my patience for the decoder. Bitmap objects size 16.8MB, on disk 1.3MB due to compression), maybe some limits don't apply in that case? That would require a lot more changes and we still don't know what the limit is.

Can you explain this further? I think what you are saying is you are saving the notification video to internal then loading it to the notification from there?

[jpelgrom]

As you can see there appears to be a strong coloration between aspect ratio and max number of frames.

Using my earlier observations, 1 Bitmap in a 16:9 ratio = 259200 bytes. 259200 * 19 = 4924800, really close to the 5MB limit and one additonal frame would put you over it! Still not entirely sure why you're able to get 19 frames for a 16:9 video, whereas I can't get 12 to work and Android seems to double the size when creating the notification. Can confirm switching to 9 frames (19/2 rounded down) does work for me.

@Rogue136198 could you perhaps check what total size is for you for the Bitmaps using a 16:9 video that is >=480px high and long enough to get 12 frames? Maybe temporarily increase the number of frames to see if you also get an error for 2x the size?

I'll admit, I don't know how to check this. I am very much a novice at this. Can you explain where I need to look or what I need to do?

So what I did is take a random 16:9 video sample that is at least 480p, for example this one. Send that URL as the video in a notification.

Add the code to count the size (frames.sumOf { frame -> frame.allocationByteCount }) somewhere after the check that you received frames, for example after line 1290, and log the value.

As mentioned, for 16:9 videos I always seem to get 3110400 bytes as the value.

I did manage to get it to work with larger files by saving the bitmaps and adding them using an URI instead, even with tons more frames (62! Limited by the video length and my patience for the decoder. Bitmap objects size 16.8MB, on disk 1.3MB due to compression), maybe some limits don't apply in that case? That would require a lot more changes and we still don't know what the limit is.

Can you explain this further? I think what you are saying is you are saving the notification video to internal then loading it to the notification from there?

Not saving the video, but saving the frames and passing the file URI to Android instead of the entire image. Because you're passing an URI the limits don't seem to apply, Android will handle loading the image.

After getting the frames, I saved them something like this:

Code

// app/src/main/java/io/homeassistant/companion/android/notifications/MessagingManager.kt
// the handleVideo function was modified to include the message ID
frames.forEachIndexed { index, frame ->
    val file = File(context.applicationContext.cacheDir.absolutePath, "notifications/flipper/$messageId-$index.jpg")
    try {
        if (!file.exists()) {
            file.parentFile?.mkdirs()
            file.createNewFile()
        }
        FileOutputStream(file).use { output ->
            frame.compress(Bitmap.CompressFormat.JPEG, 90, output)
        }
    } catch (e: IOException) {
        e.printStackTrace()
    }
}

Then, in the notification instead of adding the Bitmap I added the URI:

Code

// app/src/main/res/xml/provider_paths.xml, to allow the system to open the image
<cache-path name="temporary_files" path="notifications/"/>

// app/src/main/java/io/homeassistant/companion/android/notifications/MessagingManager.kt
// the handleVideo function was modified to include the message ID, and added this code after if (frames.isNotEmpty()) {
frames.forEachIndexed { index, frame ->
    remoteViewFlipper.addView(
        R.id.frame_flipper,
        RemoteViews(context.packageName, R.layout.view_single_frame).apply {
            val file = File(context.applicationContext.cacheDir.absolutePath, "notifications/flipper/$messageId-$index.jpg")
            val uri = FileProvider.getUriForFile(
                context,
                "${BuildConfig.APPLICATION_ID}.provider",
                file
            )
            context.grantUriPermission("com.android.systemui", uri, Intent.FLAG_GRANT_READ_URI_PERMISSION)
            setImageViewUri(
                R.id.frame,
                uri
            )
        }
    )
    frame.recycle()
}

[Rogue136198]

As you can see there appears to be a strong coloration between aspect ratio and max number of frames.

Using my earlier observations, 1 Bitmap in a 16:9 ratio = 259200 bytes. 259200 * 19 = 4924800, really close to the 5MB limit and one additional frame would put you over it! Still not entirely sure why you're able to get 19 frames for a 16:9 video, whereas I can't get 12 to work and Android seems to double the size when creating the notification. Can confirm switching to 9 frames (19/2 rounded down) does work for me.

This is strange. Which device are you seeing this on? Nexus 5X? Pixel 4a? I want to see if I can replicate it in an emulator.

Add the code to count the size (frames.sumOf { frame -> frame.allocationByteCount }) somewhere after the check that you received frames, for example after line 1290, and log the value.

Thank You for this, I now have the bitmap size outputting to the log. I also now am starting to see how I can implement dynamic frame generation up to the 5MB limit. Time to learn how Kotlin while loops work.

As mentioned, for 16:9 videos I always seem to get 3110400 bytes as the value.

I am seeing the exact same when I am set to 12 frames also only for 16:9 video.

I did manage to get it to work with larger files by saving the bitmaps and adding them using an URI instead, even with tons more frames (62! Limited by the video length and my patience for the decoder. Bitmap objects size 16.8MB, on disk 1.3MB due to compression), maybe some limits don't apply in that case? That would require a lot more changes and we still don't know what the limit is.

Can you explain this further? I think what you are saying is you are saving the notification video to internal then loading it to the notification from there?

Not saving the video, but saving the frames and passing the file URI to Android instead of the entire image. Because you're passing an URI the limits don't seem to apply, Android will handle loading the image.

Okay, yes the frames not the video, that was what I was thinking but not what I wrote. lol

This is a very cool idea but I think it surpasses what I'm trying to achieve in this PR.

[jpelgrom]

This is strange. Which device are you seeing this on? Nexus 5X? Pixel 4a? I want to see if I can replicate it in an emulator.

I just updated my Android 12 emulator image (API 31, Nexus 5X preset) and now it's no longer happening.

Still happening on my Pixel 4a with Android 13 though, real device and emulator image, both up-to-date. Maybe it's an Android bug that has been patched in stable but not yet in the beta, but 13 is coming out in a month so not sure...

[Rogue136198]

This is strange. Which device are you seeing this on? Nexus 5X? Pixel 4a? I want to see if I can replicate it in an emulator.

I just updated my Android 12 emulator image (API 31, Nexus 5X preset) and now it's no longer happening.

Still happening on my Pixel 4a with Android 13 though, real device and emulator image, both up-to-date. Maybe it's an Android bug that has been patched in stable but not yet in the beta, but 13 is coming out in a month so not sure...

This is exactly my experience. There is either a bug or a new "feature"

Worth noting that the bitmap size is still only 2592000 for me when using 10 frames. 2332800 when using 9 frames. Wonder if the limit was dropped to 2.5MB?

[Rogue136198]

I've implemented dynamic number of frames with a max size of 2.5MB. I would like to change this to 5MB before merging but I think we need some form of confirmation from Google if this is an intentional change or not. EDIT: I now see that this is not intentional. limit is still set at 5MB but the GIF is double the size on A13

Should we open an issue in the Android Issue tracker? If so is there a benefit to this being opened officially by the HA team?

Other than changing 2.5MB to 5MB I do not think there are any other changes to be made to this other than what is required based on review.

[jpelgrom]

I've implemented dynamic number of frames

Great work!

Should we open an issue in the Android Issue tracker? If so is there a benefit to this being opened officially by the HA team?

I don't believe it needs to be done by a HA team member, Google doesn't treat it differently. They will ask for a minimal sample instead of the entire HA app to reproduce the issue though, so I've made one and opened an issue: https://issuetracker.google.com/issues/239945905. Let's hope we get a confirmation soon 🙂