webrtc icon indicating copy to clipboard operation
webrtc copied to clipboard

Published 20 hours ago • verified publisher icon webrtc-sdk

Fix simulcast using hardware encoder on Android

Open graszka22 opened this issue 3 years ago • 1 comments

Hi! When using simulcast on Android with hardware encoder we noticed that all simulcast layers have the same resolution and switching between them is broken (video stops encoding when there is only one layer etc.). That's not only our problem, other people noticed it as well, there is a discussion here: https://bugs.chromium.org/p/webrtc/issues/detail?id=12328

We know what causes the problem. Some phones have weird hardware encoders that accept only resolutions aligned to 16x16. Webrtc developers added apply_alignment_to_all_simulcast_layers option to apply 16x16 alignment to all simulcast layers. There is a code in alignment_adjuster.cc that tries to change the scales to apply the alignment but for 16x16 alignment it will always set the scales to {1, 1, 1}. I added a description why it happens in the discussion linked above.

There are basically two solutions to this problem:

  • allow to customize kMaxAlignment constant. Right now it's hardcoded to 16. When kMaxAlignment = 64 the scales can be adjusted to {1, 2, 4}. However, this comes with a cost that the video can be cropped (even more than 64px) and aspect ratio of the video might be changed a bit. That's what we tried at first here https://webrtc-review.googlesource.com/c/src/+/277440 but the reviewer told us about cropping.
  • allow to disable apply_alignment_to_all_simulcast_layers, that's what the reviewer suggested to us and we think it would be a better idea. It fixes the simulcast issues and there will be no cropping. However right now apply_alignment_to_all_simulcast_layers is set in the cpp code and we can't disable it from java code. This PR changes that. It can break the encoding on some devices though. From the discussion here: https://bugs.chromium.org/p/chromium/issues/detail?id=1084702 apparently it affects only Pixel 3 and 3a with Android < 12. If that's a problem we can enable apply_alignment_to_all_simulcast_layers only on those affected devices.

Please let us know what do you think about those solutions. We're open for ideas on how to fix that. Thanks!

graszka22 avatar Oct 12 '22 14:10 graszka22

Thanks for the PR! We'll take a look.

davidliu avatar Oct 13 '22 02:10 davidliu

Seems like the apply_alignment_to_all_simulcast_layers solution seems to be more proper since it's fixed going forward from Android 12 (albeit slowly).

Would simply setting apply_alignment_to_all_simulcast_layers to true on the bad devices be enough to fix them? Will always bypassing the size check in HardwareVideoEncoder.java affect this?

davidliu avatar Nov 14 '22 17:11 davidliu

Would simply setting apply_alignment_to_all_simulcast_layers to true on the bad devices be enough to fix them? Will always bypassing the size check in HardwareVideoEncoder.java affect this?

Yes, the checks simply stops encoding if frame size is not aligned. I don't see ERR_SIZE handled anywhere in the code.

graszka22 avatar Nov 14 '22 20:11 graszka22

Ah, I think I ran into an old issue with this now:

Attempting to initEncode with odd dimensions will crash; this wasn't an issue before since alignment fixed this, but without alignment on simulcast layers, that becomes an issue again. I don't think this route will be viable without someway to at least aligning to even numbers.

I wonder if each simulcast layer can be aligned independently; that way so we don't have to deal with the cropping issues as well as not being limited to powers of 2 scales.

davidliu avatar Nov 16 '22 11:11 davidliu

I am wondering how other people use H264 hardware encoder as it clearly doesn't work as expected :slightly_frowning_face:

mickel8 avatar Nov 16 '22 11:11 mickel8

I tested this on my device some odd resolutions like 1279x719 or 1111x1111 and it works for me. Could you tell us:

  • what resolutions crash
  • what error do you get
  • on what device have you tested?

I wonder if each simulcast layer can be aligned independently

Some time ago I considered that too but it would be too many changes if it's possible at all.

graszka22 avatar Nov 16 '22 12:11 graszka22

Here, the issue is that the base layer gets aligned, but the scaled layers are scaled without any further alignment (since the previous math dictated that no further alignment was needed).

I have the base layer being 600x800, with a separate simulcast layer downscaled to 360x480 (0.6x). Since 600 isn't divisible by 16, it gets aligned to 592. The simulcast layer's width becomes 592*0.6 = 355.

I reproduced this issue on my Pixel 4, but the last time I saw this issue was around m97ish, and that pretty reliably crashed on all devices IIRC.

Crash:

11-16 22:30:04.190  1751  1976 W System.err: Caused by: java.lang.IllegalArgumentException
11-16 22:30:04.190  1751  1976 W System.err: 	at android.media.MediaCodec.native_configure(Native Method)
11-16 22:30:04.190  1751  1976 W System.err: 	at android.media.MediaCodec.configure(MediaCodec.java:2176)
11-16 22:30:04.190  1751  1976 W System.err: 	at android.media.MediaCodec.configure(MediaCodec.java:2092)
11-16 22:30:04.190  1751  1976 W System.err: 	at org.webrtc.MediaCodecWrapperFactoryImpl$MediaCodecWrapperImpl.configure(MediaCodecWrapperFactoryImpl.java:36)
11-16 22:30:04.190  1751  1976 W System.err: 	at org.webrtc.HardwareVideoEncoder.initEncodeInternal(HardwareVideoEncoder.java:268)
11-16 22:30:04.190  1751  1976 W System.err: 	at org.webrtc.HardwareVideoEncoder.initEncode(HardwareVideoEncoder.java:226)
11-16 22:30:04.190  1751  1976 W System.err: 	at io.livekit.android.webrtc.SimulcastVideoEncoderFactoryWrapper$StreamEncoderWrapper.initEncode$lambda-1(SimulcastVideoEncoderFactoryWrapper.kt:110)
11-16 22:30:04.190  1751  1976 W System.err: 	at io.livekit.android.webrtc.SimulcastVideoEncoderFactoryWrapper$StreamEncoderWrapper.$r8$lambda$-ywiG5jjBM1Bm1PVEOZ-93OY1Qg(Unknown Source:0)
11-16 22:30:04.190  1751  1976 W System.err: 	at io.livekit.android.webrtc.SimulcastVideoEncoderFactoryWrapper$StreamEncoderWrapper$$ExternalSyntheticLambda10.call(Unknown Source:6)
11-16 22:30:04.190  1751  1976 W System.err: 	at java.util.concurrent.FutureTask.run(FutureTask.java:264)
11-16 22:30:04.190  1751  1976 W System.err: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1137)
11-16 22:30:04.190  1751  1976 W System.err: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:637)
11-16 22:30:04.190  1751  1976 W System.err: 	at java.lang.Thread.run(Thread.java:1012)
11-16 22:30:04.191  1751  1976 E rtc     : #
11-16 22:30:04.191  1751  1976 E rtc     : # Fatal error in: gen/sdk/android/generated_metrics_jni/../../../../../../sdk/android/src/jni/jni_generator_helper.h, line 94
11-16 22:30:04.191  1751  1976 E rtc     : # last system error: 0
11-16 22:30:04.191  1751  1976 E rtc     : # Check failed: !env->ExceptionCheck()
11-16 22:30:04.191  1751  1976 E rtc     : # 
11-16 22:30:04.191  1751  1976 F libc    : Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 1976 (EncoderQueue - ), pid 1751 (d.composesample)
11-16 22:30:04.501  2042  2042 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
11-16 22:30:04.501  2042  2042 F DEBUG   : Build fingerprint: 'google/flame/flame:12/SP2A.220305.012/8177914:user/release-keys'
11-16 22:30:04.501  2042  2042 F DEBUG   : Revision: 'MP1.0'
11-16 22:30:04.501  2042  2042 F DEBUG   : ABI: 'arm64'
11-16 22:30:04.501  2042  2042 F DEBUG   : Timestamp: 2022-11-16 22:30:04.250535675+0900
11-16 22:30:04.501  2042  2042 F DEBUG   : Process uptime: 0s
11-16 22:30:04.501  2042  2042 F DEBUG   : Cmdline: io.livekit.android.composesample
11-16 22:30:04.501  2042  2042 F DEBUG   : pid: 1751, tid: 1976, name: EncoderQueue -  >>> io.livekit.android.composesample <<<
11-16 22:30:04.501  2042  2042 F DEBUG   : uid: 10620
11-16 22:30:04.501  2042  2042 F DEBUG   : signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
11-16 22:30:04.501  2042  2042 F DEBUG   :     x0  0000000000000000  x1  00000000000007b8  x2  0000000000000006  x3  0000007ba90b2300
11-16 22:30:04.501  2042  2042 F DEBUG   :     x4  71636d602e6a6372  x5  71636d602e6a6372  x6  71636d602e6a6372  x7  7f7f7f7f7f7f7f7f
11-16 22:30:04.501  2042  2042 F DEBUG   :     x8  00000000000000f0  x9  b1702745879ff250  x10 0000000000000000  x11 ffffff80fffffbdf
11-16 22:30:04.501  2042  2042 F DEBUG   :     x12 0000000000000001  x13 00000000000000c2  x14 0000007ba90b0cc0  x15 0000000000f1d692
11-16 22:30:04.501  2042  2042 F DEBUG   :     x16 0000007ef20fe050  x17 0000007ef20dadb0  x18 0000007ba1cce000  x19 00000000000006d7
11-16 22:30:04.501  2042  2042 F DEBUG   :     x20 00000000000007b8  x21 00000000ffffffff  x22 00000000000006d7  x23 0000000000000755
11-16 22:30:04.501  2042  2042 F DEBUG   :     x24 0000007ba90b9cb0  x25 0000007ba90b9cb0  x26 0000007ba90b9ff8  x27 0000000000104000
11-16 22:30:04.501  2042  2042 F DEBUG   :     x28 0000007ba8fb9000  x29 0000007ba90b2380
11-16 22:30:04.501  2042  2042 F DEBUG   :     lr  0000007ef208daa0  sp  0000007ba90b22e0  pc  0000007ef208dacc  pst 0000000000000000
11-16 22:30:04.501  2042  2042 F DEBUG   : backtrace:
11-16 22:30:04.501  2042  2042 F DEBUG   :       #00 pc 000000000004facc  /apex/com.android.runtime/lib64/bionic/libc.so (abort+164) (BuildId: cd7952cb40d1a2deca6420c2da7910be)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #01 pc 0000000001164770  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #02 pc 00000000011646c4  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #03 pc 00000000010e490c  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #04 pc 00000000010e46dc  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #05 pc 00000000018cf968  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #06 pc 00000000018ceed0  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #07 pc 00000000018cec18  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #08 pc 00000000018df258  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #09 pc 0000000001e5ed78  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #10 pc 0000000002353cd0  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #11 pc 000000000235b0a0  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #12 pc 00000000023596bc  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #13 pc 0000000002362a1c  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #14 pc 0000000002337a38  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #15 pc 0000000002338574  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #16 pc 0000000002338508  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #17 pc 000000000233816c  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #18 pc 0000000001e81fa4  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #19 pc 0000000001e91344  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #20 pc 0000000001e90930  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #21 pc 0000000001e83c08  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #22 pc 0000000001e83bac  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #23 pc 0000000001e83b80  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #24 pc 0000000001e83b54  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #25 pc 0000000001e83b28  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #26 pc 0000000000e5c824  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #27 pc 0000000000e5a75c  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #28 pc 00000000012dbdb4  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #29 pc 00000000012dbd68  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #30 pc 00000000012dbd3c  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #31 pc 00000000012dbd10  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #32 pc 00000000012dbce4  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #33 pc 0000000000e5c824  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #34 pc 0000000000e5a75c  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #35 pc 00000000012dba4c  /data/app/~~HEy0Fgh5SpforLdwW9gvZw==/io.livekit.android.composesample-S6IV6SCtNaCYlhj_82uzbg==/lib/arm64/libjingle_peerconnection_so.so (BuildId: 7e18ea959870567d)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #36 pc 00000000000b1810  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+264) (BuildId: cd7952cb40d1a2deca6420c2da7910be)
11-16 22:30:04.501  2042  2042 F DEBUG   :       #37 pc 00000000000512f0  /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) (BuildId: cd7952cb40d1a2deca6420c2da7910be)

davidliu avatar Nov 16 '22 13:11 davidliu

@davidliu thanks, I managed to reproduce it on my device too.

So, if the problem is that all layers aren't aligned to 2, then my idea is to set them to be aligned to 2, instead of 16 like now. Turns out it's already configurable on the Java side here: https://github.com/livekit/client-sdk-android/blob/main/livekit-android-sdk/src/main/java/io/livekit/android/webrtc/SimulcastVideoEncoderFactoryWrapper.kt#L193 Change: val future = executor.submit(Callable { return@Callable encoder.encoderInfo }) into: val future = executor.submit(Callable { return@Callable EncoderInfo(2, true) })

and all layers are aligned to 2 and with kMaxAlignment = 16 in alignment adjuster that's ok, in our case (standard resolutions and {1, 2, 4} scaling factors) there is no cropping and resolutions are scaled correctly. I tested also weird scaling factors like 1.78571 (to reproduce bug with odd resolutions) and it gets changed to 1.6 which is ok I guess (better 1.6 than 1 😄). If you don't change the code above then nothing changes anyway.

I updated the PR accordingly, now all it does is removing the crash if the resolutions are not aligned to 16x16.

graszka22 avatar Nov 17 '22 16:11 graszka22

Getting some weird outputs now:

For a base layer of 600x800, with scaleResolutionDowns of [1, 1.66667, 3.33333], the alignment adjuster changes the scales like so:

2022-11-18 01:39:08.331 30545-30680/io.livekit.android.composesample I/RTCModule: alignment_adjuster.cc: (line 44): scale_resolution_down_by 1 -> 1.08333
2022-11-18 01:39:08.331 30545-30680/io.livekit.android.composesample I/RTCModule: alignment_adjuster.cc: (line 44): scale_resolution_down_by 1.66667 -> 1.625
2022-11-18 01:39:08.331 30545-30680/io.livekit.android.composesample I/RTCModule: alignment_adjuster.cc: (line 44): scale_resolution_down_by 3.33333 -> 3.25

The 1 -> 1.08333 seems a little weird, though it still results in an even number for the dimens. The 1.625 results in an odd number though (600 / 1.625 = 369). I'll take a look into alignment adjuster to see what went wrong here.

davidliu avatar Nov 17 '22 16:11 davidliu

I think I understand what's going on here now; the issue is that no realignment seems to be done after the alignment is changed from the alignment adjuster. In the case above, the alignment changes to 13, with scales of [13/12, 13/8, 13/4]. This works as long as the resolution is a multiple of 13, as the scales will turn that multiple of 13 into multiples of 12, 8, 4 respectively, which are all divisible by the original required alignment of 2.

However, no alignment seems to be taking place, and the scales are applied directly to the original resolution, resulting in the odd number I was see. I wonder if there's something we need to do to trigger an alignment when needed.

@graszka22 would you happen to know where the alignment takes place?

davidliu avatar Nov 18 '22 16:11 davidliu

Ok, with a little more digging, I think I have a solution to this issue; in HardwareVideoEncoder, put back the width/height checks, but reduce REQUIRED_RESOLUTION_ALIGNMENT to 2 from 16. The returning of an error from initEncode seems important in getting that final alignment working properly (not exactly sure of the exact mechanism though).

With this in place (and EncoderInfo(2, true)), the streams get adjusted twice:

2022-11-19 22:51:44.049 18627-18761/io.livekit.android.composesample I/RTCModule: video_stream_encoder.cc: (line 1125): ReconfigureEncoder:
    Simulcast streams:
    0: 554x738 min_kbps: 134 target_kbps: 1500 max_kbps: 2000 max_fps: 30 max_qp: 56 num_tl: 1 active: true
    1: 369x492 min_kbps: 320 target_kbps: 320 max_kbps: 320 max_fps: 30 max_qp: 56 num_tl: 1 active: true

2022-11-19 22:51:44.221 18627-18761/io.livekit.android.composesample I/RTCModule: video_stream_encoder.cc: (line 1125): ReconfigureEncoder:
    Simulcast streams:
    0: 552x720 min_kbps: 128 target_kbps: 1500 max_kbps: 2000 max_fps: 30 max_qp: 56 num_tl: 1 active: true
    1: 368x480 min_kbps: 313 target_kbps: 320 max_kbps: 320 max_fps: 30 max_qp: 56 num_tl: 1 active: true

davidliu avatar Nov 19 '22 14:11 davidliu

Alternatively, changing the catch clause when configuring the codec in HardwareVideoEncoder.initEncodeInternal() from IllegalStateException to also handle IllegalArgumentException also works.

davidliu avatar Nov 20 '22 10:11 davidliu

@davidliu Agreed, I changed REQUIRED_RESOLUTION_ALIGNMENT to 2 and it works. I updated the PR accordingly.

graszka22 avatar Nov 21 '22 13:11 graszka22

Seems good. Any other changes you think this will need? Or ready to merge?

davidliu avatar Nov 21 '22 15:11 davidliu

It's ready to merge 👍

graszka22 avatar Nov 21 '22 15:11 graszka22

@davidliu Thank you 😄

graszka22 avatar Nov 21 '22 16:11 graszka22