scrcpy
scrcpy copied to clipboard
Genymobile
Half of frames not rendered due to inconsistent frame generation time
Environment
- OS: Windows 11
- scrcpy version: 1.25
- installation method: Windows release
- device model: Samsung S9, Xiaomi Mi 11
- Android version: 10, 12
Describe the bug
Recently I added a skipped frame counter to my Web implementation of Scrcpy client. Because rendering images on Web is slow, I have VSync on to avoid unnecessary works.
Then I found that the number of skipped frames is insanely high. On my Mi 11 with a 120FPS display and my PC monitor running at 144Hz, usually only about 70 frames are rendered, and the other 50 skipped.
In my further investigation, I found that the time those frames arrive are inconsistent. It's almost like every two frames are sent together. Here is part of my log:
frame interval: 2.4600000381469727ms
frame interval: 14.274999976158142ms
frame interval: 2.175000011920929ms
frame interval: 15.355000019073486ms
frame interval: 1.48499995470047ms
frame interval: 14.01500004529953ms
frame interval: 1.399999976158142ms
frame interval: 15.0450000166893ms
So, I also tested with the official client, the results are same. I modified server code to output time took for each frame to encode, video output is also completely disabled to minimize interference.
diff --git a/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java b/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java
index e95896d3..90e48d0c 100644
--- a/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java
+++ b/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java
@@ -9,6 +9,7 @@ import android.media.MediaCodecList;
import android.media.MediaFormat;
import android.os.Build;
import android.os.IBinder;
+import android.os.SystemClock;
import android.view.Surface;
import java.io.FileDescriptor;
@@ -42,6 +43,7 @@ public class ScreenEncoder implements Device.RotationListener {
private final boolean downsizeOnError;
private long ptsOrigin;
+ private long prevTime = 0;
private boolean firstFrameSent;
public ScreenEncoder(boolean sendFrameMeta, int bitRate, int maxFps, List<CodecOption> codecOptions, String encoderName,
@@ -146,6 +148,7 @@ public class ScreenEncoder implements Device.RotationListener {
private boolean encode(MediaCodec codec, FileDescriptor fd) throws IOException {
boolean eof = false;
+ prevTime = SystemClock.uptimeMillis();
MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
while (!consumeRotationChange() && !eof) {
@@ -157,17 +160,21 @@ public class ScreenEncoder implements Device.RotationListener {
break;
}
if (outputBufferId >= 0) {
- ByteBuffer codecBuffer = codec.getOutputBuffer(outputBufferId);
+ long now = SystemClock.uptimeMillis();
+ Ln.i("Frame interval: " + String.valueOf(now - prevTime));
+ prevTime = now;
- if (sendFrameMeta) {
- writeFrameMeta(fd, bufferInfo, codecBuffer.remaining());
- }
+ // ByteBuffer codecBuffer = codec.getOutputBuffer(outputBufferId);
- IO.writeFully(fd, codecBuffer);
- if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) == 0) {
- // If this is not a config packet, then it contains a frame
- firstFrameSent = true;
- }
+ // if (sendFrameMeta) {
+ // writeFrameMeta(fd, bufferInfo, codecBuffer.remaining());
+ // }
+
+ // // IO.writeFully(fd, codecBuffer);
+ // if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) == 0) {
+ // // If this is not a config packet, then it contains a frame
+ // firstFrameSent = true;
+ // }
}
} finally {
if (outputBufferId >= 0) {
@@ -198,7 +205,7 @@ public class ScreenEncoder implements Device.RotationListener {
headerBuffer.putLong(pts);
headerBuffer.putInt(packetSize);
headerBuffer.flip();
- IO.writeFully(fd, headerBuffer);
+ // IO.writeFully(fd, headerBuffer);
}
private static MediaCodecInfo[] listEncoders() {
And here is the output using the same size and bit rate with the default value of my Web client:
> ./scrcpy -m 1080 -b 4m --verbosity info
[...]
[server] INFO: Frame interval: 1
[server] INFO: Frame interval: 15
[server] INFO: Frame interval: 1
[server] INFO: Frame interval: 15
[server] INFO: Frame interval: 2
[server] INFO: Frame interval: 16
[server] INFO: Frame interval: 1
[server] INFO: Frame interval: 15
[server] INFO: Frame interval: 1
[server] INFO: Frame interval: 16
[...]
When --print-fps is used, Scrcpy won't report them as skipped frames, because it doesn't use VSync, so it doesn't care if the frame is truly displayed. Here I captured the original video stream with scrcpy -r (left) and what Scrcpy shows with OBS (right). the device (a Samsung S9), my PC monitor, and OBS are all set to 60FPS, then the video was slow down to 2FPS. It's pretty clear that while the left video moves every frame, the right video moves every two frames, or sometime one, sometime three.
https://user-images.githubusercontent.com/1330321/212981556-b176830f-f7c6-4c7d-95a0-acce2b56bf97.mp4
Thank you for your report.
When
--print-fpsis used, Scrcpy won't report them as skipped frames, because it doesn't use VSync, so it doesn't care if the frame is truly displayed.
That's true, but displaying all the frames with such dequeue timings would increase latency (it would consist in delaying 1 frame out of two). Scrcpy always displays the latest available as soon as possible. IMO the problem is the timings at which the frames are produced by MediaCodec.
(Btw, you should probably measure with System.nanoTime() for better precision.)
Out of curiosity, could you test:
diff --git a/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java b/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java
index e95896d3..97a27fbc 100644
--- a/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java
+++ b/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java
@@ -249,7 +249,7 @@ public class ScreenEncoder implements Device.RotationListener {
format.setString(MediaFormat.KEY_MIME, MediaFormat.MIMETYPE_VIDEO_AVC);
format.setInteger(MediaFormat.KEY_BIT_RATE, bitRate);
// must be present to configure the encoder, but does not impact the actual frame rate, which is variable
- format.setInteger(MediaFormat.KEY_FRAME_RATE, 60);
+ format.setInteger(MediaFormat.KEY_FRAME_RATE, 120);
format.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface);
format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, DEFAULT_I_FRAME_INTERVAL);
// display the very first frame, and recover from bad quality when no new frames
My description was not clear, this issue also happens on 60Hz devices like my Samsung S9. I tried your proposed change anyway but doesn't help. Looks like capture size, bit rate, and codec options all don't matter.
I'm playing a 60FPS video on my phone, and enabled video output:
> ./scrcpy -m 480 -b 1m --verbosity info --codec-options profile=2,level=4096
scrcpy 1.25 <https://github.com/Genymobile/scrcpy>
D:\dev\android\scrcpy\dist\scrcpy-win64-v1.25-6-g268e9eab\... file pushed, 0 skipped. 58.4 MB/s (42175 bytes in 0.001s)
[server] INFO: Device: samsung SM-G9600 (Android 10)
INFO: Renderer: direct3d
INFO: Initial texture: 480x232
[server] INFO: Frame interval: 100
[server] INFO: Frame interval: 7
[server] INFO: Frame interval: 3
[server] INFO: Frame interval: 3
[server] INFO: Frame interval: 5
[server] INFO: Frame interval: 2
[server] INFO: Frame interval: 1
[server] INFO: Frame interval: 1
[server] INFO: Frame interval: 0
[server] INFO: Frame interval: 1
[server] INFO: Frame interval: 1
[server] INFO: Frame interval: 1
[server] INFO: Frame interval: 0
[server] INFO: Frame interval: 2
[server] INFO: Frame interval: 24
[server] INFO: Frame interval: 2
[server] INFO: Frame interval: 33
[server] INFO: Frame interval: 2
[server] INFO: Frame interval: 34
[server] INFO: Frame interval: 3
[server] INFO: Frame interval: 27
[server] INFO: Frame interval: 2
[server] INFO: Frame interval: 35
[server] INFO: Frame interval: 2
[server] INFO: Frame interval: 29
[server] INFO: Frame interval: 2
[server] INFO: Frame interval: 29
[server] INFO: Frame interval: 2
[server] INFO: Frame interval: 35
[server] INFO: Frame interval: 2
[server] INFO: Frame interval: 33
[server] INFO: Frame interval: 2
[server] INFO: Frame interval: 28
[server] INFO: Frame interval: 2
[server] INFO: Frame interval: 30
[server] INFO: Frame interval: 3
[server] INFO: Frame interval: 32
[server] INFO: Frame interval: 3
[server] INFO: Frame interval: 31
[server] INFO: Frame interval: 3
[server] INFO: Frame interval: 30
I can reproduce the observation that MediaCodec does not produce frames regularly (although it's not as extreme as what you observe). Here is a capture playing a 60fps video:
[server] INFO: Frame interval: 15
[server] INFO: Frame interval: 10
[server] INFO: Frame interval: 23
[server] INFO: Frame interval: 10
[server] INFO: Frame interval: 24
[server] INFO: Frame interval: 12
[server] INFO: Frame interval: 17
[server] INFO: Frame interval: 11
[server] INFO: Frame interval: 23
[server] INFO: Frame interval: 9
[server] INFO: Frame interval: 26
[server] INFO: Frame interval: 8
[server] INFO: Frame interval: 22
[server] INFO: Frame interval: 10
[server] INFO: Frame interval: 23
[server] INFO: Frame interval: 10
[server] INFO: Frame interval: 26
[server] INFO: Frame interval: 9
[server] INFO: Frame interval: 20
[server] INFO: Frame interval: 11
[server] INFO: Frame interval: 22
[server] INFO: Frame interval: 11
[server] INFO: Frame interval: 27
[server] INFO: Frame interval: 6
[server] INFO: Frame interval: 19
[server] INFO: Frame interval: 9
[server] INFO: Frame interval: 21
[server] INFO: Frame interval: 11
[server] INFO: Frame interval: 26
[server] INFO: Frame interval: 12
[server] INFO: Frame interval: 20
[server] INFO: Frame interval: 9
[server] INFO: Frame interval: 24
[server] INFO: Frame interval: 9
[server] INFO: Frame interval: 27
[server] INFO: Frame interval: 6
[server] INFO: Frame interval: 23
[server] INFO: Frame interval: 9
[server] INFO: Frame interval: 23
[server] INFO: Frame interval: 10
[server] INFO: Frame interval: 26
[server] INFO: Frame interval: 10
[server] INFO: Frame interval: 19
[server] INFO: Frame interval: 10
[server] INFO: Frame interval: 24
[server] INFO: Frame interval: 9
[server] INFO: Frame interval: 25
[server] INFO: Frame interval: 9
[server] INFO: Frame interval: 21
[server] INFO: Frame interval: 10
This is annoying, and contributes to the global variation of delay between frames.
The question is: what should scrcpy do with such irregular input?
The policy is to minimize latency at all costs, so when it can draw, it always draws the latest available frame. This may be at the cost of framerate (like it your case), because the client could instead delay the more recent ones (as a side effect of vsync for example) to play all the frames.
If it is more important to preserve the original delay between frames (based on their PTS) than minimize delay, it is possible to add a buffering delay on the client side (see #2464):
scrcpy --display-buffer=30
If you have mandatory vsync on the client-side, you probably need 2 changes:
- add a small buffering to re-space your frames regularly (this adds a bit of latency), a bit similar to what
--display-bufferdoes; - estimate the vsync date to submit 2 or 3ms before the next vblank to limit the latency added by vsync (see this blog post for example).
That's true, but displaying all the frames with such dequeue timings would increase latency.
I don't mean Scrcpy should also enable VSync. Usually, FPS number is used to indicate the smoothness of a video/game/etc., but in this case, the number reported by Scrcpy is not the real number of frames the user can see, so the problem is with the FPS counter. In a non-realistic worst case, 60 frames all arrive at the same time, the user can only see 1 frame, but Scrcpy FPS counter still says 60.
The policy is to minimize latency at all costs.
This makes sense. Maybe I can try to render every frame first, and fallback to VSync if the device can't keep up. Or also adding the buffer option to let users choose.
so the problem is with the FPS counter
Yes, I agree.
But how could scrcpy know if the frame has been actually displayed or not?
What could be done to improve frame delivery without impacting latency is to add an option to halve the displayed framerate (but not the encoded framerate). This won't reduce bandwidth/CPU requirements, but it will effectively allow you to have one dropped frame without any visible hitch. This could prove useful on 120 Hz displays where 60 Hz is usually sufficient for remote usage, or for less stable connections (wifi).
I have the same problem on my web project, and the following output comes from the reception timer on the web decoder side:
