During still frame of variable frame rate (VFR) video playback: OSD + Subtitles are not refreshed

[porg]

Reproduction

• Open a video with a variable framerate (see section "My Source Material & Use Case" below)
• Start playback
  • ✅ Audio and video are in sync
  • ✅ Video speed is fine
• As soon as playback is over a still frame (which can be held for a few frames or sometimes as long as 10secs easily)
  • ✅ Audio continues fine
  • ❌ But the timing / behavior of the following features is wrong then:

ON SCREEN DISPLAY (OSD): Playback-time stuck during still frame

In my ~/.config/mpv/mpv.conf I customized my OSD to display also frame numbers and FPS.

osd-status-msg=${playback-time/full} / ${duration} (${percent-pos}%)\nFrame: ${estimated-frame-number} / ${estimated-frame-count} — FPS: ${estimated-vf-fps} / ${container-fps}\n

• ✅ The $estimated-frame-number remains still during the still frame. That's correct obviously.
• ❌ The $duration and $percent-pos remain stuck.
  • That is conceptionally wrong, because the audio continues.
  • It seems the entire logic of this operates purely from the video (series of frames) and not from the audio.

SUBTITLES which start or end during still frame are not accounted for

• If a subtitle's inpoint happens before the still frame, and its outpoint happens during the still frame, the subtitle is falsely shown longer than intended. It is shown until the still frame ends (=when the next frame comes).
• If a subtitle's inpoint happens during a still frame, it is not shown from the inpoint as intended. Only as soon as the still frame ends (=the next frame comes), that subtitle starts to get shown.
• Under the circumstance that a subtitle occurs solely during the time of a still frame, it is never shown at all!
  • Practical example: This can very easily happen during a screencast: You had some scene with visual dynamics. Then no visual change occurs on screen (=still frame) and you the narrator explain something of what has happened or what you will do next. The subtitle for this audio portion will never be shown!

My Source Material & Use Case

• I often do screen recordings to which I talk (software demos, etc)
• These screen recordings consist of
  • Dynamic scenes: Windows open/close, cursor moves, content scrolls/blinks, …
  • Static scenes: Nothing happening visually (= still frame), only talking/bg-music
• I use the ffmpeg video filter -vf mpdecimate to create variable frame rate videos from this.
  • This reduces file size significantly (!) while maintaining the exact same quality (given the CRF you use)
  • Technical explanation: An I-frame is held for 10 seconds. This is significantly larger than 10 seconds long having, each second 1 I-frame plus 59 P-frames merely stating "nothing changed", so 10 I-frames + 590 P-frames.
  • ffmpeg -i screen-recording.mov -movflags faststart -c:a copy -c:v libx264 -preset veryslow -vf mpdecimate -vsync vfr -crf 18 screen-recording.mp4
  • To these videos I also add subtitle files.

Environment

• mpv 0.35.0, installed via brew package manager
• macOS 11.7.8

[Dudemanguy]

Can you try git master? I believe https://github.com/mpv-player/mpv/commit/296d40dc6f38401085d005bb4627f8afff46b041 fixed what you are describing; well at least the subtitle part of that.

[porg]

https://mpv.io/installation/#:~:text=macOS

• Lists "macOS nightly builds by jnozsc" — The last one being 20200928 … 🙁
• Could you be so kind and create a build for me? (64bit Intel, macOS 11 Big Sur)
• Would test it then and give you feedback.

[Dudemanguy]

Sorry I do not own a mac. You will have to compile it from source yourself or maybe use homebrew to fetch git head if it's possible.

[porg]

I will wait for the next stable build and give you feedback on it then.

[porg]

I managed to build mpv 0.35.0-531-gcf0373e15b (from master as of now) myself with these instructions (were for M1 chip, but procedure was the same on my IntelMac).

• ❌ Sadly still OSD and Subtitles do not refresh during a still frame.
• Regarding subtitles I now observed it in detail, and will update the issue description accordingly:
  • The already started but not yet shown subtitle is shown as soon as the next frame comes.
  • The already over but still shown subtitle is cleared as soon as the next frame comes.
  • The succeeding subtitle has no influence on this.
  • So this is a clear indication of how rendering works: Only on the new frame update OSD and subtitle display.
    • The logic should be reworked that all events that require visual updating (OSD, subtitle, chapters, whatever I may no think of yet) refresh when there's the need for them too. Not the frame being the "only master clock" and everything else visual only follows it.

[Dudemanguy]

I must have misunderstood what you're experiencing then. Maybe it has something to do with #10176 (separate thing).

[porg]

• #10176 has some similarities (to me as a layman):
  • Subtitle not rendered as it should.
  • But the raise condition there seems to be the pause/resume
  • Other than that it has not so much to do with what I observed
• I observed that mpv renders frame content + all metadata (subtitles, OSD, etc) only per each frame, and if you have a variable frame rate video with sections were there is a still frame for 10 secs, then nothing gets updated for 10 secs.
  • Per se is great efficiency wise: Not only reduced filesize, but also reduced playback effort == better performance.
  • BUT: mpv should find a way that also the inpoints and outpoints of subtitles request a redrawing. So if the subtitle itself lasts 5secs, then again nothing to do for further 5secs (unless another visual repainting event comes earlier than that).
  • And the OSD if on should repaint really every display frame (= all the time) even if video content and subbtitle content do not change. Because audio continues. So technically the timecode continues. Therefore the $duration and $percent-pos should be repainted too.

[Dudemanguy]

Okay thanks I understand this better I think. It's definitely specific to VFR and the other stuff was just vaguely similar but not actually related.

[porg]

Great that the problem is now better understood.

What's your estimate? Is this principally do-able? Or are there some big obstacles in that regard in the architecture of mpv?

[Dudemanguy]

I'm not really sure honestly. This is a bit out of my wheelhouse.

[porg]

Who in the mpv team could estimate the feasibility of this?

[po5]

The subtitle "issue" is completely unrelated to VFR, and the behavior should not be changed (it may not even be possible for mpv to make this change, I suspect this concept would be too deeply integrated in libass).
ASS subtitles (which all text subs get converted to internally) are expected to start and end in sync with video frames.
The format uses 2 decimal precision for timestamps, which is not enough to precisely point to the very start/end of a frame.
Video sync is very important so that subtitles do not bleed across scene changes, and allows for transform (\t) tags to animate in sync with the video. Therefore subtitles should appear and disappear in sync with video frames.
Making any change to how subtitles start and end will break millions of files.

A better solution would for you to not hold a frame for 10 seconds through timestamps, but keep the video CFR and insert "null" frames. Since you're re-encoding anyway, skip your decimation filter and allow a higher i-frame interval.

It is annoying that osd updates have the same limitation, as they're also subtitles.
How uosc deals with it is by telling users to use video-sync=display-resample. That may fix your issue.
As I understand it, it allows scripts to force a repaint of their own subtitle overlay mid-frame. I don't know how osd status messages are handled internally, it's possible that they only update each frame.

[Akemi]

did video-sync=display-resample work around this problem? to me this sounds like a general problem how we render and update.

a test video file would be nice too.

[porg]

I had already started with a meaningful test video plus test subtitle file, but not finished this back then.

Will post it here as soon as I get time, possibly will get to it over the new year holidays.

[Akemi]

@porg any progress on the test file?

[porg]

Great to read your re-inquiry. Motivates me to deliver the test material.

End of march is the next realistic chance that I will get to this. I put a reminder into my calendar.

[xaionaro]

Just for an archive is somebody else would hit the same issue:

I think I'm hitting another similar issue with the same root cause as this ticket.

For me it is important to get the precise position via the mpv's IPC socket. However, the position does not move during still frames (and then jumps), despite audio is still going:

{"data":0.667000,"request_id":0,"error":"success"}
{"data":0.768000,"request_id":0,"error":"success"}
{"data":0.868000,"request_id":0,"error":"success"}
{"data":0.934000,"request_id":0,"error":"success"}
{"data":0.934000,"request_id":0,"error":"success"}
{"data":0.934000,"request_id":0,"error":"success"}
{"data":0.934000,"request_id":0,"error":"success"}
{"data":0.934000,"request_id":0,"error":"success"}
{"data":0.934000,"request_id":0,"error":"success"}
{"data":0.934000,"request_id":0,"error":"success"}
{"data":0.934000,"request_id":0,"error":"success"}
{"data":0.934000,"request_id":0,"error":"success"}
{"data":2.002000,"request_id":0,"error":"success"}
{"data":2.069000,"request_id":0,"error":"success"}
{"data":2.169000,"request_id":0,"error":"success"}
{"data":2.269000,"request_id":0,"error":"success"}

--video-sync=display-resample did not help. But in my case there is a very easy workaround: just read audio-pts instead of time-pos.