AmusementClub
Feature Request: (RIFE) Please add support for rational frame rate changes (like vs-rife has)
Unfortunately multi is an int.
vs-rife replaced multi with factor_num, factor_den with version 3.0.0 to support rational frame rate changes.
see: https://github.com/HolyWu/vs-rife/releases
Unfortunately, the feature in https://github.com/AmusementClub/vs-mlrt/commit/2f6d5989a33833042a7a5f49ea0d22c0d30ec92d is not as safe as factor_num, factor_den in rife-ncnn-vk.
If I tried with mpv, it would used up memory and make my program frozen.
I cannot. I remembered holywu once pushed some commits too in his original repo to fix the similar issue. But the related issues are gone by deleting his repo.
Since we have used it successfully in encoding, without reliable reproduction we can not validate the claim.
This repository should contain the relevant commit. Do you mean this one?
Why do you think fps_num is better? It is equivalent to multi=target.fps / source.fps for CFR videos, and I don't think that implementation handles VFR videos.
No, i never talked about fps_num...
To make it clear, factor_num=5 factor_num_den=2 works fine in rife-ncnn-vk. multi=Fraction(5,2) makes the program frozen.
Then factor_num and factor_den, which is equivalent to multi=Fraction(factor_num, factor_den), isn't it?
multi=Fraction(5,2) makes the program frozen.
I can't reproduce that with vsedit or vs-preview.
I can't reproduce that with vsedit or vs-preview.
I knew, that's why I said But the related issues are gone by deleting his repo.
The more details explaination are only recorded in those deleted issue.
Then we can do nothing.
And we are using different implementation here compared to the original approach.
multi=Fraction(5,2)
I can confirm this, I tried with MPV and MPC, and both froze. MPC sometimes froze for ~3 minutes and then ran successfully but it froze most of the time.
but it froze most of the time
Could you provide more information? For example, does the freezing occur at the same frame? Is it related to scene change, and what if you disable scene change detection?
but it froze most of the time
Could you provide more information? For example, does the freezing occur at the same frame? Is it related to scene change, and what if you disable scene change detection?
It froze when opening any video file or when seeking(when it ran successfully). it's eating a lot of RAM while freezing
I disabled scene change detection, still froze.
Are you using SVP with "Duplicate frames removal" enabled?
By "opening any video file", do you mean it does not return any frame at all?
Does the frozen time depends on the number of frames of the video?
seems so, video > 20 minutes took forever to load.
That's a new information. What if you change your script to the following:
-
before:
output = vsmlrt.RIFE(xxx) -
after:
output = vsmlrt.RIFE(xxx).std.BlankClip(color=[0.5] * 3)
Does the playback still froze?
cut2 = RIFE(cut1, multi=Fraction(5,2), model=46, backend=BackendV2.TRT(force_fp16=True, num_streams=2, output_format=1, static_shape=True)).std.BlankClip(color=[0.5] * 3)
video still froze
Do you know Python? I want to know which line of code in the RIFE() function of vsmlrt.py blocks.
This can be checked by inserting a line return clip before each function calls (i.e. line 1147, 1164-1167, 1169, 1176-1179). With this change the original video is returned, but some insertions may not have immediate display.
I inserted return clip into: line 1146: video play normal w/o interpolation line 1163: froze about 5 seconds then started playing normally w/o interpolation line 1168: video froze
so these lines cause issues?
left_clip = core.akarin.PickFrames(clip, left_indices)
right_clip = core.akarin.PickFrames(clip, right_indices)
tp_clip = core.std.BlankClip(clip, format=gray_format, length=len(timepoints))
tp_clip = tp_clip.akarin.PropExpr(lambda: dict(_tp=timepoints)).akarin.Expr('x._tp')
line 1163: froze about 5 seconds then started playing normally w/o interpolation
This is alarming because it should not froze.
In the original program, what if you repeat line 1142-1162 https://github.com/AmusementClub/vs-mlrt/blob/2f6d5989a33833042a7a5f49ea0d22c0d30ec92d/scripts/vsmlrt.py#L1142-L1162 several times? Will each repetition increase the frozen time by 5 seconds?
Will each repetition increase the frozen time by 5 seconds?
yes, it freeze for a very long time. around 30~40 seconds
That's a very useful information.
What if you simply call len(timepoints) multiple times in the loop without assignment, does it affect performance?
Alternatively, insert the following piece of code
with open("num_frames.txt", "w") as f:
f.write(str(clip.num_frames))
This will create a file. What's the value in it?
called len(timepoints) 5 times
took 10 seconds to play a 90 seconds video
25s for 20 minutes video
What's the value in it?
10810800
10810800
Thanks, now I fully understand what happens.
Video playback is different from encoding in that the player will create a clip with a large number of frame, while for encoding the value is at most hundreds of thousands. This make the initialization much more demanding for vudeo playback.
I will start to fix the issue now.
input = VpsFilterSource.std.Trim(length=500000) #mpc input = video_in.std.Trim(length=500000) #mpv
this prevents the freezing forever, but still slow
the value 10000 is fast at least, still don't know about the side effects thought.
The effect of 10000 may be the player closed after playing 10000 frames?
Another side effect with frame interpolation in video players is that, when it jumps to another frame and returns to the previous frame, the player may renders differently.
How is the variable video_in defined?
How is the variable video_in defined?
https://github.com/mpv-player/mpv/blob/master/video/filter/vf_vapoursynth.c
the value 10000 is fast at least, still don't know about the side effects thought. The effect of 10000 may be the player closed after playing 10000 frames?
The player would destroy the vf_vapoursytn filter after 10000 frames. User have to re-enable the filter again to make it actually work.
when it jumps to another frame and returns to the previous frame, the player may renders differently.
When users are seeking, mpv would destroy the filter and re-activate it again automatically.
Thanks for the support.
Please try this initial implementation with RIFE(video_player=True).
This code may change later.