DVR-Scan
DVR-Scan copied to clipboard
Breakthrough
Add different frame skipping modes
Currently, using --frame-skip acts on the entire video itself, including once we enter a motion event. This improves processing performance, but also reduces accuracy. Based on the fact that motion events likely take up a small proportion of the video, I would like to propose a hybrid solution, where frame skipping is only used outside of motion events. Then, once a motion event is detected, we start processing every frame individually.
This assumes for the most part that the motion score between two frames is always <= the score between skipped ones, but that is generally true since the further apart the frames are in time, the more likely there are to be large differences in pixel values when there is motion present. Since the background subtractors can only handle static backgrounds, there shouldn't be much interference there either.
As another benefit, the bounding boxes would also be updated each frame properly with this change, and it also reduces the variance in detection performance during events. Lastly, the performance loss is likely negligible to that which will be added in by #81, since video encoding is typically a more expensive operation (at least with the current encoder). Performing the encoding in a separate thread (#52) will also help offset any costs of this change.
As identified in the comments below, it also is possible for the more accurate scan to be done in a separate thread. This would require a different lookahead buffer size to be added to the configuration file, but that should be acceptable for most use cases in terms of accuracy (I suspect there would be almost no difference between a regular scan with frame skip 0). This would be a rather large change to the internal scanning logic, but is a promising performance improvement for nearly all common use cases (and would likely be much faster than the hybrid solution discussed above).
It might be better to allow different frame skip modes, so will treat this as less of a change and more of a feature.
Sounds good. Actually, it seems to be a very reasonable change in behavior that solves a few problems while having only very limited downsides.
It makes a lot of sense to skip some frames outside of motion events, where you can miss only very, very short events (that may not even be noticeable and often below the minimum event length), or miss a tiny part at the beginning of a motion event. Depending on the video’s FPS and how many frames you skip, in most configurations this should not even be something a human can recognize.
On the other hand, it makes much less sense to skip frames during an event, where you lose quality (reducing FPS), in case output files are written, and processing becomes inconsistent with the default behavior – for very little gain (in most configurations and for most inputs with limited motion).
@ocram your last comment on https://github.com/Breakthrough/DVR-Scan/issues/70#issuecomment-1037553747 is a great idea:
As an alternative, you can quickly scan source videos with --frame-skip enabled, and if motion is found, you can re-scan with --frame-skip omitted for frame-accurate results, depending on how likely motion is to be found.
That might actually be faster if the more accurate scan is done in a separate thread in parallel, and we intelligently seek to the next event (e.g. if there's a large gap between events, wait for the frame-skipped thread to identify the next event, and then seek to N frames before and restart the background subtractor).
This might require 3 frame skip modes (all - current behavior, scan - proposed in comment 1, and two-pass, your suggestion), but eager to experiment with this.
Oh I was actually thinking about the user of the program, who could just re-run their command without frame skipping if they really need frame-accurate results.
But sure, this could also be done by the program itself. I just don’t know if the amount of work and the added complexity is worth it for those few use cases where somebody needs frame skipping but also cannot tolerate inaccuracies by a few frames. If it’s easy to integrate, however, why not.