Breakthrough
Improvements to common video use-cases, using @albanie's shot-detection-benchmarks
This issue is for the discussion of PySceneDetect's performance with respect to the shot-detection-benchmarks repo by @albanie, which includes a variety of shot detection benchmarks with real-world footage (e.g. movies, sports, TV shows).
The discussion here should focus on (but is certainly not limited to) determining the optimal parameters for current and any future detection methods, for each of the test cases in the shot-detection-benchmarks repo. The parameters listed will include those that both improve the accuracy of PySceneDetect for a particular test case, as well as the run-time performance where possible.
Furthermore, any other algorithm-specific changes or issues specifically pertaining to issues with the test footage will be added to this post for tracking and future reference. In the resources branch, I will eventually include a downloaded .MP4 version of the test cases and incorporate them as unit tests for PySceneDetect. The test cases may be expanded in the future should additional benchmarks be included in @albanie's shot-detection-benchmarks repo.
Our discussion from the previous comment thread in another issue, also serving as a reminder to update the nomenclature in the documentation (perhaps including a section on definitions of terms, especially those that differ between cinematography and what PySceneDetect denotes as a "scene".
@albanie commented: Do you think pyscenedetect should work well for shot detection (rather than scene detection) under content aware mode?
@Breakthrough commented: [...] each shot already is a "scene" in content-aware mode, as in the following example in the documentation: http://pyscenedetect.readthedocs.org/en/latest/examples/usage-example/
As you can see from the output images on that page, the Youtube clip gets split on each shot cut (this is what PySceneDetect calls a "scene"), so in this case, a shot and a scene are equivalent terms.
Is this what you meant? If so, do you think it would be worth explicitly stating that content-aware mode detects individual shots as opposed to scenes? [...]
@albanie commented: [...] Thanks - I figured that was the meaning but I've noticed that sometimes the two are considered distinct (e.g. in this explanation http://production.4filmmaking.com/cinematography1.html).
In case it's of interest to others, I've benchmarked some shot detection tools (including PySceneDetect) on a few videos: https://github.com/albanie/shot-detection-benchmarks
Currently, the threshold value is set manually for the entire duration of the video. After examining the statistics on the current videos in the set, I think exploring either an adaptive/dynamic threshold selected automatically using some kind of peak detection, or allowing a deviation from the threshold/sensitivity value by a user-defined amount (either absolute +/-, or different values for the positive/negative shifts).
This would be found manually for now, by analyzing the mean and deviation/variance of the peaks in the videos, and using those as the threshold/sensitivity and absolute +/- values, respectively. In the future, this could also be combined with peak detection to attempt to compute the values automatically.
@albanie, I've also heard of people using PySceneDetect for automated commercial/advertisement removal. Do you think it would be possible to include a clip of a television or news show, interlaced with real commercials? (I don't think this would pose any copyright issues for this purpose, although I am certainly not an expert on the topic.)
@Breakthrough That sounds like an appropriate use case - I'll add a test that includes commercials.
