Media: Video Segmentation Prototype

[timarney]

Context

Automatically split videos into smaller parts.

Example: you upload a 2 minute video and we allow you to split it into 15 seconds long segments (so 8 parts in total), putting every segment on its own page.

For this prototype we can create a new feature flag and some simple button (and maybe a range slider to set the segmentation length) somewhere that triggers this action.

Summary

Adds a segment video panel under styles when you select a video

Pressing segment will split the selected video into parts based on the number of seconds selected

Relevant Technical Choices

To-do

User-facing changes

Testing Instructions

👉 Note: the video is segmented based on the videos keyframes i.e. using a 30 sec segment length will not result in exactly 30 secs segments but rather the segments will be based on the closet keyframe.

Note that splitting may not be accurate, unless you force the reference stream key-frames at the given time. See the introductory notice and the examples below.

• [ ] This is a non-user-facing change and requires no QA

This PR can be tested by following these steps:

• Enable the segment video feature
• upload a video to be segmented
• select the video from the canvas
• visit the style tab / video segment panel
• select the # of seconds to segment the video into
• press segment
• the video should get segmented + split onto multiple pages

Reviews

Does this PR have a security-related impact?

No

Does this PR change what data or activity we track or use?

No

Does this PR have a legal-related impact?

No

Checklist

• [x] This PR addresses an existing issue and I have linked this PR to it in ZenHub
• [x] I have tested this code to the best of my abilities
• [ ] I have verified accessibility to the best of my abilities (docs)
• [ ] I have verified i18n and l10n (translation, right-to-left layout) to the best of my abilities
• [ ] This code is covered by automated tests (unit, integration, and/or e2e) to verify it works as intended (docs)
• [ ] I have added documentation where necessary
• [x] I have added a matching Type: XYZ label to the PR

---

Fixes #12164

[timarney]

@spacedmonkey @swissspidy --- adding this as ready for review now to get some feedback --- still testing on my end.

1. I think the UI (to let the user know something is happening) needs more work --- currently just using a disabled button and the snackbar.

2. I'm seeing some cases where the segmented videos end up out of order --- typically seeing this error when that happens.

---

https://github.com/GoogleForCreators/web-stories-wp/pull/12335#issuecomment-1255910357

[googleforcreators-bot]

Plugin builds for 5fd476f389113a2bc2f3f4fe0c9bdaf25f25ac60 are ready :bellhop_bell:!

• Download development build
• Download production build

[github-actions[bot]]

Size Change: +1.41 kB (0%)

Total Size: 2.72 MB

Filename	Size	Change
assets/js/wp-story-editor.js	1.44 MB	+1.41 kB (0%)

ℹ️ View Unchanged

Filename	Size
assets/css/carousel-view-rtl.css	702 B
assets/css/carousel-view.css	701 B
assets/css/web-stories-block-rtl.css	4.52 kB
assets/css/web-stories-block.css	4.56 kB
assets/css/web-stories-embed-rtl.css	318 B
assets/css/web-stories-embed.css	317 B
assets/css/web-stories-list-styles-rtl.css	2.36 kB
assets/css/web-stories-list-styles.css	2.39 kB
assets/css/web-stories-theme-style-twentyeleven-rtl.css	102 B
assets/css/web-stories-theme-style-twentyeleven.css	102 B
assets/css/web-stories-theme-style-twentyfifteen-rtl.css	251 B
assets/css/web-stories-theme-style-twentyfifteen.css	251 B
assets/css/web-stories-theme-style-twentyfourteen-rtl.css	287 B
assets/css/web-stories-theme-style-twentyfourteen.css	287 B
assets/css/web-stories-theme-style-twentyseventeen-rtl.css	288 B
assets/css/web-stories-theme-style-twentyseventeen.css	288 B
assets/css/web-stories-theme-style-twentysixteen-rtl.css	224 B
assets/css/web-stories-theme-style-twentysixteen.css	224 B
assets/css/web-stories-theme-style-twentyten-rtl.css	143 B
assets/css/web-stories-theme-style-twentyten.css	143 B
assets/css/web-stories-theme-style-twentytwelve-rtl.css	256 B
assets/css/web-stories-theme-style-twentytwelve.css	256 B
assets/css/web-stories-theme-style-twentytwenty-rtl.css	86 B
assets/css/web-stories-theme-style-twentytwenty.css	86 B
assets/css/web-stories-theme-style-twentytwentyone-rtl.css	326 B
assets/css/web-stories-theme-style-twentytwentyone.css	326 B
assets/css/web-stories-widget-rtl.css	482 B
assets/css/web-stories-widget.css	482 B
assets/css/wp-dashboard-rtl.css	657 B
assets/css/wp-dashboard.css	659 B
assets/css/wp-story-editor-rtl.css	737 B
assets/css/wp-story-editor.css	738 B
assets/js/1583.js	7.75 kB
assets/js/4422.js	49.3 kB
assets/js/5369.js	90.4 kB
assets/js/81.js	208 kB
assets/js/9419.js	35.1 kB
assets/js/9750.js	12.8 kB
assets/js/carousel-view.js	3.41 kB
assets/js/chunk-colorthief.js	2.64 kB
assets/js/chunk-ffmpeg.js	5.89 kB
assets/js/chunk-focus-visible.js	1.01 kB
assets/js/chunk-html-to-image.js	4.5 kB
assets/js/chunk-opentype.js	96 B
assets/js/chunk-react-calendar.js	12.5 kB
assets/js/chunk-react-color.js	44.3 kB
assets/js/chunk-web-animations-js.js	14.6 kB
assets/js/chunk-web-stories-template-0-metaData.js	545 B
assets/js/chunk-web-stories-template-0.js	11.4 kB
assets/js/chunk-web-stories-template-1-metaData.js	539 B
assets/js/chunk-web-stories-template-1.js	9.61 kB
assets/js/chunk-web-stories-template-10-metaData.js	533 B
assets/js/chunk-web-stories-template-10.js	7.37 kB
assets/js/chunk-web-stories-template-11-metaData.js	539 B
assets/js/chunk-web-stories-template-11.js	9.09 kB
assets/js/chunk-web-stories-template-12-metaData.js	497 B
assets/js/chunk-web-stories-template-12.js	9.7 kB
assets/js/chunk-web-stories-template-13-metaData.js	525 B
assets/js/chunk-web-stories-template-13.js	7.4 kB
assets/js/chunk-web-stories-template-14-metaData.js	582 B
assets/js/chunk-web-stories-template-14.js	7.37 kB
assets/js/chunk-web-stories-template-15-metaData.js	544 B
assets/js/chunk-web-stories-template-15.js	9 kB
assets/js/chunk-web-stories-template-16-metaData.js	588 B
assets/js/chunk-web-stories-template-16.js	10.9 kB
assets/js/chunk-web-stories-template-17-metaData.js	539 B
assets/js/chunk-web-stories-template-17.js	9.2 kB
assets/js/chunk-web-stories-template-18-metaData.js	585 B
assets/js/chunk-web-stories-template-18.js	9.91 kB
assets/js/chunk-web-stories-template-19-metaData.js	501 B
assets/js/chunk-web-stories-template-19.js	10.8 kB
assets/js/chunk-web-stories-template-2-metaData.js	586 B
assets/js/chunk-web-stories-template-2.js	9.3 kB
assets/js/chunk-web-stories-template-20-metaData.js	548 B
assets/js/chunk-web-stories-template-20.js	9.01 kB
assets/js/chunk-web-stories-template-21-metaData.js	535 B
assets/js/chunk-web-stories-template-21.js	9.85 kB
assets/js/chunk-web-stories-template-22-metaData.js	525 B
assets/js/chunk-web-stories-template-22.js	7.83 kB
assets/js/chunk-web-stories-template-23-metaData.js	605 B
assets/js/chunk-web-stories-template-23.js	7.48 kB
assets/js/chunk-web-stories-template-24-metaData.js	516 B
assets/js/chunk-web-stories-template-24.js	11.7 kB
assets/js/chunk-web-stories-template-25-metaData.js	544 B
assets/js/chunk-web-stories-template-25.js	7.06 kB
assets/js/chunk-web-stories-template-26-metaData.js	601 B
assets/js/chunk-web-stories-template-26.js	7.27 kB
assets/js/chunk-web-stories-template-27-metaData.js	542 B
assets/js/chunk-web-stories-template-27.js	7.82 kB
assets/js/chunk-web-stories-template-28-metaData.js	532 B
assets/js/chunk-web-stories-template-28.js	9.07 kB
assets/js/chunk-web-stories-template-29-metaData.js	562 B
assets/js/chunk-web-stories-template-29.js	9.25 kB
assets/js/chunk-web-stories-template-3-metaData.js	539 B
assets/js/chunk-web-stories-template-3.js	8.42 kB
assets/js/chunk-web-stories-template-30-metaData.js	576 B
assets/js/chunk-web-stories-template-30.js	7.89 kB
assets/js/chunk-web-stories-template-31-metaData.js	502 B
assets/js/chunk-web-stories-template-31.js	10.3 kB
assets/js/chunk-web-stories-template-32-metaData.js	552 B
assets/js/chunk-web-stories-template-32.js	13.3 kB
assets/js/chunk-web-stories-template-33-metaData.js	492 B
assets/js/chunk-web-stories-template-33.js	9.07 kB
assets/js/chunk-web-stories-template-34-metaData.js	571 B
assets/js/chunk-web-stories-template-34.js	7.58 kB
assets/js/chunk-web-stories-template-35-metaData.js	566 B
assets/js/chunk-web-stories-template-35.js	8.91 kB
assets/js/chunk-web-stories-template-36-metaData.js	577 B
assets/js/chunk-web-stories-template-36.js	12.7 kB
assets/js/chunk-web-stories-template-37-metaData.js	528 B
assets/js/chunk-web-stories-template-37.js	6.71 kB
assets/js/chunk-web-stories-template-38-metaData.js	572 B
assets/js/chunk-web-stories-template-38.js	7.94 kB
assets/js/chunk-web-stories-template-39-metaData.js	588 B
assets/js/chunk-web-stories-template-39.js	8.08 kB
assets/js/chunk-web-stories-template-4-metaData.js	564 B
assets/js/chunk-web-stories-template-4.js	12.7 kB
assets/js/chunk-web-stories-template-40-metaData.js	557 B
assets/js/chunk-web-stories-template-40.js	10.2 kB
assets/js/chunk-web-stories-template-41-metaData.js	572 B
assets/js/chunk-web-stories-template-41.js	7.75 kB
assets/js/chunk-web-stories-template-42-metaData.js	521 B
assets/js/chunk-web-stories-template-42.js	7 kB
assets/js/chunk-web-stories-template-43-metaData.js	557 B
assets/js/chunk-web-stories-template-43.js	8.76 kB
assets/js/chunk-web-stories-template-44-metaData.js	584 B
assets/js/chunk-web-stories-template-44.js	11.1 kB
assets/js/chunk-web-stories-template-45-metaData.js	565 B
assets/js/chunk-web-stories-template-45.js	7.52 kB
assets/js/chunk-web-stories-template-46-metaData.js	531 B
assets/js/chunk-web-stories-template-46.js	5.22 kB
assets/js/chunk-web-stories-template-47-metaData.js	592 B
assets/js/chunk-web-stories-template-47.js	9.42 kB
assets/js/chunk-web-stories-template-48-metaData.js	555 B
assets/js/chunk-web-stories-template-48.js	9.09 kB
assets/js/chunk-web-stories-template-49-metaData.js	518 B
assets/js/chunk-web-stories-template-49.js	9.69 kB
assets/js/chunk-web-stories-template-5-metaData.js	556 B
assets/js/chunk-web-stories-template-5.js	9.94 kB
assets/js/chunk-web-stories-template-50-metaData.js	503 B
assets/js/chunk-web-stories-template-50.js	9.15 kB
assets/js/chunk-web-stories-template-51-metaData.js	526 B
assets/js/chunk-web-stories-template-51.js	10.4 kB
assets/js/chunk-web-stories-template-52-metaData.js	602 B
assets/js/chunk-web-stories-template-52.js	10.4 kB
assets/js/chunk-web-stories-template-53-metaData.js	553 B
assets/js/chunk-web-stories-template-53.js	5.78 kB
assets/js/chunk-web-stories-template-54-metaData.js	547 B
assets/js/chunk-web-stories-template-54.js	7.67 kB
assets/js/chunk-web-stories-template-55-metaData.js	573 B
assets/js/chunk-web-stories-template-55.js	7.13 kB
assets/js/chunk-web-stories-template-56-metaData.js	542 B
assets/js/chunk-web-stories-template-56.js	9.87 kB
assets/js/chunk-web-stories-template-57-metaData.js	527 B
assets/js/chunk-web-stories-template-57.js	14.9 kB
assets/js/chunk-web-stories-template-58-metaData.js	555 B
assets/js/chunk-web-stories-template-58.js	5.74 kB
assets/js/chunk-web-stories-template-59-metaData.js	590 B
assets/js/chunk-web-stories-template-59.js	8.96 kB
assets/js/chunk-web-stories-template-6-metaData.js	569 B
assets/js/chunk-web-stories-template-6.js	7.07 kB
assets/js/chunk-web-stories-template-60-metaData.js	510 B
assets/js/chunk-web-stories-template-60.js	9.51 kB
assets/js/chunk-web-stories-template-7-metaData.js	569 B
assets/js/chunk-web-stories-template-7.js	7.46 kB
assets/js/chunk-web-stories-template-8-metaData.js	569 B
assets/js/chunk-web-stories-template-8.js	8.93 kB
assets/js/chunk-web-stories-template-9-metaData.js	581 B
assets/js/chunk-web-stories-template-9.js	8.46 kB
assets/js/chunk-web-stories-templates.js	1.17 kB
assets/js/chunk-web-stories-textset-0.js	5.06 kB
assets/js/chunk-web-stories-textset-1.js	6.65 kB
assets/js/chunk-web-stories-textset-2.js	7.65 kB
assets/js/chunk-web-stories-textset-3.js	15.1 kB
assets/js/chunk-web-stories-textset-4.js	4.15 kB
assets/js/chunk-web-stories-textset-5.js	5.47 kB
assets/js/chunk-web-stories-textset-6.js	5.28 kB
assets/js/chunk-web-stories-textset-7.js	10.2 kB
assets/js/generateBlurhash.worker.worker.js	1.1 kB
assets/js/imgareaselect.js	3.77 kB
assets/js/lightbox.js	550 B
assets/js/tinymce-button.js	2.85 kB
assets/js/web-stories-activation-notice.js	27.1 kB
assets/js/web-stories-block.js	22.6 kB
assets/js/web-stories-embed.js	20 B
assets/js/web-stories-widget.js	587 B
assets/js/wp-dashboard.js	63.8 kB

_{compressed-size-action}

[spacedmonkey]

This UI should check the length of the original video and not give option to break in to segments longer than the video. I could not be only able to select valid options. This UI should be hidden if shorter than 20 seconds.

[spacedmonkey]

https://user-images.githubusercontent.com/237508/191723828-1c28a6b7-8394-48c9-bc2c-011bf748ab80.mov

In my testing, I found two things.

• It seems like some of the segments are 19 / 21 seconds. This seems odd, worth looking into.
• When the page is created, it automatically changes to the new page and starts playing the video. This makes a lot of noise.

[timarney]

https://github.com/GoogleForCreators/web-stories-wp/pull/12335#issuecomment-1254836974

Screen.Recording.2022-09-22.at.11.25.31.mov In my testing, I found two things.

• It seems like some of the segments are 19 / 21 seconds. This seems odd, worth looking into.
• When the page is created, it automatically changes to the new page and starts playing the video. This makes a lot of noise.

It seems like some of the segments are 19 / 21 seconds. This seems odd, worth looking into.

I added a note about that under testing in the summary above

👉 Note: the video is segmented based on the videos keyframes -- we can force keyframes to happen at more exact intervals but this will result in larger videos --- as the keyframes we would be forcing to ensure the segment length are for our purposes vs the actual file compression (more keyframes = larger file) so it's a trade-off. In this case the keyframe are frame diff information not "animation" keyframes.

-- the segment muxer will start the new segment with the key frame found next after the specified start time.

When the page is created, it automatically changes to the new page and starts playing the video. This makes a lot of noise.

Thinking the same --- I'm using the existing logic to add pages .... do we have a way to insert a new page without moving to it?

[swissspidy]

It seems like some of the segments are 19 / 21 seconds. This seems odd, worth looking into.

I added a note about that under testing in the summary above

After thinking a bit more about it, I am definitely not a fan. Let's force the segments to be accurate.

Thinking the same --- I'm using the existing logic to add pages .... do we have a way to insert a new page without moving to it?

I think we can add a new option to the addPage reducer to accomplish that.

See: https://github.com/GoogleForCreators/web-stories-wp/pull/12335#issuecomment-1260689724

[swissspidy]

2. I'm seeing some cases where the segmented videos end up out of order --- typically seeing this error when that happens.

Sounds like we can add some hardening there. Mind filing a ticket?

Update --- added #12377

[lgtm-com[bot]]

This pull request introduces 3 alerts when merging 83c29d829ef6a673057b7346ffc49cfedc6a215a into 3f59057a177a76bf886d466fcb0e7703b060c5cf - view on LGTM.com

new alerts:

• 2 for Unused variable, import, function or class
• 1 for Syntax error

[timarney]

RE:

https://github.com/GoogleForCreators/web-stories-wp/pull/12335#issuecomment-1254883207

After thinking a bit more about it, I am definitely not a fan. Let's force the segments to be accurate.

Added force key frames , segment times and segment delta

Seeing times off by 1, 2 secs

Thinking the same --- I'm using the existing logic to add pages .... do we have a way to insert a new page without moving to it?

Reducer has been updated.

[timarney]

@swissspidy ---

Thoughts on best way to get the batchId back to videoSegment ... to check using the useEffect ?

[spacedmonkey]

In my testing, I was seeing a video uploading with 0:00. It seems to be less than 1 second. Maybe we should not upload the video unless it more than 1 second?

[spacedmonkey]

I am seeing

https://user-images.githubusercontent.com/237508/192821714-515417de-385d-4336-bbe8-87893cef106a.mov

I am seeing this work with a 7 minute video. But after the videos are done processing, it then pages are not created. Any idea why this is happening?

[swissspidy]

In my testing, I was seeing a video uploading with 0:00. It seems to be less than 1 second. Maybe we should not upload the video unless it more than 1 second?

Not uploading would possibly mean losing parts of the video.

One of many UX questions to resolve after prototype.

But after the videos are done processing, it then pages are not created. Any idea why this is happening?

I think that's simply because of the open to-do Tim mentioned above & in our sync

[timarney]

I am seeing this work with a 7 minute video. But after the videos are done processing, it then pages are not created. Any idea why this is happening?

A few things --

• Any chance your using local WP can't quite tell in your video --- seems to drop the connection some times
• Any ffMpeg errors ?--- should have some log output around exit() ... there's a lot of noise with frame creation so hard to spot
• The hard coded batchId could mess things up for sure
• Looks like you ran into this issue https://github.com/GoogleForCreators/web-stories-wp/issues/12377 when can trip things up

My testing has been mostly with 2, 4 min videos so far

[swissspidy]

I just pushed some changes after looking at this for a bit. Added a new reducer that makes things a lot easier to comprehend. Insertion should work better now too.

There's plenty of things that need to be addressed from a UX perspective, but this prototype is almost in a mergeable state now. Just need some unit tests for the changes to the story reducers.

PTAL again.

[timarney]

@swissspidy --- taking another look / testing this.

We now end up with a blank page at the start --- is this what we want?

The previous code created new page(s) "after" the first element -- for the first page I was inserting the new element (also selected it) + deleted the old one i.e. re-using the existing page where the segment button was pressed.

[swissspidy]

We now end up with a blank page at the start --- is this what we want?

It's not necessarily blank, it's just the current page you were on. There could have been other elements on it than just the video.

But in any case, yes, this is good enough for the prototype. It's one of the many UX questions we need to figure out later.

[spacedmonkey]

Not uploading would possibly mean losing parts of the video.

What could you miss in less than a second? A video of less than a second is not useful to a user. Maybe the cut off should be at least 75% of a second.

[spacedmonkey]

I am still not seeing the pages create at all. Not seeing any errors.

https://user-images.githubusercontent.com/237508/193244275-f7c6b347-16c4-4ba7-827f-84633286ff13.mov

[swissspidy]

Let's figure out this scenario later as part of UX rounds. It's a good point.

As mentioned on Wednesday, I'd like us to write down things like this for the future.

[timarney]

What could you miss in less than a second? A video of less than a second is not useful to a user. Maybe the cut off should be at least 75% of a second.

For a later PR we could account for this type of thing when we calc the segment times

https://github.com/GoogleForCreators/web-stories-wp/pull/12335/files#diff-8383cf154a6a03f61cb68645b7fd4267892b7df793c6b3cceb999e40bcf2522bR306

Check the length of the last segment time and if <x tag it onto the previous segment time --- something like that would prevent videos of x length when doing the split.

[timarney]

I am still not seeing the pages create at all. Not seeing any errors.

Re-tested with a few different videos

4-5 times worked ... the other time I didn't see any errors and no pages created. Possibly hitting the early return on the add pages code.

Let's figure out this scenario later as part of UX rounds. It's a good point.

For sure need better UX feedback to show what's happening.

[spacedmonkey]

@swissspidy @timarney I spent a lot time debugging this. The answer was straight in front of me. In my testing, after starting the segmenting running, I change to the media tab in the sidebar. This destorys all the state in VideoSegmentPanel and then the hook never runs and never creates the page.

I believe we should start by moving everything into a hook, the click is a light length as is. We can use trimming component as a base to copy from.

[swissspidy]

Ah, good to know. Yeah we can do such a refactoring as a follow-up item.

[timarney]

@spacedmonkey @swissspidy

Anything else needed on this one given it's behind the feature flag?

Follow-ups:

State issue https://github.com/GoogleForCreators/web-stories-wp/issues/12463

• I believe we should start by moving everything into a hook, the click is a light length as is. We can use trimming component as a base to copy from.

End video issue (x sec last segment) https://github.com/GoogleForCreators/web-stories-wp/issues/12465

• https://github.com/GoogleForCreators/web-stories-wp/pull/12335#issuecomment-1263407802

UI Progress https://github.com/GoogleForCreators/web-stories-wp/issues/12464

• Need to provide better feedback
• https://github.com/GoogleForCreators/web-stories-wp/issues/12164#issuecomment-1261130051

[spacedmonkey]

Let's discuss this in our next meeting.

[spacedmonkey]

I am not sure I want to approve this PR. Change the sidebar to another tab breaks it. At the very least, I would expect todos in the code, a callout in the testing / QA notes to say not to do this and a ticket explain how this functionality will be fixed in the future. Prototype that breaks this easily is going to make problems for those testing it and evaluating the functionality. I already used a lot of time debugging.

[timarney]

@spacedmonkey

Added a QA note with the following issues for follow-up

https://github.com/GoogleForCreators/web-stories-wp/issues/12463 https://github.com/GoogleForCreators/web-stories-wp/issues/12464 https://github.com/GoogleForCreators/web-stories-wp/issues/12465

Also added a todo for the state issue on in the code

https://github.com/GoogleForCreators/web-stories-wp/pull/12335/commits/2779003127ee7004d9eaee4bbeb1b62fcbf0170d

[swissspidy]

Thanks for creating the follow-up tickets 👍

I think we're good to proceed here for now.