Video Cropping: Fix appearance in media library

[swissspidy]

While the video is processing, the poster in the media library looks wrong and the correct poster image kicks in later.

[spacedmonkey]

I played around with generating a poster image here.

https://github.com/GoogleForCreators/web-stories-wp/blob/3a05d3346e3f945c8a4e4dfdcfb94ea07637a31e/packages/story-editor/src/app/media/utils/useProcessMedia.js#L533-L537

Generate a poster file and blob and set it here.

We maybe about to generate a cropped poster image by passing x,y, height and width. here

https://github.com/GoogleForCreators/web-stories-wp/blob/3a05d3346e3f945c8a4e4dfdcfb94ea07637a31e/packages/media/src/getImageFromVideo.ts#L41

Either using the video or the existing poster image.

[spacedmonkey]

Related: https://github.com/GoogleForCreators/web-stories-wp/issues/8715

[swissspidy]

Ideally the poster generation should continue to happen as part of the regular media upload queue. I'd wanna void doing that in multiple different places.

Right now we use useUploadVideoFrame() (responsible for poster generation & upload) in only three places and I'd like to keep it that way:

1. Media upload queue, for all regular uploads
2. Media hotlinking
3. Media library, for backfilling of existing videos

But anyhow, right now I don't see any issues with the poster generation. Could you perhaps share a video of what is looking wrong?

[spacedmonkey]

https://user-images.githubusercontent.com/237508/190669367-63f01ae0-d529-4205-9101-b43943e60be1.mov

As you can see in the video, why the video is processing, the video poster does not represent what the user see on the canvas. It does correct itself after the video is processed.

Maybe we should tweak this effect

https://github.com/GoogleForCreators/web-stories-wp/blob/2c06dea77e044465aae90170a6104e2b2bbe2ba2/packages/story-editor/src/app/media/utils/useMediaUploadQueue/index.js#L153-L211

Generate the poster image should ffmpeg and pass the crop param to the getFirstFrameOfVideo function.

[swissspidy]

Thanks for the video!

As you can see in the video, why the video is processing, the video poster does not represent what the user see on the canvas. It does correct itself after the video is processed.

To be clear, are you referring to the video/poster in the media library or on the canvas? Because on the canvas the poster looks correct to me.

[spacedmonkey]

video/poster in the media library

On canvas is look correct. But in the media library is does look incorrect.

[swissspidy]

Got it! OK, we're on the same page then.

[swissspidy]

The reason why that happens is pretty clear I think: we add new uploading items to the library even before we do the cropping. Because of that the poster is off.

Now we can fix the poster, but when hovering over the poster to play the video, it will be off again.

[spacedmonkey]

Sadly I think we are going to have to use ffmpeg to generate the poster image. Until the video is done processing, we can't use the existing video / canavas method as it the cropping of the video is not complete.

Now we can fix the poster, but when hovering over the poster to play the video, it will be off again.

Maybe we disable the hover to play in that case. Or we could use css or replication the effect using the crop params we have in state.

[swissspidy]

Cropping with canvas is very easily possible, we don't need to use FFmpeg. It would be a one line change here:

https://github.com/GoogleForCreators/web-stories-wp/blob/2c06dea77e044465aae90170a6104e2b2bbe2ba2/packages/media/src/getImageFromVideo.ts#L41

Not sure yet what to do about the video part, will need to experiment with that I think.

[spacedmonkey]

I had a play around with this. I am not sure the best way forward. I could not get the canvas to work, as I dont understand how the params work.