utility for video reader and manipulation

[innat]

trafficstars

Unlike keras.utils.load_img API for image data reader, currently there is no API for video reader. It would be useful to have this for video modeling dataloader. Currently for video reader in the tf.data API, user has to use tf.py_function which slow downs the process. Below are some expected utility in such regard

video_file = 'sample.mp4'
num_frames = 20

frames = keras.utils.load_video(video_file) # (100, 224, 224, 3)
sub_frames = keras.utils.uniform_temporal_subsample(
    frames, num_frames, temporal_dim=-4
) # (20, 224, 224, 3)

• torch uniform_temporal_subsample references.
• tf uniform_temporal_subsample references
• video reader: decord, opencv, av

[nkovela1]

Hi @innat, this sounds like an exciting idea! Would you be willing to contribute a PR to add this op to Keras 3?

[innat]

Yes, I'm willing to contribute. But which tools keras would prefer for video decoding (decord, opencv, av)? ( I would recommend decord but it's thrid party. )

[nkovela1]

@fchollet Do you have a preference for video decoding? (decord, opencv, av)

[innat-asj]

@nkovela1 I forgot to mention that, torch-vision has its own dedicated video utility. But not sure if anything available in tensor-flow or jax. If there are something well performant available, then those can be used instead, leveraging each backend for their own way to decode (similar to onnx support).

Possible video decoder in tensor-flow: tfio.experimental.ffmpeg.decode_video. - open issue https://github.com/tensorflow/io/issues/1064

[Collins-Webdev]

Bonjour tout le monde 👋 Votre projet me semble bien être très intéressant. J'aimerais aussi contribuer au projet et plancher notamment sur le problème de utilitaire de lecture et de manipulation vidéo. Veuillez bien vouloir m'ajouter. Merci d'avance 🤝.

[github-actions[bot]]

This issue is stale because it has been open for 180 days with no activity. It will be closed if no further activity occurs. Thank you.