creamy-videos icon indicating copy to clipboard operation
creamy-videos copied to clipboard

Feature: automatically convert videos to a streamable MP4 format on upload ("transcode")

Open AlbinoDrought opened this issue 6 years ago • 0 comments

Not 100% satisfied with this one, it introduces some issues:

  • Thumbnail generation and video transcoding could finish at similar times for the same video. This could cause data to be lost (either thumbnail, or transcoded video path). This also goes for updates from the web UI. This is a big one to me. I think (My)SQL locks can handle this?

  • We can no longer use original_file_name as the file name when downloading the video. If we upload some movie.mov, it may be transcoded to transcoded.mp4 - these extensions are most likely not compatible.

  • Transcode queue is lost on process crash/restart/whatever - it is entirely in memory. It is also easy to backlog (locally, a transcode of a ~260MB dashcam video took 1m20s+) and cannot be divided-and-conquered (no form of distribution).

  • Memory and CPU usage both go through the roof when transcoding something. This might be expected... but oof

Possible solutions:

  • Can we just update a single field using go-pg? Probably somehow, maybe this happens already, not sure. Otherwise we'll probably need to do some FOR UPDATE lock and handle clashes here 😕

  • Only transcode problematic videos to reduce queue backlog and resource usage. Not sure how this could be determined. Right now, the most problematic videos seem to be >20MB MOVs. MP4/WEBMs both work well enough.

  • Use real job queue and distributed worker pool to divide-and-conquer jobs. Easy enough, but adds dependency on job queue (probably another feature flag for this one, just like Postgres). TransformedFileSystem can work across systems assuming mounted docker volume can be ReadWriteMany (NFS).

AlbinoDrought avatar Feb 06 '19 07:02 AlbinoDrought