triton icon indicating copy to clipboard operation
triton copied to clipboard

Create jobs per media file, not per media

Open jaredallard opened this issue 4 years ago • 0 comments

We should stop creating a single media job for one media item. This causes scale problems with massive media collections requiring converter instances to download the entire media before starting (on every retry).

We should, instead, create a job for every single media file in a logical "media" object.

Work to do

  • [ ] Twilight: Make stateless (don't do any ++ logic, since ordering is no longer guaranteed)
  • [ ] Downloader: Create job during upload loop, not in main.js
  • [ ] Converter: support actually almost exists, so restructure code to handle just one.
  • [ ] core: add v2.converter object that is geared towards a single file..

jaredallard avatar Jul 23 '19 14:07 jaredallard