livepeer
Add ability to specify a primary and backup Transcoder for an Orchestrator
Is your feature request related to a problem? Please describe. This morning the Transcoder I have in SIN went offline do to a hosting issue so I was unable to receive work in the region until I manually started another slower T that can also serve the region. I don't keep both Ts connected because the local T has better performance and with the current logic work is evenly split between the 2 Ts.
Describe the solution you'd like The ability to specify a Transcoder as the primary T and a second T as the backup.
Because an O will accept any T that connects with the correct -orchSecret there will probably be the need to specify a T as the primary (or backup) on the T in addition to adding logic to the O to determine which T to use.
i.e. -transcoderA could designate a primary T. Multiple primary T should be able to be used for setups where operators run split O/T setups as their preferred configuration. Basically any T started as -transcoderA would act in the current manner that multiple Ts connected to an O do now. When a T is connected with -transcoderB specified it will only be used if no -transcoderA is available.
That's a good idea to have some sort of backup transcoders on O level. It could be implemented by adding a CLI flag -secondary for a standalone T, with which it will communicate a new Secondary capability to O. O, then, will not schedule jobs on these transcoders, unless every other transcoder is at capacity, or not available.
However, it also feels like a redundant mechanism for a very specific use case. We already have fail-over logic on B for Os and on O for Ts, and I'm not sure it's a good idea to introduce 'secondary' transcoders on top of that. WDYT, @thomshutt @leszko @iameli ?
Another use case for the option to add a -secondary flag that would schedule jobs to "secondary" Ts if the primary T was capacity would be for multi-GPU setups with mixed GPUs so that work can be prioritized to be sent to faster GPUs first and then only use slower GPUs when the fast GPU has reached capacity. I have machines now that I leave cards idle on because I don't have a way to prioritize which GPU a job is sent to.
Hmm, why not just omit less powerful GPUs from the -nvidia arg, if you don't want to use them at all?
But I see why it may be desirable to prioritize the hardware managed by a single transcoder by efficiency, in terms of $/kWh. We can add a configuration flag, which controls how T's internal load balancer selects particular unit of hardware for new sessions. It's a bit different, than transcoder prioritization on O discussed above.
I think having a config flag that allows some control over how work is load balanced would be a great feature. It ties in to the request I made to be able to set -maxSessions on a per GPU basis https://github.com/livepeer/go-livepeer/issues/2034.
I don't know how granular would be realistic in allowing prioritization but an example of how it would be useful if you take my LAX node for example, it has a GTX 1070 and an RTX 4000, the 1070 with it's dual NVENCs is better for multi resolution transcoding because the single NVDEC maxes out before the NVENCs and bottleneck the transcoding. In contrast the RTX 4000 is the opposite with 2x NVDECs and 1x NVENC is better for transcoding single 480p sessions because with multi resolution work the NVENC is the bottleneck. If there was a way to prioritize GPUs or Transcoders based on the number of pixels per frame (or similar) that would allow for optimal load balancing.
Sounds like #2034 is higher priority and needs to be addressed first, because we are having increasingly heterogeneous hardware not just across the network, but even on specific T level.