Remote Runners Hardware Acceleration

[MattyBoombalatty]

Describe the problem to be solved

Remote runners are great, but they could be better.

Describe the solution you would like

Some people may use remote runners for transcoding, so if those runners utilize graphics cards or other hardware that could accelerate transcoding, it would be good to be able to add a configuration option to the .toml file to pass flags to FFMPEG to enable hardware acceleration.

[kodxana]

Was thinking at the same thing would love to have support for quick sync support.

[normen]

For a quick workaround hack you can "intercept" the runners command by just placing a bash script / bat file called ffmpeg in the path of the runner, which it will then use as if it was the real ffmpeg. Then you can modify what it does.

For example I wanted to remove the scaling:

#!/usr/bin/env bash
arguments="$@"
pattern='(.*) -vcodec copy (.*) -vf scale[^ ]* (.*)'
if [[ "$arguments" =~ $pattern ]];then
  command_str="/usr/bin/ffmpeg ${BASH_REMATCH[1]} -vcodec copy ${BASH_REMATCH[2]} ${BASH_REMATCH[3]}"
  $command_str
else
  /usr/bin/ffmpeg $@
fi

[MattyBoombalatty]

Cool idea, but I wouldn't know how to do this. If you could provide a short guide I'd appreciate it!

[normen]

I don't know how your setup looks but you could put that above script into your /usr/local/bin folder as a file called ffmpeg and make it executable (chmod +x). Then peertube-runner will call that script instead when it wants to execute ffmpeg. What it currently does is take the arguments that peertube-runner sends and change them. If you just want to change the "-vcodec libx264" part for example it would look something like this:

#!/usr/bin/env bash
# get arguments into variable
arguments="$@"
# create regex pattern to split up arguments
pattern='(.*) -vcodec libx264 (.*)'
# do pattern search
if [[ "$arguments" =~ $pattern ]];then
  # construct new command from (.*) parts (BASH_REMATCH)
  command_str="/usr/bin/ffmpeg ${BASH_REMATCH[1]} -vcodec h264_nvenc ${BASH_REMATCH[2]}"
  # execute new command
  $command_str
else
  # just execute ffmpeg with arguments if the pattern wasn't found
  /usr/bin/ffmpeg $@
fi

[MattyBoombalatty]

That's a pretty interesting way to do it. I don't do much with regex or really bash scripting, but I'll definitely keep this in mind and test it as well. Thank you!

[MattyBoombalatty]

I tried many variations of this, including creating a wrapper in /usr/local/bin/, creating an alias and so on and so forth but to no avail.. It works when I manually enter the FFMPEG command (libx264 is changed to h264_nvenc automatically), but for some reason it is just not working with the PeerTube runner. I wonder if the fluent-ffmpeg NPM package handles the commands differently.

[normen]

Hm, I have this working on a linux machine but I didn't update the runner in a while.. Did you also try to replace the actual /usr/bin/ffmpeg? Just to see if the runner doesn't by now use its own packaged ffmpeg or something?

Edit: Also I think it picked it up if it was in the same folder I run peertube-runner from..

[MattyBoombalatty]

I tried that, and then had it reference another ffmpeg binary but that didn't work. At the moment I'm looking through the peertube-runner NPM module and seeing if I can edit the -vcodec flag manually so that it's just passed to FFMPEG as the required flag rather than trying to rig it another way. Probably best to edit the source and maybe fork it. My JavaScript is a little rusty :)

86694 function getDefaultEncodersToTry() {
86695   return {
86696     vod: {
86697       video: ["libx264"],
86698       audio: ["libfdk_aac", "aac"]
86699     },
86700     live: {
86701       video: ["libx264"],
86702       audio: ["libfdk_aac", "aac"]
86703     }
86704   };

[MattyBoombalatty]

@normen , what node version are you using for the PeerTube runner? If your intercept works I'd like to try it with your version.

[normen]

That would be peertube-runner 0.0.5 on node 18 on that machine.

[MattyBoombalatty]

I tried downgrading but unfortunately that didn't work. It looks like the PeerTube runner has its own FFMPEG integration, so I think the settings would have to be changed/added within the runner JavaScript file.

[normen]

Thats strange, you sure you're not running in some docker environment or something? I see no trace of any ffmpeg on that machine except the system installed ones. peertube-runner instantly fails when theres no ffmpeg installed.

[MattyBoombalatty]

Nope, no crazy weird environments. Just Ubuntu. Perhaps I'm missing something I was supposed to do with FFMPEG? Maybe I had to compile it? I have no idea. When I send a test command to FFMPEG it works fine and redirects to using hardware acceleration but when it comes from the runner, it doesn't. I don't know how that's possible because in theory the interception should come after the runner builds the command, right?

[Chocobozzz]

Hi,

To use a custom ffmpeg command you can set FFMPEG_PATH and FFPROBE_PATH environment variables when running peertube runner in server mode.

[MattyBoombalatty]

In that case, we'd have to compile FFMPEG with the rule enabled? I'll try just using the script as the path, see what happens.

[MattyBoombalatty]

Tried adding this to ~/.bashrc but it didn't seem to work. Still being encoded with libx264.

export FFMPEG_PATH='/usr/bin/ffmpeg -vcodec h264_nvenc'

[normen]

Tried adding this to ~/.bashrc but it didn't seem to work. Still being encoded with libx264.

export FFMPEG_PATH='/usr/bin/ffmpeg -vcodec h264_nvenc'

Nah, you have to set FFMPEG_PATH to the location of your script (e.g. /usr/local/bin/ffmpeg)

[MattyBoombalatty]

I finally got the transcoding redirection to work, and the HLS videos transcode correctly but the web-video runner job does not complete correctly, and an error from ffprobe is given that it cannot find the moov atom. I'll post the script and error below.

Error:

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x564d944d6ec0] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible!
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x564d944d6ec0] moov atom not found

Script which is located at /usr/bin/ffmpeg

#!/bin/bash

# Path to the original ffmpeg binary
ORIGINAL_FFMPEG="/usr/bin/ffmpeg.bak"

# Log all output to a file
log_file="/var/log/ffmpeg/exec.log"
exec > >(tee -a "$log_file")
exec 2>&1

# Intercept the command-line arguments
input_command=("$@")

# Log the original command to syslog
logger "Original Command: ${input_command[*]}"

# Find the index of the video codec option ("-vcodec")
codec_index=-1

for ((i=0; i<${#input_command[@]}; i++)); do
    if [[ "${input_command[i]}" == "-vcodec" ]]; then
        codec_index=$i
        break
    fi
done

# If video codec option is found, replace it with h264_nvenc
if [ $codec_index -ne -1 ]; then
    input_command[$((codec_index + 1))]="h264_nvenc"
fi

# Find the index of the preset option ("-preset")
preset_index=-1

for ((i=0; i<${#input_command[@]}; i++)); do
    if [[ "${input_command[i]}" == "-preset" && "${input_command[i+1]}" == "veryfast" ]]; then
        preset_index=$i
        break
    fi
done

# If preset option is found, replace it with medium
if [ $preset_index -ne -1 ]; then
    input_command[$preset_index + 1]="medium"
fi

# Find the index of the output file option
output_index=-1

for ((i=0; i<${#input_command[@]}; i++)); do
    if [[ "${input_command[i]}" == "-hls_segment_filename" ]]; then
        output_index=$i
        break
    fi
done

# Add -bf 4 and -gpu 0 options before the output file
if [ $output_index -ne -1 ]; then
    input_command=("${input_command[@]:0:$output_index}" "-bf" "4" "-gpu" "0" "${input_command[@]:$output_index}")
fi

# Log the modified command to syslog
logger "Modified Command: ${input_command[*]}"

# Execute the modified command with the original ffmpeg binary
eval "$ORIGINAL_FFMPEG ${input_command[*]}" 2>&1 | tee -a "$log_file"

[MattyBoombalatty]

Never mind, I found the issue. I had to make sure that the -bf 4 and -gpu 0 switches were added after the -g:v [value] rather than after the -hls_segment_filename switch since the web-videos don't contain that. It works now. However I think it would definitely be cool to have native support for this.

[MattyBoombalatty]

It works for VODs, but not livestreams. This was the final script I came up with and the RTMP errors I got when trying to use it. So close to getting it to work.

Script:

#!/bin/bash

# Path to the original ffmpeg binary
ORIGINAL_FFMPEG="/usr/bin/ffmpeg"

# Log all output to a file
log_file="/var/log/ffmpeg/exec.log"
exec > >(tee -a "$log_file")
exec 2>&1

# Intercept the command-line arguments
input_command=("$@")

# Log the original command to syslog
logger "Original Command: ${input_command[*]}"

# Find the index of the video codec option ("-vcodec")
codec_index=-1

for ((i=0; i<${#input_command[@]}; i++)); do
    if [[ "${input_command[i]}" == "-vcodec" ]]; then
        codec_index=$i
        break
    fi
done

# If video codec option is found, replace it with h264_nvenc
if [ $codec_index -ne -1 ]; then
    input_command[$((codec_index + 1))]="h264_nvenc"
fi

# Find the index of the preset option ("-preset")
preset_index=-1

for ((i=0; i<${#input_command[@]}; i++)); do
    if [[ "${input_command[i]}" == "-preset" && "${input_command[i+1]}" == "veryfast" ]]; then
        preset_index=$i
        break
    fi
done

# If preset option is found, replace it with medium
if [ $preset_index -ne -1 ]; then
    input_command[$preset_index + 1]="medium"
fi

# Find the index of the output file option
output_index=-1

for ((i=0; i<${#input_command[@]}; i++)); do
    if [[ "${input_command[i]}" == "-g:v" ]]; then
        output_index=$i
        break
    fi
done

# Add -bf 4 and -gpu 0 options after -g:v value
if [ $output_index -ne -1 ]; then
    input_command=("${input_command[@]:0:$output_index+2}" "-bf" "4" "-gpu" "0" "${input_command[@]:$output_index+2}")
fi

# Log the modified command to syslog
logger "Modified Command: ${input_command[*]}"

# Execute the modified command with the original ffmpeg binary
eval "$ORIGINAL_FFMPEG ${input_command[*]}" 2>&1 | tee -a "$log_file"

RTMP errors:

Filter split has an unconnected output
Filter split has an unconnected output
/usr/local/bin/ffmpeg: line 57: [vtemp960]scale=w=-2:h=960[vout960]: command not found
/usr/local/bin/ffmpeg: line 57: [vtemp960]scale=w=-2:h=960[vout960]: command not found

[MattyBoombalatty]

I have an updated script that works pretty well. But, does not work well with the remote studio. The quality gets all messed up.

[emansom]

Working on this: emansom/PeerTube@932c2336ae175562b59e3c5077e3d1c44215b1d6 in the branch emansom/PeerTube@feature/ffmpeg-hwaccel.

I'll open a PR once I've integrated it within the admin web UI and once it's working with multiple vendors. Currently i've only tested it on Intel and AMD using VAAPI acceleration. I plan on supporting VAAPI initially, NVENC and QSV later.

ETA until PR is about one month from now.

NVENC and QSV will come later as I currently do not have any NVIDIA GPU nor Intel Core Gen 8+ hardware to test with.

[SimplyCorbett]

@emansom

What's the status on this? I'd like to setup some remote runners on some orange pis and use the built in hardware acceleration on the GPU. (mostly to tinker with / see how it works in production).

They use a different encoder though but I don't mind modifying files to get things working.

[emansom]

@emansom

What's the status on this? I'd like to setup some remote runners on some orange pis and use the built in hardware acceleration on the GPU. (mostly to tinker with / see how it works in production).

They use a different encoder though but I don't mind modifying files to get things working.

I've run into some hardware QoS problems. My current workstation has an i7-4790 and RX 580. Both have too slow hardware encoders to test things at reasonable speed.

At the end of this month I'll be building a more modern AM5 based system. RDNA2/RDNA3 comes with improved hardware encoders, which I hope are fast enough to iterate, test and develop with.

The branch I linked is in a functional state, as long as you're comfortable with tweaking it yourself you can give it a go.