feat(server): trancoding hardware acceleration

[mertalev]

Description

WIP support for NVENC, Quick Sync and VAAPI transcoding backends. This will make transcoding much faster at the cost of lower quality at the same bitrate compared to software encoding.

Notes:

• Some options require uncommenting sections in the docker-compose.yml, so be sure to check that yours looks correct.
• This uses a Debian-based image, so if running in a dev environment you should delete your current immich-microservices container and the dist and node_modules folders relating to it. Ideally the immich-server container should also be using Dockerfile.accel to avoid issues. Otherwise you might see errors stemming from it trying to load Alpine binaries.

How Has This Been Tested?

Current testing in production build:

• NVENC
  • ✅ HEVC
  • ✅ H.264
  • ❌ VP9 does not work (not supported)
• VAAPI
  • ✅ H.264
  • 🤷🏽 HEVC
  • 🤷🏽 VP9
• QSV
  • 🤷🏽

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
immich	⬜️ Ignored (Inspect)	Visit Preview		Aug 2, 2023 0:34am

[martabal]

Does Dockerfile.accel support raw images ? It's just for development ?

[mertalev]

The image has to support image formats like RAW since it's responsible for thumbnail generation.

[martabal]

Does it work ? With this image we have to build imagemagick & libvips to support raw images.

[martabal]

And you can possibly have a version mismatch between libvips & sharp. Currently immich uses sharp v31.3 which requires libvips v8.13.3 and bookworm provides libvips v8.14.1-3, so it's not an issue for now

[mertalev]

It seems Imagemagick can handle raw images, but sharp throws an error. I can try compiling libvips as well to see if that helps.

[mertalev]

I can confirm RAW images work now after compiling libvips. The image is also a decent bit smaller at 1.07gb instead of 1.22gb.

[samip5]

I tried this with Quick Sync, and well, libva errors on 6th gen Intel :(

# ls -al /dev/dri/
crw-rw----    1 root     44        226,   0 Jul 17 20:22 card0
crw-rw----    1 root     109       226, 128 Jul 17 20:22 renderD128

[Nest] 7  - 07/17/2023, 10:20:31 PM     LOG [MediaService] Start encoding video 5fceb8ef-581b-43ea-9610-951b7675f0ee {"inputOptions":["-init_hw_device qsv=accel:/dev/dri/renderD128","-filter_hw_device accel"],"outputOptions":["-vcodec h264_qsv","-acodec aac","-movflags faststart","-fps_mode passthrough","-vf format=nv12,hwupload=extra_hw_frames=64","-preset 7","-global_quality 23"],"twoPass":false}
[Nest] 7  - 07/17/2023, 10:20:31 PM   ERROR [MediaRepository] ffmpeg version 6.0 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 12.2.1 (Alpine 12.2.1_git20220924-r10) 20220924
  configuration: --prefix=/usr --disable-librtmp --disable-lzma --disable-static --disable-stripping --enable-avfilter --enable-gnutls --enable-gpl --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libdrm --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libmp3lame --enable-libopenmpt --enable-libopus --enable-libplacebo --enable-libpulse --enable-librist --enable-libsoxr --enable-libsrt --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-lto --enable-pic --enable-postproc --enable-pthreads --enable-shared --enable-vaapi --enable-vdpau --enable-vulkan --optflags=-O3 --enable-libjxl --enable-libsvtav1 --enable-libvpl
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   5.  0.100 /  5.  0.100
  libpostproc    57.  1.100 / 57.  1.100
[AVHWDeviceContext @ 0x7fc4ffe2ef00] Failed to initialise VAAPI connection: -1 (unknown libva error).
Device creation failed: -5.
Failed to set value 'qsv=accel:/dev/dri/renderD128' for option 'init_hw_device': I/O error
Error parsing global options: I/O error

[Nest] 7  - 07/17/2023, 10:20:31 PM   ERROR [MediaService] Error: ffmpeg exited with code 1: Device creation failed: -5.
Failed to set value 'qsv=accel:/dev/dri/renderD128' for option 'init_hw_device': I/O error
Error parsing global options: I/O error

[mertalev]

From that log it's using Alpine binaries. For the dev environment, I think you need to set both immich-server and immich-microservices to use Dockerfile.accel and rebuild the containers.

(That should probably be the default for dev, though the bigger conversation to have is whether this image should replace the existing Alpine one or be an alternate version.)

Also double check that you have the hwaccel.yml and enabled the renderD128 device on it.

Edit: It actually does seem to be the default, so I guess it's something else after all.

[samip5]

From that log it's using Alpine binaries. For the dev environment, I think you need to set both immich-server and immich-microservices to use Dockerfile.accel and rebuild the containers.

(That should probably be the default for dev, though the bigger conversation to have is whether this image should replace the existing Alpine one or be an alternate version.)

Also double check that you have the hwaccel.yml and enabled the renderD128 device on it.

I'm using the ghcr.io/immich-app/immich-server:pr-3171 image tag though?

Can be seen here

[jrasm91]

Does the PR build push hardware accelerated enabled images?

[samip5]

Does the PR build push hardware accelerated enabled images?

The option exists in settings, but it seems to be alpine based images only.

[mertalev]

It seems GH was building with the Alpine-based Dockerfile before this. I moved Dockerfile.accel to Dockerfile for now to avoid the headache of managing both.

[mertalev]

As of the latest version of this PR, I can confirm all three APIs work when used on a supporting system and the hwaccel.yml is configured.

[alextran1502]

I just run the first test uploading from iPhone and ran into this issue with websocket.

[mertalev]

Hmm that's odd, I didn't touch that part of the codebase at all.

[alextran1502]

@mertalev Agree, might it be the changed in the node distro? Do you see the same thing when upload from the mobile app?

[mertalev]

I can't seem to reproduce this issue on iOS. I can upload and view images without any errors.

[alextran1502]

I can't seem to reproduce this issue on iOS. I can upload and view images without any errors.

Hmm let me try again, I can confirm that the upload is success and I can view the image, just a bunch of those errors showing up

[martabal]

Hey @mertalev can you store the libvips/imagemagick version in a txt/json file for easy parsing ?

[mertalev]

I made a build-lock.json file for build versions and sha256.

[martabal]

I made a build-lock.json file for build versions and sha256.

This is perfect. Thanks !

[mertalev]

Looks complicated AF! 😄

There was a brief moment where I thought I'd just need to add -hwaccel auto and it would automagically take care of everything... I wish.

[mertalev]

Tested QSV on a 13th gen CPU. For Quick Sync support for 12th gen and above, it seems FFmpeg needs to be built with libvpl. The current version of FFmpeg in Debian Bookworm is built with libmfx instead so it doesn't work on these CPUs.

How do we want to handle this? We could install from a different repo or maybe even use Jellyfin's own FFmpeg package, or we could just add it to the list of limitations in the docs for now.

[mertalev]

I can confirm that switching to Jellyfin's FFmpeg build fixed the issue with Quick Sync (and shrunk the image down to under 1GiB). VP9 also works in Quick Sync. With that, the only outstanding issue seems to be the errors @alextran1502 got. If we can resolve that, it otherwise seems good. Maybe a rebase could fix it?

[mertalev]

Rebased