Hardware transcoding does not work for some vertical videos

[Knallli]

The bug

Hardware transcoding fails on vertical videos since the vertical resolution goes too high on 4k content.

VAAPI reports following resolution constraints on HEVC:

[hevc_vaapi @ 0x5565e27c180] Hardware does not support encoding at size 1080x2344 (constraints: width 256-4096 height 128-2304).

Possible workaround could be to rotate the video for transcoding and then rotate it back on finish no?

The OS that Immich Server is running on

Unraid 6.12.6 w/ Linux 6.8rc3

Version of Immich Server

v1.94.1

Version of Immich Mobile App

Platform with the issue

• [X] Server
• [ ] Web
• [ ] Mobile

Your docker-compose.yml content

version: "3.8"

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /mnt/user/appdata/immich/thumbs:/usr/src/app/upload/thumbs
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    extends:
      file: hwaccel.transcoding.yml 
      service: vaapi
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /mnt/user/appdata/immich/thumbs:/usr/src/app/upload/thumbs
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
      file: hwaccel.ml.yml
      service: openvino
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /mnt/user/appdata/immich/model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: redis:latest
    restart: always

  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.1.11@sha256:0335a1a22f8c5dd1b697f14f079934f5152eaaa216c09b61e293be285491f8ee
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      PG_DATA: /var/lib/postgresql/data
    volumes:
      - /mnt/user/appdata/immich/pgdata:/var/lib/postgresql/data
    restart: always

volumes:
  pgdata:
  model-cache:

Your .env content

DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=xxx
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis
UPLOAD_LOCATION=/mnt/user/main/immich/uploads
TYPESENSE_API_KEY=xxx
PUBLIC_LOGIN_PAGE_MESSAGE=Hello
IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003

Reproduction steps

1. Have a vertical video exceeding the vertical constraint of vaapi
2. Enable HW Transcoding with VAAPI
3. Set transcoding resolution to original 
4. crash hw transcoding by transcoding a vertical video with resolution above the constraint

Additional information

Console output of the FFMPEG error.

[Nest] 7  - 02/05/2024, 4:05:35 PM     LOG [MediaService] Start encoding video a535251b-bdb7-43a8-977b-6212f8954c2c {"inputOptions":["-init_hw_device vaapi=accel:/dev/dri/renderD128","-filter_hw_device accel"],"outputOptions":["-c:v hevc_vaapi","-c:a aac","-movflags faststart","-fps_mode passthrough","-map 0:0","-map 0:1","-g 256","-tag:v hvc1","-v verbose","-vf format=nv12,hwupload","-compression_level 7","-qp 28","-global_quality 28","-rc_mode 1"],"twoPass":false}
[Nest] 7  - 02/05/2024, 4:05:35 PM   ERROR [MediaRepository] ffmpeg version 6.0-Jellyfin Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
[AVHWDeviceContext @ 0x5565e170200] libva: VA-API version 1.18.0
[AVHWDeviceContext @ 0x5565e170200] libva: Trying to open /usr/lib/jellyfin-ffmpeg/lib/dri/radeonsi_drv_video.so
[AVHWDeviceContext @ 0x5565e170200] libva: Found init function __vaDriverInit_1_18
amdgpu: os_same_file_description couldn't determine if two DRM fds reference the same file description.
If they do, bad things may happen!
[AVHWDeviceContext @ 0x5565e170200] libva: va_openDriver() returns 0
[AVHWDeviceContext @ 0x5565e170200] Initialised VAAPI connection: version 1.18
[AVHWDeviceContext @ 0x5565e170200] VAAPI driver: Mesa Gallium driver 23.2.0-devel for AMD Radeon Graphics (gfx1103_r1, LLVM 15.0.6, DRM 3.57, 6.8.0-rc3-thor-Unraid+).
[AVHWDeviceContext @ 0x5565e170200] Driver not found in known nonstandard list, using standard behaviour.
[h264 @ 0x5565e27ad80] Reinit context to 1088x2352, pix_fmt: yuv420p
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'upload/library/eea6a086-0ea0-4706-b3d2-4528dde59942/2019/2019-05-26/19-05-26-04-38-18+1.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    creation_time   : 2019-05-26T02:38:31.000000Z
    com.android.version: 9
  Duration: 00:00:12.10, start: 0.000000, bitrate: 15090 kb/s
  Stream #0:0[0x1](eng): Video: h264 (High), 1 reference frame (avc1 / 0x31637661), yuv420p(tv, smpte170m/bt470bg/smpte170m, progressive, left), 1080x2340 (1088x2352), 14990 kb/s, SAR 1:1 DAR 6:13, 27.43 fps, 45k tbr, 90k tbn (default)
    Metadata:
      creation_time   : 2019-05-26T02:38:31.000000Z
      handler_name    : VideoHandle
      vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, mono, fltp, 96 kb/s (default)
    Metadata:
      creation_time   : 2019-05-26T02:38:31.000000Z
      handler_name    : SoundHandle
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> hevc (hevc_vaapi))
  Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[graph_1_in_0_1 @ 0x5565e1a9840] tb:1/48000 samplefmt:fltp samplerate:48000 chlayout:mono
[h264 @ 0x5565e27d180] Reinit context to 1088x2352, pix_fmt: yuv420p
[graph 0 input from stream 0:0 @ 0x5565e1a6540] w:1080 h:2340 pixfmt:yuv420p tb:1/90000 fr:3735000/136141 sar:1/1
[auto_scale_0 @ 0x5565e0c4280] w:iw h:ih flags:'' interl:0
[Parsed_format_0 @ 0x5565e1a66c0] auto-inserting filter 'auto_scale_0' between the filter 'graph 0 input from stream 0:0' and the filter 'Parsed_format_0'
[auto_scale_0 @ 0x5565e0c4280] w:1080 h:2340 fmt:yuv420p sar:1/1 -> w:1080 h:2340 fmt:nv12 sar:1/1 flags:0x00000004
    Last message repeated 3 times
[hevc_vaapi @ 0x5565e27c180] Using input frames context (format vaapi) with hevc_vaapi encoder.
[hevc_vaapi @ 0x5565e27c180] Input surface format is nv12.
[hevc_vaapi @ 0x5565e27c180] Using VAAPI profile VAProfileHEVCMain (17).
[hevc_vaapi @ 0x5565e27c180] Using VAAPI entrypoint VAEntrypointEncSlice (6).
[hevc_vaapi @ 0x5565e27c180] Using VAAPI render target format YUV420 (0x1).
[hevc_vaapi @ 0x5565e27c180] Using CTU size 64x64, min CB size 8x8.
[hevc_vaapi @ 0x5565e27c180] RC mode: CQP.
[hevc_vaapi @ 0x5565e27c180] RC quality: 28.
[hevc_vaapi @ 0x5565e27c180] RC framerate: 65432/2385 (27.43 fps).
[hevc_vaapi @ 0x5565e27c180] Driver does not report any additional prediction constraints.
[hevc_vaapi @ 0x5565e27c180] Using intra and P-frames (supported references: 1 / 0).
[hevc_vaapi @ 0x5565e27c180] Driver does not support some wanted packed headers (wanted 0xd, found 0x1).
[hevc_vaapi @ 0x5565e27c180] Hardware does not support encoding at size 1080x2344 (constraints: width 256-4096 height 128-2304).
[vost#0:0/hevc_vaapi @ 0x5565e2e7c80] Error initializing output stream: Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[aac @ 0x5565e27cd80] Qavg: 53678.656
[aac @ 0x5565e27cd80] 2 frames left in the queue on closing
[AVIOContext @ 0x5565e1b8700] Statistics: 0 bytes written, 0 seeks, 0 writeouts
Terminating demuxer thread 0
[AVIOContext @ 0x5565e21a2c0] Statistics: 344470 bytes read, 3 seeks
Conversion failed!

[Nest] 7  - 02/05/2024, 4:05:35 PM   ERROR [MediaService] Error: ffmpeg exited with code 1: Conversion failed!

[aviv926]

Can you attach a sample file so we can reproduce the problem?

[Knallli]

https://www.pexels.com/video/road-trip-4434242/

@aviv926 the above sample can reproduce the problem on my end.

Edit: I have also attached the ffmpeg log for the sample and my vainfo vainfo.txt ffmpeg_log.txt

[mertalev]

It seems this is an issue with AMD devices in particular. Intel supports 4096x4096 and I can't see a limit on this for Nvidia. I also don't know if it affects discrete AMD GPUs or just APUs, and if there are different limits for different APUs.

All in all, it might be worth rotating for VAAPI only, but needs more investigation as to the surface area of the issue and how much of a performance hit it would be for devices that don't need this.

[Knallli]

Could this issue be on VAAPI itself? According to the datasheet for my Phoenix iGPU there should be 8k encode, decode support for h265 8/10 bit

[ext4xfs]

https://patchwork.freedesktop.org/patch/577299/ - I believe this is related.

[mertalev]

That does look related. I think we can wait for that to make its way into an FFmpeg release.