NixOS error in encoder thread: Failed to transfer Vulkan image to CUDA frame Generic error in an external library

[JackPala]

On NixOS Latest Unstable I get the following error

error in encoder thread: Failed to transfer Vulkan image to CUDA frame Generic error in an external library

Encoder :

This on an RTX3090 and Ryzen CPU ALVR v20.13.0

[JackPala]

Note : Using the software Force software encoding flag allows it to open at the expense of a lot of performance

[uttarayan21]

Having the same problem on NixOS hw-probe

 [SESSION] Updated
21:44:56.845 [WARNING] Make sure you have put the following line in your SteamVR launch options and restart it:
/home/servius/.local/share/Steam/steamapps/common/SteamVR/bin/vrmonitor.sh %command%
21:44:59.605 [INFO] Server connected
21:45:04.175 [INFO] Initial socket buffer size: send: 87040B, recv: 131072B
21:45:04.188 [WARNING] Chosen refresh rate not supported. Using 72Hz
21:45:04.194 [INFO] Initial socket buffer size: send: 212992B, recv: 212992B
21:45:04.194 [INFO] Set socket send buffer succeeded: 425984
21:45:04.194 [INFO] Set socket recv buffer succeeded: 425984
21:45:04.195 [INFO] Render Target: 3072 1600
21:45:04.195 [INFO] Refresh Rate: 72
21:45:04.195 [INFO] CEncoder::Run
21:45:04.195 [INFO] CEncoder Listening
21:45:04.201 [INFO] CEncoder client connected, pid 495645, cmdline /home/servius/.local/share/Steam/steamapps/common/SteamVR/bin/linux64/vrcompositor
21:45:04.221 [INFO] Received new playspace with size: [1.2020816, 1.2020816]
21:45:04.244 [INFO] Using Vulkan device NVIDIA GeForce RTX 4070 SUPER
21:45:04.464 [INFO] Using device path /dev/dri/renderD128
21:45:04.469 [INFO] FrameRender: Input size 3072x1600
21:45:04.469 [INFO] FrameRender: Output size 1856x832
21:45:04.672 [INFO] Using NvEnc encoder
21:45:04.676 [ERROR] Encoder: cu->cuMemcpy2DAsync(&cpy, cuda_dev->stream) failed
21:45:04.676 [ERROR] Encoder:  -> CUDA_ERROR_INVALID_VALUE: invalid argument
21:45:04.676 [ERROR] Encoder: 
21:45:05.022 [ERROR] error in encoder thread: Failed to transfer Vulkan image to CUDA frame Generic error in an external library
21:45:18.387 [INFO] Received new playspace with size: [1.2020816, 1.2020816]
21:45:18.388 [INFO] Received new playspace with size: [1.2020816, 1.2020816]
21:45:18.389 [INFO] Received new playspace with size: [1.2020816, 1.2020816]
21:45:34.696 [INFO] Client disconnected. Cause: Broken pipe (os error 32)
21:49:11.593 [INFO] Server disconnected
21:49:12.005 [WARNING] Make sure you have put the following line in your SteamVR launch options and restart it:
/home/servius/.local/share/Steam/steamapps/common/SteamVR/bin/vrmonitor.sh %command%
21:49:14.595 [INFO] Server connected
21:49:19.090 [INFO] Initial socket buffer size: send: 87040B, recv: 131072B
21:49:19.110 [WARNING] Chosen refresh rate not supported. Using 72Hz
21:49:19.117 [INFO] Initial socket buffer size: send: 212992B, recv: 212992B
21:49:19.117 [INFO] Set socket send buffer succeeded: 425984
21:49:19.117 [INFO] Set socket recv buffer succeeded: 425984
21:49:19.118 [INFO] Render Target: 3072 1600
21:49:19.118 [INFO] Refresh Rate: 72
21:49:19.118 [INFO] CEncoder::Run
21:49:19.118 [INFO] CEncoder Listening
21:49:19.126 [INFO] CEncoder client connected, pid 500418, cmdline /home/servius/.local/share/Steam/steamapps/common/SteamVR/bin/linux64/vrcompositor
21:49:19.142 [INFO] Received new playspace with size: [1.2020816, 1.2020816]
21:49:19.172 [INFO] Using Vulkan device NVIDIA GeForce RTX 4070 SUPER
21:49:19.418 [INFO] Using device path /dev/dri/renderD128
21:49:19.420 [INFO] FrameRender: Input size 3072x1600
21:49:19.421 [INFO] FrameRender: Output size 1856x832
21:49:19.465 [INFO] x264: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
21:49:19.467 [INFO] x264: profile Constrained Baseline, level 4.2, 4:2:0, 8-bit
21:49:19.479 [INFO] Using SW encoder
21:49:19.551 [INFO] Received new playspace with size: [1.2020816, 1.2020816]

Have tried on both stable and nightly builds and doesn't work on either of them

[MatthewCroughan]

Same on NixOS

16:27:59.544 WARNING Make sure you have put the following line in your SteamVR launch options and restart it:
/home/doesuser/.local/share/Steam/steamapps/common/SteamVR/bin/vrmonitor.sh %command%
16:27:59.564 INFO GPU supports H264 profile.
16:27:59.564 INFO GPU supports HEVC profile.
16:28:02.144 INFO Server connected
16:28:17.150 INFO Initial socket buffer size: send: 87040B, recv: 131072B
16:28:17.189 INFO Initial socket buffer size: send: 212992B, recv: 212992B
16:28:17.189 INFO Set socket send buffer succeeded: 425984
16:28:17.189 INFO Set socket recv buffer succeeded: 425984
16:28:17.191 INFO Render Target: 4288 2240
16:28:17.191 INFO Refresh Rate: 72
16:28:17.221 INFO Received new playspace with size: [2.7815573, 4.188445]
16:28:18.039 INFO CEncoder::Run
16:28:18.039 INFO CEncoder Listening
16:28:18.046 INFO CEncoder client connected, pid 10327, cmdline /home/doesuser/.local/share/Steam/steamapps/common/SteamVR/bin/linux64/vrcompositor
16:28:18.198 INFO Using Vulkan device NVIDIA GeForce RTX 3060
16:28:18.625 INFO Using device path /dev/dri/renderD128
16:28:18.685 INFO FrameRender: Input size 4288x2240
16:28:18.694 INFO FrameRender: Output size 2560x1184
16:28:19.042 INFO Using NvEnc encoder
16:28:19.046 ERROR Encoder: cu->cuMemcpy2DAsync(&cpy, cuda_dev->stream) failed
16:28:19.048 ERROR Encoder:  -> CUDA_ERROR_INVALID_VALUE: invalid argument
16:28:19.048 ERROR Encoder: 
16:28:19.367 ERROR error in encoder thread: Failed to transfer Vulkan image to CUDA frame Generic error in an external library
16:28:28.203 INFO Server disconnected

[MattheDev53]

Same here

Edit: from what I can see, it's having issues using NVENC. I've had similar problems with NVENC on KDENLiVE, so it might just be a NixOS bug.

 [SESSION] Updated
11:59:46.278 [WARNING] Make sure you have put the following line in your SteamVR launch options and restart it:
/home/magames/.local/share/Steam/steamapps/common/SteamVR/bin/vrmonitor.sh %command%
11:59:49.141 [INFO] Server connected
12:05:07.045 [INFO] Initial socket buffer size: send: 87040B, recv: 131072B
12:05:07.119 [INFO] Initial socket buffer size: send: 212992B, recv: 212992B
12:05:07.119 [INFO] Set socket send buffer succeeded: 425984
12:05:07.119 [INFO] Set socket recv buffer succeeded: 425984
12:05:07.121 [INFO] Render Target: 4288 2336
12:05:07.121 [INFO] Refresh Rate: 60
12:05:07.121 [INFO] CEncoder::Run
12:05:07.121 [INFO] CEncoder Listening
12:05:07.182 [INFO] Received new playspace with size: [1.2020816, 1.2020816]
12:05:07.205 [INFO] CEncoder client connected, pid 395899, cmdline /home/magames/.local/share/Steam/steamapps/common/SteamVR/bin/linux64/vrcompositor
12:05:07.497 [INFO] Using Vulkan device NVIDIA GeForce RTX 3060 Ti
12:05:07.721 [INFO] Using device path /dev/dri/renderD128
12:05:07.726 [INFO] FrameRender: Input size 4288x2336
12:05:07.726 [INFO] FrameRender: Output size 2560x1216
12:05:07.881 [INFO] Using NvEnc encoder
12:05:07.883 [ERROR] Encoder: cu->cuMemcpy2DAsync(&cpy, cuda_dev->stream) failed
12:05:07.883 [ERROR] Encoder:  -> CUDA_ERROR_INVALID_VALUE: invalid argument
12:05:07.883 [ERROR] Encoder: 
12:05:08.250 [ERROR] error in encoder thread: Failed to transfer Vulkan image to CUDA frame Generic error in an external library

[syshotdev]

Did NVidia encoding ever work on a previous version of NixOS?

[syshotdev]

On NixOS 25.05, I get this error as well.

I tested it, and Nvidia encoding only works with NixOS 24.05, no newer. This means the problem is between NixOS 24.05 and 24.11, probably library or graphics driver changes.

Link to development repository: https://github.com/syshotdev/nix-vr/tree/502586a00b6d17f989a7b903235745ca1452bc4c

[armin-th]

I am also using nixos, and also confirmed when I roll ALVR back to 20.6.1 (which is what is in nixos-24.05 channel) it runs fine but using the later nixos channels 25.05 (using ALVR 20.14.0) and likely 24.11 fails with this issue's exact error.

Glad I'm not the only!

[armin-th]

Hello again, I just followed https://nix.dev/guides/faq#how-to-run-non-nix-executables

added to my nixos

programs.nix-ld.enable = true;

programs.alvr = {
    enable = true;
    openFirewall = true; # we still need the firewall rules
}

then from the extracted release alvr_streamer_linux.tar.gz

nix-shell -p steam-run --run "steam-run ./alvr_dashboard"

and I no longer get this issue and can stream over wifi

This leads me to believe we should open an issue either in nixos or nixpkgs repos, not here.

Hope this helps!

[Shringe]

I can't replicate using steam-run ./alvr_dashboard to fix the issue. The only things I have found to "fix" the issue so far is to force software encoding, or to use ALVR 20.6.1 from Nixpkgs 24.05.

[CharlesAttend]

Any news about that? I spent my evening on it, but to no avail :( I can't launch SteamVR with ALVR 20.6.1 so I couldn't test this solution

I've been trying NixOS for a while, but this is the last straw. I have so many unsolved problems with this distro 😭

EDIT: As you can see I was going to give up but then I saw a version missmatch between my quest 3 and the computer. On device apk was 20.14.0 and my computer was 20.14.1. I switch version on my computer using ASWR laucher and then it work ! Didn't try to launch a game yet

[SoanGlengan]

also having this issue

[Pablo1107]

Still an issue. Workaround for me was to use the official release binaries with steam-run as some suggested here.