llvmpipe used instead of actual GPU, despite Vulkan-capable hardware GPU being detected.

[Cat-Lady]

Hello, Judging by the logs and CPU usage (100%) vs GPU usage (1%), video2x tries to use lavapipe instead of actual GPU, when trying to upscale via CLI (notice the WARNING: lavapipe is not a conformant vulkan implementation, testing use only. before actual upscale process starts):

12:47:58.881069 | INFO     | Video2X 5.0.0-beta5
12:47:58.881224 | INFO     | Copyright (C) 2018-2022 K4YT3X and contributors.
12:47:58.881312 | INFO     | Reading input video information
12:47:59.002920 | INFO     | Starting video decoder
12:47:59.008225 | INFO     | Starting video encoder
12:47:59.014432 | INFO     | Upscaler process 0 initiating
12:47:59.015706 | INFO     | Upscaler process 1 initiating
12:47:59.017466 | INFO     | Upscaler process 2 initiating
12:47:59.019731 | INFO     | Upscaler process 3 initiating
12:47:59.021330 | INFO     | Upscaler process 4 initiating
[0 llvmpipe (LLVM 13.0.1, 256 bits)]  queueC=0[1]  queueG=0[1]  queueT=0[1]
[0 llvmpipe (LLVM 13.0.1, 256 bits)]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=0
[0 llvmpipe (LLVM 13.0.1, 256 bits)]  fp16-p/s/a=1/1/1  int8-p/s/a=1/1/1
[0 llvmpipe (LLVM 13.0.1, 256 bits)]  subgroup=8  basic=1  vote=1  ballot=1  shuffle=0
[1 NVIDIA GeForce GTX 1660 Ti]  queueC=2[8]  queueG=0[16]  queueT=1[2]
[1 NVIDIA GeForce GTX 1660 Ti]  bugsbn1=0  bugbilz=0  bugcopc=0  bugihfa=0
[1 NVIDIA GeForce GTX 1660 Ti]  fp16-p/s/a=1/1/1  int8-p/s/a=1/1/1
[1 NVIDIA GeForce GTX 1660 Ti]  subgroup=32  basic=1  vote=1  ballot=1  shuffle=1

WARNING: lavapipe is not a conformant vulkan implementation, testing use only.
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'test.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 1
    compatible_brands: isomiso2avc1mp41mp423gp5
    encoder         : Lavf55.33.100
  Duration: 00:07:47.37, start: 0.000000, bitrate: 501 kb/s
  Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 400x300, 375 kb/s, 24 fps, 24 tbr, 12288 tbn, 48 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
  Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 22050 Hz, stereo, fltp, 96 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
  Stream #0:2(und): Data: none (mp4s / 0x7334706D), 0 kb/s (default)
    Metadata:
      creation_time   : 2015-04-01T19:11:05.000000Z
      handler_name    : GPAC MPEG-4 OD Handler
  Stream #0:3(und): Data: none (mp4s / 0x7334706D), 0 kb/s (default)
    Metadata:
      creation_time   : 2015-04-01T19:11:05.000000Z
      handler_name    : GPAC MPEG-4 Scene Description Handler
  Stream #0:4(und): Data: none (rtp  / 0x20707472), 20 kb/s (default)
    Metadata:
      creation_time   : 2015-04-01T19:11:05.000000Z
      handler_name    : GPAC ISO Hint Handler
  Stream #0:5(und): Data: none (rtp  / 0x20707472), 5 kb/s (default)
    Metadata:
      creation_time   : 2015-04-01T19:11:05.000000Z
      handler_name    : GPAC ISO Hint Handler
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> rawvideo (native))
Press [q] to stop, [?] for help
Output #0, rawvideo, to 'pipe:1':
  Metadata:
    major_brand     : mp42
    minor_version   : 1
    compatible_brands: isomiso2avc1mp41mp423gp5
    encoder         : Lavf58.76.100
  Stream #0:0(und): Video: rawvideo (RGB[24] / 0x18424752), rgb24(pc, gbr/unknown/unknown, progressive), 400x300, q=2-31, 69120 kb/s, 24 fps, 24 tbn 
(default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc58.134.100 rawvideo

I tried to find any documentation on how to pass which GPU to use, but found absolutely no info about it (the documentation is "lacking", to say it very, very kindly). I also tried to get info about arguments accepted for upscale algorithm python -m video2x -a waifu2x -- --help as per documentation (also tried python -m video2x -a upscale -- --help, but none of it works as "advertised", throwing errors about "invalid choice".

I am specifically interested in using video2x locally on the linux machine (not through Google colab), docker-less.

Cheers, /CatLady

[k4yt3x]

Hi there, 5.0.0 is still in its early stages and I haven't baked a lot of the functions in yet (e.g., model selection and FFmpeg command customizations). They'll be coming in later beta versions. In fact model selection is already implemented on my local branch, and multi-GPU is already queued. Although a lot of the code has already been written, I might still overhaul the project yet again to make it compatible with k8s clusters (which requires components to be less coupled).

Usually speaking, the default GPU selected by *-ncnn-vulkan is the most powerful GPU on your machine, so I left it til now. I'll add an option soon to allow users to change it. I still don't know how to get a reliable list of Vulkan-compatible GPUs and their corresponding IDs... so you'll have to run the program and see the output to see what the GPUs are and their IDs.

[Cat-Lady]

Hello, Thank you for the answer. I was suspecting that - due to the fact that lavapipe is virtual vulkan implementation on CPU - it might be reporting "better" capabilities than real GPU.

For now, I was able to bypass the issue by checking Vulkan ICDs (ls /usr/share/vulkan/icd.d), and manually selecting the one(s) I want to use by export VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.json in my case). it can be, actually, a comma-separated list of ICDs exported as this env variable. Anything that doesn't get exported will become "invisible" to video2x (or any vulkan-needing application started after this env is set), so virtual or unwanted ones can get "hidden" this way.

BTW, I think that parsing ICDs might be your answer to find Vulkan-capable GPUs, at least on linux-based machines.

Cheers, /CatLady

[nihui]

    if (gpuid.empty())
    {
        gpuid.push_back(ncnn::get_default_gpu_index());
    }

In xxx-ncnn-vulkan, we use the default gpu if not specified, that should be discrete gpu first, integrated gpu next and others last.

[k4yt3x]

It seems like ncnn is just doing a brute-force search over all of the devices to find the ones that support Vulkan and then number them according to the order of which they're found. I think I can either make a FFI for ncnn or implement the same algorithm in v2x to obtain the same list.

  // find proper device and queue
  int gpu_info_index = 0;
  for (uint32_t i = 0; i < physicalDeviceCount; i++)
  {
      const VkPhysicalDevice& physicalDevice = physicalDevices[i];
      delete g_gpu_infos[gpu_info_index];
      g_gpu_infos[gpu_info_index] = new GpuInfo;
      GpuInfoPrivate& gpu_info = *(g_gpu_infos[gpu_info_index]->d);

https://github.com/Tencent/ncnn/blob/master/src/gpu.cpp#L1031-L1038

For the record, this function returns the first available GPU in the list, with dGPUs prioritized over iGPUs:

static int find_default_vulkan_device_index()
{
    // first try, discrete gpu
    for (int i = 0; i < g_gpu_count; i++)
    {
        if (g_gpu_infos[i]->type() == 0)
            return i;
    }

    // second try, integrated gpu
    for (int i = 0; i < g_gpu_count; i++)
    {
        if (g_gpu_infos[i]->type() == 1)
            return i;
    }

    // third try, any probed device
    if (g_gpu_count > 0)
        return 0;

    NCNN_LOGE("no vulkan device");
    return -1;
}

https://github.com/Tencent/ncnn/blob/master/src/gpu.cpp#L810-L832

It would be nice though if there's a stable API provided by ncnn or Vulkan that will produce a list of available devices like:

import vulkan

devices: list = vulkan.get_devices()
assert devices == [{"id": 0, "name": "Intel UHD Graphics 620"}]

[ArchieMeng]

How about using this demo to get available vulkan devices? I tested it on my devices, and it works. But it has not yet been tested on multi-GPU devices.

import vulkan as vk
import cffi


ffi = cffi.FFI()
inst_info = vk.VkInstanceCreateInfo()
device_types = [
        "VK_PHYSICAL_DEVICE_TYPE_OTHER",
        "VK_PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU",
        "VK_PHYSICAL_DEVICE_TYPE_DISCRETE_GPU",
        "VK_PHYSICAL_DEVICE_TYPE_VIRTUAL_GPU",
        "VK_PHYSICAL_DEVICE_TYPE_CPU",
]

# Vulkan functions usually return a VkResult, 
# which returns the success and error codes/states 
# of the function. 
# vulkan is pythonic and converts VkResult to exception: 
# if the result is not VK_SUCCESS, an exception is raised.
inst = vk.vkCreateInstance(inst_info, None)
devices = vk.vkEnumeratePhysicalDevices(inst)
for i, device in enumerate(devices):
    device_properties = vk.VkPhysicalDeviceProperties()
    vk.vkGetPhysicalDeviceProperties(device, device_properties)
    print(f"device_index={i}")
    print(f"deviceName=\"{ffi.string(device_properties.deviceName).decode()}\"")
    print(f"deviceType=\"{device_types[device_properties.deviceType]}\"")
    print()

vk.vkDestroyInstance(inst, None)

[k4yt3x]

@ArchieMeng this looks awesome, should be pretty close to what it needs to be. I'll try it when I have time (which is probably after this semester is over).

[knorr-goose]

Hello, Thank you for the answer. I was suspecting that - due to the fact that lavapipe is virtual vulkan implementation on CPU - it might be reporting "better" capabilities than real GPU.

For now, I was able to bypass the issue by checking Vulkan ICDs (ls /usr/share/vulkan/icd.d), and manually selecting the one(s) I want to use by export VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.json in my case). it can be, actually, a comma-separated list of ICDs exported as this env variable. Anything that doesn't get exported will become "invisible" to video2x (or any vulkan-needing application started after this env is set), so virtual or unwanted ones can get "hidden" this way.

BTW, I think that parsing ICDs might be your answer to find Vulkan-capable GPUs, at least on linux-based machines.

Cheers, /CatLady

Hi!! I am running Kubuntu LTS 20.04. My GTX 1080 Ti is not being used by Video2x. I have tried to use your comments to force my GPU usage, but my attempt was unsuccessful. I invoke video2x using docker CE, by command line. Could you please, be so gentle to help me to get my goal, if as I understand, it is posible. I also understand that in the future, author will provide a way to detect and use GPU automatically, or so. Thanks in advance... Norberto !!