GPU Operator fails to install into IBM Cloud OpenShift service (OCP v4.14)

Open darrellschrag opened this issue 1 year ago • 8 comments

1. Quick Debug Information

OS/Version(e.g. RHEL8.6, Ubuntu22.04): RHEL 8.8
Kernel Version: .18.0-477.27.1.el8_8.x86_64
Container Runtime Type/Version(e.g. Containerd, CRI-O, Docker): CRI-O
K8s Flavor/Version(e.g. K8s, OCP, Rancher, GKE, EKS): OCP
GPU Operator Version: v23.9.1

2. Issue or feature description

The driver install completes but fails with no signal. I also get a GPUOperatorOpenshiftDriverToolkitEnabledNfdTooOld event.

3. Steps to reproduce the issue

Provision a new OCP 4.14 cluster in IBM Cloud
Install the NFD Operator and an NFD instance
validate that the worker nodes have been tagged properly to indicate GPU worker nodes
Install the GPU operator and default cluster policy

4. Information to attach (optional if deemed irrelevant)

(⎈|dal2gpu:default)➜ rhods-da git:(main) ✗ kk get po -n nvidia-gpu-operator NAME READY STATUS RESTARTS AGE gpu-feature-discovery-pg42n 0/1 Init:0/1 0 5d15h gpu-feature-discovery-tllwf 0/1 Init:0/1 0 5d15h gpu-operator-6559b97ffb-kbc67 1/1 Running 0 5d15h nvidia-container-toolkit-daemonset-94xl2 0/1 Init:0/1 0 5d15h nvidia-container-toolkit-daemonset-w8ltf 0/1 Init:0/1 0 5d15h nvidia-dcgm-exporter-4mc4z 0/1 Init:0/1 0 5d15h nvidia-dcgm-exporter-fhmxm 0/1 Init:0/1 0 5d15h nvidia-dcgm-vd95v 0/1 Init:0/1 0 5d15h nvidia-dcgm-zqc2p 0/1 Init:0/1 0 5d15h nvidia-device-plugin-daemonset-54g5l 0/1 Init:0/1 0 5d15h nvidia-device-plugin-daemonset-8plm6 0/1 Init:0/1 0 5d15h nvidia-driver-daemonset-cdd7x 0/1 Running 382 (18m ago) 5d15h nvidia-driver-daemonset-x9fms 0/1 Running 382 (12m ago) 5d15h nvidia-node-status-exporter-cmxvg 1/1 Running 0 5d15h nvidia-node-status-exporter-m5twj 1/1 Running 0 5d15h nvidia-operator-validator-245qd 0/1 Init:0/4 0 5d15h nvidia-operator-validator-29sb2 0/1 Init:0/4 0 5d15h

(⎈|dal2gpu:default)➜ rhods-da git:(main) ✗ kk get ds -n nvidia-gpu-operator NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE gpu-feature-discovery 2 2 0 2 0 nvidia.com/gpu.deploy.gpu-feature-discovery=true 5d15h nvidia-container-toolkit-daemonset 2 2 0 2 0 nvidia.com/gpu.deploy.container-toolkit=true 5d15h nvidia-dcgm 2 2 0 2 0 nvidia.com/gpu.deploy.dcgm=true 5d15h nvidia-dcgm-exporter 2 2 0 2 0 nvidia.com/gpu.deploy.dcgm-exporter=true 5d15h nvidia-device-plugin-daemonset 2 2 0 2 0 nvidia.com/gpu.deploy.device-plugin=true 5d15h nvidia-driver-daemonset 2 2 0 2 0 nvidia.com/gpu.deploy.driver=true 5d15h nvidia-mig-manager 0 0 0 0 0 nvidia.com/gpu.deploy.mig-manager=true 5d15h nvidia-node-status-exporter 2 2 2 2 2 nvidia.com/gpu.deploy.node-status-exporter=true 5d15h nvidia-operator-validator 2 2 0 2 0 nvidia.com/gpu.deploy.operator-validator=true 5d15h

(⎈|dal2gpu:default)➜ rhods-da git:(main) ✗ kk describe po nvidia-driver-daemonset-cdd7x -n nvidia-gpu-operator Name: nvidia-driver-daemonset-cdd7x Namespace: nvidia-gpu-operator Priority: 2000001000 Priority Class Name: system-node-critical Service Account: nvidia-driver Node: 10.240.0.11/10.240.0.11 Start Time: Fri, 29 Dec 2023 18:52:56 -0600 Labels: app=nvidia-driver-daemonset app.kubernetes.io/component=nvidia-driver controller-revision-hash=66b94cb756 nvidia.com/precompiled=false pod-template-generation=1 Annotations: cni.projectcalico.org/containerID: b3a0a527352d2b383a3d5caf378b682cfe20562e48843b30f5b3fec7798cfdb9 cni.projectcalico.org/podIP: 172.17.0.73/32 cni.projectcalico.org/podIPs: 172.17.0.73/32 k8s.v1.cni.cncf.io/network-status: [{ "name": "k8s-pod-network", "ips": [ "172.17.0.73" ], "default": true, "dns": {} }] kubectl.kubernetes.io/default-container: nvidia-driver-ctr openshift.io/scc: nvidia-driver Status: Running IP: 172.17.0.73 IPs: IP: 172.17.0.73 Controlled By: DaemonSet/nvidia-driver-daemonset Init Containers: k8s-driver-manager: Container ID: cri-o://6b4298980f20a16070b917a12cb1a194e2c07a99473e0b08de7f9b063a98b4dd Image: nvcr.io/nvidia/cloud-native/k8s-driver-manager@sha256:27c44f4720a4abf780217bd5e7903e4a008ebdbcf71238c4f106a0c22654776c Image ID: nvcr.io/nvidia/cloud-native/k8s-driver-manager@sha256:27c44f4720a4abf780217bd5e7903e4a008ebdbcf71238c4f106a0c22654776c Port: Host Port: Command: driver-manager Args: uninstall_driver State: Terminated Reason: Completed Exit Code: 0 Started: Fri, 29 Dec 2023 18:53:09 -0600 Finished: Fri, 29 Dec 2023 18:53:43 -0600 Ready: True Restart Count: 0 Environment: NODE_NAME: (v1:spec.nodeName) NVIDIA_VISIBLE_DEVICES: void ENABLE_GPU_POD_EVICTION: true ENABLE_AUTO_DRAIN: true DRAIN_USE_FORCE: false DRAIN_POD_SELECTOR_LABEL: DRAIN_TIMEOUT_SECONDS: 0s DRAIN_DELETE_EMPTYDIR_DATA: false OPERATOR_NAMESPACE: nvidia-gpu-operator (v1:metadata.namespace) Mounts: /host from host-root (ro) /run/nvidia from run-nvidia (rw) /sys from host-sys (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-x98cm (ro) Containers: nvidia-driver-ctr: Container ID: cri-o://6c1aca6e737f3c5e9cdda52b36c0a386aa9368f7bca4e93f124128a2f055c2c6 Image: nvcr.io/nvidia/driver@sha256:028cc6cf5944c0df4fbee8fe0f45654c09996ac13b5b37fb803d79ae2d37d378 Image ID: nvcr.io/nvidia/driver@sha256:028cc6cf5944c0df4fbee8fe0f45654c09996ac13b5b37fb803d79ae2d37d378 Port: Host Port: Command: nvidia-driver Args: init State: Running Started: Thu, 04 Jan 2024 09:50:17 -0600 Last State: Terminated Reason: Completed Exit Code: 0 Started: Thu, 04 Jan 2024 09:29:07 -0600 Finished: Thu, 04 Jan 2024 09:50:17 -0600 Ready: False Restart Count: 382 Startup: exec [sh -c nvidia-smi && touch /run/nvidia/validations/.driver-ctr-ready] delay=60s timeout=60s period=10s #success=1 #failure=120 Environment: Mounts: /dev/log from dev-log (rw) /host-etc/os-release from host-os-release (ro) /lib/firmware from nv-firmware (rw) /run/mellanox/drivers from run-mellanox-drivers (rw) /run/mellanox/drivers/usr/src from mlnx-ofed-usr-src (rw) /run/nvidia from run-nvidia (rw) /run/nvidia-topologyd from run-nvidia-topologyd (rw) /sys/devices/system/memory/auto_online_blocks from sysfs-memory-online (rw) /sys/module/firmware_class/parameters/path from firmware-search-path (rw) /var/log from var-log (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-x98cm (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: run-nvidia: Type: HostPath (bare host directory volume) Path: /run/nvidia HostPathType: DirectoryOrCreate var-log: Type: HostPath (bare host directory volume) Path: /var/log HostPathType: dev-log: Type: HostPath (bare host directory volume) Path: /dev/log HostPathType: host-os-release: Type: HostPath (bare host directory volume) Path: /etc/os-release HostPathType: run-nvidia-topologyd: Type: HostPath (bare host directory volume) Path: /run/nvidia-topologyd HostPathType: DirectoryOrCreate mlnx-ofed-usr-src: Type: HostPath (bare host directory volume) Path: /run/mellanox/drivers/usr/src HostPathType: DirectoryOrCreate run-mellanox-drivers: Type: HostPath (bare host directory volume) Path: /run/mellanox/drivers HostPathType: DirectoryOrCreate run-nvidia-validations: Type: HostPath (bare host directory volume) Path: /run/nvidia/validations HostPathType: DirectoryOrCreate host-root: Type: HostPath (bare host directory volume) Path: / HostPathType: host-sys: Type: HostPath (bare host directory volume) Path: /sys HostPathType: Directory firmware-search-path: Type: HostPath (bare host directory volume) Path: /sys/module/firmware_class/parameters/path HostPathType: sysfs-memory-online: Type: HostPath (bare host directory volume) Path: /sys/devices/system/memory/auto_online_blocks HostPathType: nv-firmware: Type: HostPath (bare host directory volume) Path: /run/nvidia/driver/lib/firmware HostPathType: DirectoryOrCreate kube-api-access-x98cm: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: QoS Class: BestEffort Node-Selectors: nvidia.com/gpu.deploy.driver=true Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists node.kubernetes.io/pid-pressure:NoSchedule op=Exists node.kubernetes.io/unreachable:NoExecute op=Exists node.kubernetes.io/unschedulable:NoSchedule op=Exists nvidia.com/gpu:NoSchedule op=Exists Events: Type Reason Age From Message

Warning Unhealthy 17m (x4703 over 5d15h) kubelet Startup probe failed: NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running. Warning Unhealthy 2m6s (x41235 over 5d15h) kubelet Startup probe failed: No devices were found

(⎈|dal2gpu:default)➜ rhods-da git:(main) ✗ kk logs nvidia-driver-daemonset-cdd7x -n nvidia-gpu-operator

set -eu
RUN_DIR=/run/nvidia
PID_FILE=/run/nvidia/nvidia-driver.pid
DRIVER_VERSION=535.129.03
KERNEL_UPDATE_HOOK=/run/kernel/postinst.d/update-nvidia-driver
NUM_VGPU_DEVICES=0
NVIDIA_MODULE_PARAMS=()
NVIDIA_UVM_MODULE_PARAMS=()
NVIDIA_MODESET_MODULE_PARAMS=()
NVIDIA_PEERMEM_MODULE_PARAMS=()
TARGETARCH=amd64
USE_HOST_MOFED=false
DNF_RELEASEVER=
OPEN_KERNEL_MODULES_ENABLED=false
[[ false == \t\r\u\e ]]
KERNEL_TYPE=kernel
DRIVER_ARCH=x86_64
DRIVER_ARCH=x86_64
echo 'DRIVER_ARCH is x86_64' DRIVER_ARCH is x86_64 +++ dirname -- /usr/local/bin/nvidia-driver ++ cd -- /usr/local/bin ++ pwd
SCRIPT_DIR=/usr/local/bin
source /usr/local/bin/common.sh ++ GPU_DIRECT_RDMA_ENABLED=false ++ GDS_ENABLED=false
'[' 1 -eq 0 ']'
command=init
shift
case "${command}" in ++ getopt -l accept-license -o a --
options=' --'
'[' 0 -ne 0 ']'
eval set -- ' --' ++ set -- --
ACCEPT_LICENSE= ++ uname -r
KERNEL_VERSION=4.18.0-477.27.1.el8_8.x86_64
PRIVATE_KEY=
PACKAGE_TAG=
for opt in ${options}
case "$opt" in
shift
break
'[' 0 -ne 0 ']'
_resolve_rhel_version
'[' -f /host-etc/os-release ']' Resolving RHEL version...
echo 'Resolving RHEL version...'
local version= ++ cat /host-etc/os-release ++ grep '^ID=' ++ awk -F= '{print $2}' ++ sed -e 's/^"//' -e 's/"$//'
local id=rhel
'[' rhel = rhcos ']'
'[' rhel = rhel ']' ++ cat /host-etc/os-release ++ grep VERSION_ID ++ awk -F= '{print $2}' ++ sed -e 's/^"//' -e 's/"$//'
version=8.8
'[' -z 8.8 ']'
RHEL_VERSION=8.8
echo 'Proceeding with RHEL version 8.8' Proceeding with RHEL version 8.8
[[ -z '' ]]
DNF_RELEASEVER=8.8
return 0
init
_prepare_exclusive
_prepare
'[' passthrough = vgpu ']'
sh NVIDIA-Linux-x86_64-535.129.03.run -x Creating directory NVIDIA-Linux-x86_64-535.129.03 Verifying archive integrity... OK Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 535.129.03........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
cd NVIDIA-Linux-x86_64-535.129.03
sh /tmp/install.sh nvinstall DRIVER_ARCH is x86_64

WARNING: Unable to determine the default X library path. The path /tmp/null/lib will be used, but this path was not detected in the ldconfig(8) cache, and no directory exists at this path, so it is likely that libraries installed there will not be found by the loader.

WARNING: You specified the '--no-kernel-modules' command line option, nvidia-installer will not install any kernel modules as part of this driver installation, and it will not remove existing NVIDIA kernel modules not part of an earlier NVIDIA driver installation. Please ensure that NVIDIA kernel modules matching this driver version are installed separately.

WARNING: This NVIDIA driver package includes Vulkan components, but no Vulkan ICD loader was detected on this system. The NVIDIA Vulkan ICD will not function without the loader. Most distributions package the Vulkan loader; try installing the "vulkan-loader", "vulkan-icd-loader", or "libvulkan1" package.

WARNING: Unable to determine the path to install the libglvnd EGL vendor library config files. Check that you have pkg-config and the libglvnd development libraries installed, or specify a path with --glvnd-egl-config-path.

mkdir -p /usr/src/nvidia-535.129.03
mv LICENSE mkprecompiled kernel /usr/src/nvidia-535.129.03
sed '9,${/^(kernel|LICENSE)/!d}' .manifest

========== NVIDIA Software Installer ==========

echo -e '\n========== NVIDIA Software Installer ==========\n'
echo -e 'Starting installation of NVIDIA driver version 535.129.03 for Linux kernel version 4.18.0-477.27.1.el8_8.x86_64\n' Starting installation of NVIDIA driver version 535.129.03 for Linux kernel version 4.18.0-477.27.1.el8_8.x86_64
exec
flock -n 3
echo 510145
trap 'echo '''Caught signal'''; exit 1' HUP INT QUIT PIPE TERM
trap _shutdown EXIT
_unload_driver
rmmod_args=()
local rmmod_args
local nvidia_deps=0
local nvidia_refs=0
local nvidia_uvm_refs=0
local nvidia_modeset_refs=0
local nvidia_peermem_refs=0 Stopping NVIDIA persistence daemon...
echo 'Stopping NVIDIA persistence daemon...'
'[' -f /var/run/nvidia-persistenced/nvidia-persistenced.pid ']'
'[' -f /var/run/nvidia-gridd/nvidia-gridd.pid ']'
'[' -f /var/run/nvidia-fabricmanager/nv-fabricmanager.pid ']' Unloading NVIDIA driver kernel modules...
echo 'Unloading NVIDIA driver kernel modules...'
'[' -f /sys/module/nvidia_modeset/refcnt ']'
'[' -f /sys/module/nvidia_uvm/refcnt ']'
'[' -f /sys/module/nvidia/refcnt ']'
'[' -f /sys/module/nvidia_peermem/refcnt ']'
'[' 0 -gt 0 ']'
'[' 0 -gt 0 ']'
'[' 0 -gt 0 ']'
'[' 0 -gt 0 ']'
'[' 0 -gt 0 ']'
return 0
_unmount_rootfs Unmounting NVIDIA driver rootfs...
echo 'Unmounting NVIDIA driver rootfs...'
findmnt -r -o TARGET
grep /run/nvidia/driver
_build
_kernel_requires_package Checking NVIDIA driver packages...
local proc_mount_arg=
echo 'Checking NVIDIA driver packages...'
[[ ! -d /usr/src/nvidia-535.129.03/kernel ]]
cd /usr/src/nvidia-535.129.03/kernel
proc_mount_arg='--proc-mount-point /lib/modules/4.18.0-477.27.1.el8_8.x86_64/proc' ++ ls -d -1 'precompiled/**'
return 0
_update_package_cache
'[' '' '!=' builtin ']'
echo 'Updating the package cache...' Updating the package cache...
yum -q makecache
_install_prerequisites ++ mktemp -d
local tmp_dir=/tmp/tmp.j3nPK7QMA3
trap 'rm -rf /tmp/tmp.j3nPK7QMA3' EXIT
cd /tmp/tmp.j3nPK7QMA3
echo 'Installing elfutils...' Installing elfutils...
dnf install -q -y elfutils-libelf.x86_64 elfutils-libelf-devel.x86_64

Upgraded: elfutils-libelf-0.189-3.el8.x86_64 elfutils-libs-0.189-3.el8.x86_64 Installed: elfutils-debuginfod-client-0.189-3.el8.x86_64 elfutils-libelf-devel-0.189-3.el8.x86_64 libzstd-devel-1.4.4-1.el8.x86_64 zlib-devel-1.2.11-21.el8_7.x86_64

rm -rf /lib/modules/4.18.0-477.27.1.el8_8.x86_64
mkdir -p /lib/modules/4.18.0-477.27.1.el8_8.x86_64/proc Enabling RHOCP and EUS RPM repos...
echo 'Enabling RHOCP and EUS RPM repos...'
'[' -n '' ']'
dnf config-manager --set-enabled rhel-8-for-x86_64-baseos-eus-rpms Updating Subscription Management repositories. Unable to read consumer identity Subscription Manager is operating in container mode.
dnf makecache --releasever=8.8 Updating Subscription Management repositories. Unable to read consumer identity Subscription Manager is operating in container mode. Red Hat Enterprise Linux 8 for x86_64 - AppStre 122 MB/s | 54 MB 00:00 Red Hat Enterprise Linux 8 for x86_64 - BaseOS 118 MB/s | 61 MB 00:00 Red Hat Enterprise Linux 8 for x86_64 - BaseOS 126 MB/s | 66 MB 00:00 Red Hat Universal Base Image 8 (RPMs) - BaseOS 19 kB/s | 3.8 kB 00:00 Red Hat Universal Base Image 8 (RPMs) - AppStre 27 kB/s | 4.2 kB 00:00 Red Hat Universal Base Image 8 (RPMs) - CodeRea 19 kB/s | 3.8 kB 00:00 Metadata cache created. Installing Linux kernel headers...
echo 'Installing Linux kernel headers...'
dnf -q -y --releasever=8.8 install kernel-headers-4.18.0-477.27.1.el8_8.x86_64 kernel-devel-4.18.0-477.27.1.el8_8.x86_64
ln -s /usr/src/kernels/4.18.0-477.27.1.el8_8.x86_64 /lib/modules/4.18.0-477.27.1.el8_8.x86_64/build Installing Linux kernel module files...
echo 'Installing Linux kernel module files...'
dnf -q -y --releasever=8.8 install kernel-core-4.18.0-477.27.1.el8_8.x86_64
touch /lib/modules/4.18.0-477.27.1.el8_8.x86_64/modules.order
touch /lib/modules/4.18.0-477.27.1.el8_8.x86_64/modules.builtin
depmod 4.18.0-477.27.1.el8_8.x86_64 Generating Linux kernel version string...
echo 'Generating Linux kernel version string...'
'[' amd64 = arm64 ']'
extract-vmlinux /lib/modules/4.18.0-477.27.1.el8_8.x86_64/vmlinuz
strings
grep -E '^Linux version'
sed 's/^(.)\s+(.)$/\1/'
'[' -z 'Linux version 4.18.0-477.27.1.el8_8.x86_64 ([email protected]) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-18) (GCC)) #1 SMP Thu Aug 31 10:29:22 EDT 2023' ']'
mv version /lib/modules/4.18.0-477.27.1.el8_8.x86_64/proc ++ cat /lib/modules/4.18.0-477.27.1.el8_8.x86_64/proc/version ++ grep -Eo 'gcc version ([0-9.]+)' ++ grep -Eo '([0-9.]+)'
local gcc_version=8.5.0 ++ rpm -qa gcc
local current_gcc=gcc-8.5.0-18.el8.x86_64 kernel requires gcc version: 'gcc-8.5.0', current gcc version is 'gcc-8.5.0-18.el8.x86_64'
echo 'kernel requires gcc version: '''gcc-8.5.0''', current gcc version is '''gcc-8.5.0-18.el8.x86_64''''
[[ gcc-8.5.0-18.el8.x86_64 =~ gcc-8.5.0-.* ]] ++ rm -rf /tmp/tmp.j3nPK7QMA3
_create_driver_package
local pkg_name=nvidia-modules-4.18.0
local nvidia_sign_args=
local nvidia_modeset_sign_args=
local nvidia_uvm_sign_args=
trap 'make -s -j SYSSRC=/lib/modules/4.18.0-477.27.1.el8_8.x86_64/build clean > /dev/null' EXIT
echo 'Compiling NVIDIA driver kernel modules...' Compiling NVIDIA driver kernel modules...
cd /usr/src/nvidia-535.129.03/kernel
_gpu_direct_rdma_enabled
'[' false = true ']'
return 1
make -s -j SYSSRC=/lib/modules/4.18.0-477.27.1.el8_8.x86_64/build nv-linux.o nv-modeset-linux.o /usr/src/nvidia-535.129.03/kernel/nvidia-drm/nvidia-drm-crtc.c: In function '__nv_drm_plane_atomic_destroy_state': /usr/src/nvidia-535.129.03/kernel/nvidia-drm/nvidia-drm-crtc.c:695:5: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement] struct nv_drm_plane_state *nv_drm_plane_state = ^~~~~~ Relinking NVIDIA driver kernel modules...
echo 'Relinking NVIDIA driver kernel modules...'
rm -f nvidia.ko nvidia-modeset.ko
ld -d -r -o nvidia.ko ./nv-linux.o ./nvidia/nv-kernel.o_binary
ld -d -r -o nvidia-modeset.ko ./nv-modeset-linux.o ./nvidia-modeset/nv-modeset-kernel.o_binary
'[' -n '' ']'
echo 'Building NVIDIA driver package nvidia-modules-4.18.0...' Building NVIDIA driver package nvidia-modules-4.18.0...
../mkprecompiled --pack nvidia-modules-4.18.0 --description 4.18.0-477.27.1.el8_8.x86_64 --proc-mount-point /lib/modules/4.18.0-477.27.1.el8_8.x86_64/proc --driver-version 535.129.03 --kernel-interface nv-linux.o --linked-module-name nvidia.ko --core-object-name nvidia/nv-kernel.o_binary --target-directory . --kernel-interface nv-modeset-linux.o --linked-module-name nvidia-modeset.ko --core-object-name nvidia-modeset/nv-modeset-kernel.o_binary --target-directory . --kernel-module nvidia-uvm.ko --target-directory .
mkdir -p precompiled
mv nvidia-modules-4.18.0 precompiled ++ make -s -j SYSSRC=/lib/modules/4.18.0-477.27.1.el8_8.x86_64/build clean
_cleanup_package_cache
'[' '' '!=' builtin ']'
echo 'Cleaning up the package cache...' Cleaning up the package cache...
rm -rf '/var/cache/yum/*' Installing NVIDIA driver kernel modules...
_install_driver
install_args=()
local install_args
echo 'Installing NVIDIA driver kernel modules...'
cd /usr/src/nvidia-535.129.03
rm -rf /lib/modules/4.18.0-477.27.1.el8_8.x86_64/video
'[' '' = yes ']'
IGNORE_CC_MISMATCH=1
nvidia-installer --kernel-module-only --no-drm --ui=none --no-nouveau-check -m=kernel

WARNING: The nvidia-drm module will not be installed. As a result, DRM-KMS will not function with this installation of the NVIDIA driver.

ERROR: Unable to open 'kernel/dkms.conf' for copying (No such file or directory)

Welcome to the NVIDIA Software Installer for Unix/Linux

Detected 16 CPUs online; setting concurrency level to 16. Installing NVIDIA driver version 535.129.03. A precompiled kernel interface for kernel '4.18.0-477.27.1.el8_8.x86_64' has been found here: ./kernel/precompiled/nvidia-modules-4.18.0. Required file 'nvidia-peermem.ko' not found in package './kernel/precompiled/nvidia-modules-4.18.0' Performing CC sanity check with CC="/usr/bin/cc". Performing CC check. Kernel source path: '/lib/modules/4.18.0-477.27.1.el8_8.x86_64/source'

Kernel output path: '/lib/modules/4.18.0-477.27.1.el8_8.x86_64/build'

Performing Compiler check. Performing Dom0 check. Performing Xen check. Performing PREEMPT_RT check. Performing vgpu_kvm check. Cleaning kernel module build directory. Building kernel modules : [##############################] 100% Kernel module compilation complete. Unable to determine if Secure Boot is enabled: No such file or directory Kernel messages: [675196.271753] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0 [675201.092060] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2566) [675201.092224] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0 [675206.261458] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2566) [675206.261623] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0 [675211.089309] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2566) [675211.089432] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0 [675216.264395] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2566) [675216.264547] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0 [675221.082153] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2566) [675221.082307] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0 [675226.257357] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2566) [675226.257516] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0 [675231.079304] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2566) [675231.079495] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0 [675231.439865] nvidia-modeset: Unloading [675231.454329] nvidia-uvm: Unloaded the UVM driver. [675231.484279] nvidia-nvlink: Unregistered Nvlink Core, major device number 238 [675415.504970] nvidia-nvlink: Nvlink Core is being initialized, major device number 238 [675415.504977] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 535.129.03 Thu Oct 19 18:56:32 UTC 2023 [675415.776454] nvidia-uvm: Loaded the UVM driver, major device number 236. [675415.784761] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 535.129.03 Thu Oct 19 18:42:12 UTC 2023 [675415.790904] nvidia-modeset: Unloading [675415.805510] nvidia-uvm: Unloaded the UVM driver. [675415.835358] nvidia-nvlink: Unregistered Nvlink Core, major device number 238 Installing 'NVIDIA Accelerated Graphics Driver for Linux-x86_64' (535.129.03): Installing: [##############################] 100% Driver file installation is complete. Running post-install sanity check: Checking: [##############################] 100% Post-install sanity check passed.

Installation of the kernel module for the NVIDIA Accelerated Graphics Driver for Linux-x86_64 (version: 535.129.03) is now complete.

_load
_load_driver Parsing kernel module parameters...
echo 'Parsing kernel module parameters...'
_get_module_params
local base_path=/drivers
'[' -f /drivers/nvidia.conf ']'
'[' -f /drivers/nvidia-uvm.conf ']'
'[' -f /drivers/nvidia-modeset.conf ']'
'[' -f /drivers/nvidia-peermem.conf ']'
local nv_fw_search_path=/run/nvidia/driver/lib/firmware
local set_fw_path=true
local fw_path_config_file=/sys/module/firmware_class/parameters/path
[[ true == \t\r\u\e ]] Configuring the following firmware search path in '/sys/module/firmware_class/parameters/path': /run/nvidia/driver/lib/firmware
echo 'Configuring the following firmware search path in '''/sys/module/firmware_class/parameters/path''': /run/nvidia/driver/lib/firmware' ++ grep '[^[:space:]]' /sys/module/firmware_class/parameters/path
[[ ! -z /run/nvidia/driver/lib/firmware ]] WARNING: A search path is already configured in /sys/module/firmware_class/parameters/path Retaining the current configuration Loading ipmi and i2c_core kernel modules...
echo 'WARNING: A search path is already configured in /sys/module/firmware_class/parameters/path'
echo ' Retaining the current configuration'
echo 'Loading ipmi and i2c_core kernel modules...'
modprobe -a i2c_core ipmi_msghandler ipmi_devintf Loading NVIDIA driver kernel modules...
echo 'Loading NVIDIA driver kernel modules...'
set -o xtrace +o nounset
modprobe nvidia
modprobe nvidia-uvm
modprobe nvidia-modeset
set +o xtrace -o nounset Starting NVIDIA persistence daemon... ls: cannot access '/proc/driver/nvidia-nvswitch/devices/*': No such file or directory Mounting NVIDIA driver rootfs... Check SELinux status SELinux is enabled Change device files security context for selinux compatibility Done, now waiting for signal

Sending an email with the must gather bundle.

Jan 04 '24 16:01 darrellschrag

tried to send must gather via email but keeps getting rejected for size but it is well under the limit.

Jan 04 '24 16:01 darrellschrag

sent them via another email address. I think it went through.

Jan 04 '24 17:01 darrellschrag

Looks related to https://github.com/NVIDIA/gpu-operator/issues/617

Jan 04 '24 19:01 KodieGlosserIBM

@darrellschrag Can you attach dmesg logs from the GPU node. From the below error GPU initialization might have failed causing "nvidia-smi" probes to fail. If they don't succeed for a while, driver pod will be restarted due to readiness check.

Warning Unhealthy 2m6s (x41235 over 5d15h) kubelet Startup probe failed: No devices were found

Jan 09 '24 02:01 shivamerla

I have a very similar issue on IBM Cloud running OCP 4.12 - I've tried two different GPU nodes and both fail in the same way. These are the dmesg logs from one of the nodes, where I can see a RmInitAdapter failure for the GPUs

[    6.201288] systemd[1]: systemd-modules-load.service: Succeeded.
[    6.202365] systemd[1]: Stopped Load Kernel Modules.
[    6.203985] systemd[1]: systemd-tmpfiles-setup.service: Succeeded.
[    6.205672] systemd[1]: Stopped Create Volatile Files and Directories.
[    6.207477] systemd[1]: Stopped target Local File Systems.
[    6.208463] systemd[1]: Stopped target Local File Systems (Pre).
[    6.210107] systemd[1]: dracut-initqueue.service: Succeeded.
[    6.211144] systemd[1]: Stopped dracut initqueue hook.
[    6.212672] systemd[1]: systemd-udev-trigger.service: Succeeded.
[    6.213749] systemd[1]: Stopped udev Coldplug all Devices.
[    6.215312] systemd[1]: dracut-pre-trigger.service: Succeeded.
[    6.216356] systemd[1]: Stopped dracut pre-trigger hook.
[    6.217846] systemd[1]: Stopping udev Kernel Device Manager...
[    6.219292] systemd[1]: Stopping Open-iSCSI...
[    6.220136] iscsid[892]: iscsid shutting down.
[    6.221424] systemd[1]: iscsid.service: Succeeded.
[    6.222869] systemd[1]: Stopped Open-iSCSI.
[    6.224248] systemd[1]: Stopping iSCSI UserSpace I/O driver...
[    6.225301] systemd[1]: iscsid.socket: Succeeded.
[    6.226790] systemd[1]: Closed Open-iSCSI iscsid Socket.
[    6.227758] systemd[1]: iscsiuio.service: Succeeded.
[    6.229376] systemd[1]: Stopped iSCSI UserSpace I/O driver.
[    6.230393] systemd[1]: initrd-cleanup.service: Succeeded.
[    6.232099] systemd[1]: Started Cleaning Up and Shutting Down Daemons.
[    6.233257] systemd[1]: iscsiuio.socket: Succeeded.
[    6.234164] systemd[1]: Closed Open-iSCSI iscsiuio Socket.
[    6.245278] systemd[1]: systemd-udevd.service: Succeeded.
[    6.246277] systemd[1]: Stopped udev Kernel Device Manager.
[    6.248185] systemd[1]: systemd-tmpfiles-setup-dev.service: Succeeded.
[    6.249348] systemd[1]: Stopped Create Static Device Nodes in /dev.
[    6.251332] systemd[1]: kmod-static-nodes.service: Succeeded.
[    6.252378] systemd[1]: Stopped Create list of required static device nodes for the current kernel.
[    6.255030] systemd[1]: dracut-pre-udev.service: Succeeded.
[    6.256042] systemd[1]: Stopped dracut pre-udev hook.
[    6.257672] systemd[1]: dracut-cmdline.service: Succeeded.
[    6.258680] systemd[1]: Stopped dracut cmdline hook.
[    6.260219] systemd[1]: systemd-udevd-kernel.socket: Succeeded.
[    6.261315] systemd[1]: Closed udev Kernel Socket.
[    6.262833] systemd[1]: systemd-udevd-control.socket: Succeeded.
[    6.263942] systemd[1]: Closed udev Control Socket.
[    6.266206] systemd[1]: Starting Cleanup udevd DB...
[    6.316423] systemd[1]: initrd-udevadm-cleanup-db.service: Succeeded.
[    6.317643] systemd[1]: Started Cleanup udevd DB.
[    6.319139] systemd[1]: Reached target Switch Root.
[    6.321268] systemd[1]: Starting Switch Root...
[    6.371420] systemd[1]: Switching root.
[    6.432020] printk: systemd: 32 output lines suppressed due to ratelimiting
[    6.455552] audit: type=1404 audit(1710495092.165:2): enforcing=1 old_enforcing=0 auid=4294967295 ses=4294967295 enabled=1 old-enabled=1 lsm=selinux res=1
[    6.550041] SELinux:  policy capability network_peer_controls=1
[    6.551112] SELinux:  policy capability open_perms=1
[    6.551992] SELinux:  policy capability extended_socket_class=1
[    6.552985] SELinux:  policy capability always_check_network=0
[    6.554015] SELinux:  policy capability cgroup_seclabel=1
[    6.554980] SELinux:  policy capability nnp_nosuid_transition=1
[    6.584049] audit: type=1403 audit(1710495092.293:3): auid=4294967295 ses=4294967295 lsm=selinux res=1
[    6.585242] systemd[1]: Successfully loaded SELinux policy in 129.912ms.
[    6.616553] systemd[1]: Relabelled /dev, /run and /sys/fs/cgroup in 20.308ms.
[    6.619634] systemd[1]: systemd 239 (239-58.el8_6.9) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy)
[    6.623360] systemd[1]: Detected virtualization kvm.
[    6.624264] systemd[1]: Detected architecture x86-64.
[    6.627134] systemd[1]: Set hostname to <redhat-cp4d-dallas-dxsj8-gx2-32x256x2v100-1-26nqf>.
[    6.712616] systemd[1]: /usr/lib/systemd/system/bootupd.service:22: Unknown lvalue 'ProtectHostname' in section 'Service'
[    6.736346] systemd[1]: systemd-journald.service: Succeeded.
[    6.738113] systemd[1]: systemd-journald.service: Consumed 0 CPU time
[    6.739401] systemd[1]: initrd-switch-root.service: Succeeded.
[    6.740963] systemd[1]: Stopped Switch Root.
[    6.742390] systemd[1]: initrd-switch-root.service: Consumed 0 CPU time
[    6.743651] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart.
[    6.745404] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1.
[    6.747185] systemd[1]: Stopped Journal Service.
[    6.748884] systemd[1]: systemd-journald.service: Consumed 0 CPU time
[    6.751734] systemd[1]: Starting Journal Service...
[    6.753423] systemd[1]: systemd-fsck-root.service: Succeeded.
[    6.755295] systemd[1]: Stopped File System Check on Root Device.
[    6.757229] systemd[1]: systemd-fsck-root.service: Consumed 0 CPU time
[    6.758433] systemd[1]: Reached target Remote File Systems.
[    6.760703] systemd[1]: Created slice system-systemd\x2dfsck.slice.
[    6.763364] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[    6.767226] systemd[1]: Mounting Kernel Debug File System...
[    6.769362] systemd[1]: Created slice system-sshd\x2dkeygen.slice.
[    6.771697] systemd[1]: Created slice system-getty.slice.
[    6.773531] systemd[1]: Started Forward Password Requests to Clevis Directory Watch.
[    6.776506] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[    6.778957] systemd[1]: Listening on initctl Compatibility Named Pipe.
[    6.780929] systemd[1]: Reached target Local Encrypted Volumes (Pre).
[    6.783033] systemd[1]: Reached target Remote Encrypted Volumes.
[    6.785252] systemd[1]: Created slice system-serial\x2dgetty.slice.
[    6.787937] systemd[1]: Listening on udev Control Socket.
[    6.789850] systemd[1]: Listening on Device-mapper event daemon FIFOs.
[    6.792894] systemd[1]: Starting Load Kernel Modules...
[    6.794890] systemd[1]: Created slice User and Session Slice.
[    6.796693] systemd[1]: Reached target Slices.
[    6.798230] systemd[1]: Reached target Swap.
[    6.799945] systemd[1]: Stopped target Switch Root.
[    6.801407] systemd[1]: Stopped target Initrd Root File System.
[    6.803340] systemd[1]: Stopped target Initrd File Systems.
[    6.805905] systemd[1]: Mounting Temporary Directory (/tmp)...
[    6.807615] systemd[1]: Reached target Synchronize [email protected] template instances.
[    6.810996] systemd[1]: Starting Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
[    6.814710] systemd[1]: Starting CoreOS: Set printk To Level 4 (warn)...
[    6.817521] systemd[1]: Starting Create list of required static device nodes for the current kernel...
[    6.821153] systemd[1]: Starting Create System Users...
[    6.824678] systemd[1]: Starting Rebuild Hardware Database...
[    6.826544] systemd[1]: Reached target Host and Network Name Lookups.
[    6.828807] systemd[1]: Listening on udev Kernel Socket.
[    6.831483] systemd[1]: Starting udev Coldplug all Devices...
[    6.833436] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[    6.835693] systemd[1]: Reached target Local Encrypted Volumes.
[    6.839599] systemd[1]: Listening on Process Core Dump Socket.
[    6.841401] systemd[1]: ostree-prepare-root.service: Succeeded.
[    6.843220] systemd[1]: Stopped OSTree Prepare OS/.
[    6.844735] systemd[1]: ostree-prepare-root.service: Consumed 0 CPU time
[    6.847094] systemd[1]: Mounting POSIX Message Queue File System...
[    6.849796] systemd[1]: Listening on LVM2 poll daemon socket.
[    6.852272] systemd[1]: Listening on RPCbind Server Activation Socket.
[    6.854166] systemd[1]: Reached target RPC Port Mapper.
[    6.856764] systemd[1]: Mounting Huge Pages File System...
[    6.862543] systemd[1]: sysroot-sysroot.mount: Succeeded.
[    6.864115] systemd[1]: sysroot-sysroot.mount: Consumed 0 CPU time
[    6.865250] systemd[1]: sysroot-sysroot-ostree-deploy-rhcos-var.mount: Succeeded.
[    6.867153] systemd[1]: sysroot-sysroot-ostree-deploy-rhcos-var.mount: Consumed 0 CPU time
[    6.868645] systemd[1]: sysroot-etc.mount: Succeeded.
[    6.870124] systemd[1]: sysroot-etc.mount: Consumed 0 CPU time
[    6.871171] systemd[1]: sysroot-usr.mount: Succeeded.
[    6.872607] systemd[1]: sysroot-usr.mount: Consumed 0 CPU time
[    6.873800] systemd[1]: Started Journal Service.
[    7.168037] input: PC Speaker as /devices/platform/pcspkr/input/input5
[    7.168068] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0
[    7.231593] cirrus 0000:00:02.0: vgaarb: deactivate vga console
[    7.256160] Console: switching to colour dummy device 80x25
[    7.256458] [drm] Initialized cirrus 2.0.0 2019 for 0000:00:02.0 on minor 0
[    7.257238] fbcon: cirrusdrmfb (fb0) is primary device
[    7.261221] Console: switching to colour frame buffer device 128x48
[    7.261874] cirrus 0000:00:02.0: [drm] fb0: cirrusdrmfb frame buffer device
[    7.281819] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer
[    8.985276] EXT4-fs (vda3): mounted filesystem with ordered data mode. Opts: (null)
[   11.093217] openvswitch: Open vSwitch switching datapath
[   12.422644] IPv6: ADDRCONF(NETDEV_UP): ens3: link is not ready
[   13.471450] device ovs-system entered promiscuous mode
[   13.472648] Timeout policy base is empty
[   13.472651] Failed to associated timeout policy `ovs_test_tp'
[   13.561619] device ens3 entered promiscuous mode
[   13.702831] device ens3 left promiscuous mode
[   13.706921] device ovs-system left promiscuous mode
[   13.719244] device ovs-system entered promiscuous mode
[   13.720122] No such timeout policy "ovs_test_tp"
[   13.720124] Failed to associated timeout policy `ovs_test_tp'
[   13.736910] device ens3 entered promiscuous mode
[   14.129816] device br-ex entered promiscuous mode
[   33.127613] cgroup: cgroup: disabling cgroup2 socket matching due to net_prio or net_cls activation
[   34.969271] EXT4-fs (vda3): re-mounted. Opts: 
[   68.648963] device br-int entered promiscuous mode
[   71.817168] device ovn-k8s-mp0 entered promiscuous mode
[   71.862150] device genev_sys_6081 entered promiscuous mode
[   79.412825] IPv6: ADDRCONF(NETDEV_UP): 8bcd6490fec1288: link is not ready
[   79.412922] IPv6: ADDRCONF(NETDEV_UP): 8ab667f1710da1f: link is not ready
[   79.413188] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   79.415431] IPv6: ADDRCONF(NETDEV_CHANGE): 8bcd6490fec1288: link becomes ready
[   79.415501] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   79.415534] IPv6: ADDRCONF(NETDEV_CHANGE): 8ab667f1710da1f: link becomes ready
[   79.439158] device 8ab667f1710da1f entered promiscuous mode
[   79.453758] device 8bcd6490fec1288 entered promiscuous mode
[   80.463695] IPv6: ADDRCONF(NETDEV_UP): 1ed4e3625a7f698: link is not ready
[   80.463913] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   80.464869] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   80.464921] IPv6: ADDRCONF(NETDEV_CHANGE): 1ed4e3625a7f698: link becomes ready
[   80.489703] device 1ed4e3625a7f698 entered promiscuous mode
[   80.547293] IPv6: ADDRCONF(NETDEV_UP): 2272641eaa318fd: link is not ready
[   80.548790] IPv6: ADDRCONF(NETDEV_CHANGE): 2272641eaa318fd: link becomes ready
[   80.583504] device 2272641eaa318fd entered promiscuous mode
[   80.586655] IPv6: ADDRCONF(NETDEV_UP): ffa2eb7a3ab15a7: link is not ready
[   80.588275] IPv6: ADDRCONF(NETDEV_CHANGE): ffa2eb7a3ab15a7: link becomes ready
[   80.614234] IPv6: ADDRCONF(NETDEV_UP): 7aacd2a9e3b8d09: link is not ready
[   80.616162] IPv6: ADDRCONF(NETDEV_CHANGE): 7aacd2a9e3b8d09: link becomes ready
[   80.617615] device ffa2eb7a3ab15a7 entered promiscuous mode
[   80.648984] device 7aacd2a9e3b8d09 entered promiscuous mode
[   80.652308] IPv6: ADDRCONF(NETDEV_UP): 7fe1b16e363138f: link is not ready
[   80.653600] IPv6: ADDRCONF(NETDEV_CHANGE): 7fe1b16e363138f: link becomes ready
[   80.672124] IPv6: ADDRCONF(NETDEV_UP): 55cbb1e3bad76c0: link is not ready
[   80.673653] IPv6: ADDRCONF(NETDEV_CHANGE): 55cbb1e3bad76c0: link becomes ready
[   80.680218] device 7fe1b16e363138f entered promiscuous mode
[   80.701459] device 55cbb1e3bad76c0 entered promiscuous mode
[  112.803331] IPv6: ADDRCONF(NETDEV_UP): 3e85a1fd02a5f27: link is not ready
[  112.803645] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[  112.804641] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  112.804687] IPv6: ADDRCONF(NETDEV_CHANGE): 3e85a1fd02a5f27: link becomes ready
[  112.816423] IPv6: ADDRCONF(NETDEV_UP): 4bae9f5cb604bfb: link is not ready
[  112.818213] IPv6: ADDRCONF(NETDEV_CHANGE): 4bae9f5cb604bfb: link becomes ready
[  112.828860] IPv6: ADDRCONF(NETDEV_UP): bdbca40fc11196a: link is not ready
[  112.830453] IPv6: ADDRCONF(NETDEV_CHANGE): bdbca40fc11196a: link becomes ready
[  112.843591] device 3e85a1fd02a5f27 entered promiscuous mode
[  112.862525] device 4bae9f5cb604bfb entered promiscuous mode
[  112.878120] device bdbca40fc11196a entered promiscuous mode
[  120.961908] device 2272641eaa318fd left promiscuous mode
[  122.914334] device 1ed4e3625a7f698 left promiscuous mode
[  151.092802] device 7fe1b16e363138f left promiscuous mode
[  151.163302] device ffa2eb7a3ab15a7 left promiscuous mode
[  152.927247] IPv6: ADDRCONF(NETDEV_UP): 74203e9ce260bb2: link is not ready
[  152.929311] IPv6: ADDRCONF(NETDEV_CHANGE): 74203e9ce260bb2: link becomes ready
[  152.962029] device 74203e9ce260bb2 entered promiscuous mode
[  152.982191] IPv6: ADDRCONF(NETDEV_UP): 3b9d107b26945f2: link is not ready
[  152.983974] IPv6: ADDRCONF(NETDEV_CHANGE): 3b9d107b26945f2: link becomes ready
[  152.988561] IPv6: ADDRCONF(NETDEV_UP): 22f319d0436b0e2: link is not ready
[  152.989959] IPv6: ADDRCONF(NETDEV_CHANGE): 22f319d0436b0e2: link becomes ready
[  153.020244] device 22f319d0436b0e2 entered promiscuous mode
[  153.035785] device 3b9d107b26945f2 entered promiscuous mode
[  153.076143] IPv6: ADDRCONF(NETDEV_UP): 45fd759699c7071: link is not ready
[  153.077482] IPv6: ADDRCONF(NETDEV_CHANGE): 45fd759699c7071: link becomes ready
[  153.103231] device 45fd759699c7071 entered promiscuous mode
[  214.295917] FS-Cache: Loaded
[  214.358737] RPC: Registered named UNIX socket transport module.
[  214.358740] RPC: Registered udp transport module.
[  214.358741] RPC: Registered tcp transport module.
[  214.358742] RPC: Registered tcp NFSv4.1 backchannel transport module.
[  214.516350] FS-Cache: Netfs 'nfs' registered for caching
[  214.528747] Key type dns_resolver registered
[  214.695975] NFS: Registering the id_resolver key type
[  214.695985] Key type id_resolver registered
[  214.695986] Key type id_legacy registered
[  214.698233] FS-Cache: Duplicate cookie detected
[  214.699018] FS-Cache: O-cookie c=00000000038ee484 [p=0000000042ad2ba5 fl=222 nc=0 na=1]
[  214.700193] FS-Cache: O-cookie d=00000000caff5027 n=0000000003f29b58
[  214.701112] FS-Cache: O-key=[16] '04000000020000000200080134748e41'
[  214.702047] FS-Cache: N-cookie c=00000000cd358d70 [p=0000000042ad2ba5 fl=2 nc=0 na=1]
[  214.703151] FS-Cache: N-cookie d=00000000caff5027 n=0000000080f32c99
[  214.704162] FS-Cache: N-key=[16] '04000000020000000200080134748e41'
[  215.255005] IPv6: ADDRCONF(NETDEV_UP): 95695e500d0a199: link is not ready
[  215.255235] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[  215.256396] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  215.256443] IPv6: ADDRCONF(NETDEV_CHANGE): 95695e500d0a199: link becomes ready
[  215.273730] IPv6: ADDRCONF(NETDEV_UP): ac02e3ddefe3bf7: link is not ready
[  215.275717] IPv6: ADDRCONF(NETDEV_CHANGE): ac02e3ddefe3bf7: link becomes ready
[  215.300234] device 95695e500d0a199 entered promiscuous mode
[  215.318025] device ac02e3ddefe3bf7 entered promiscuous mode
[  251.283884] device 95695e500d0a199 left promiscuous mode
[  252.353863] device ac02e3ddefe3bf7 left promiscuous mode
[  265.877593] VFIO - User Level meta-driver version: 0.3
[  265.934375] nvidia: loading out-of-tree module taints kernel.
[  265.934386] nvidia: module license 'NVIDIA' taints kernel.
[  265.934387] Disabling lock debugging due to kernel taint
[  265.956069] nvidia: module verification failed: signature and/or required key missing - tainting kernel
[  265.968279] nvidia-nvlink: Nvlink Core is being initialized, major device number 237
[  265.968284] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  525.105.17  Tue Mar 28 18:02:59 UTC 2023
[  266.457436] nvidia-uvm: Loaded the UVM driver, major device number 235.
[  266.464918] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  525.105.17  Tue Mar 28 22:18:37 UTC 2023
[  266.471379] nvidia-modeset: Unloading
[  266.487647] nvidia-uvm: Unloaded the UVM driver.
[  266.534571] nvidia-nvlink: Unregistered Nvlink Core, major device number 237
[  312.518289] nvidia-nvlink: Nvlink Core is being initialized, major device number 238
[  312.518296] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  525.105.17  Tue Mar 28 18:02:59 UTC 2023
[  312.985585] nvidia-uvm: Loaded the UVM driver, major device number 236.
[  312.990683] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  525.105.17  Tue Mar 28 22:18:37 UTC 2023
[  313.026328] ACPI Warning: \_SB.PCI0.S38.S08._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20210604/nsarguments-68)
[  317.942836] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  317.943057] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  322.767612] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  322.767805] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  323.001317] ACPI Warning: \_SB.PCI0.S38.S10._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20210604/nsarguments-68)
[  327.934919] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  327.935102] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  332.774161] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  332.774359] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  337.610697] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  337.610884] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  342.444337] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  342.444487] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  347.279804] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  347.279970] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  352.120590] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  352.120730] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  357.012725] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  357.012985] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  361.848341] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  361.848530] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  366.678965] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  366.679130] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  371.499885] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  371.500077] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  376.375225] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  376.375383] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  381.213173] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  381.213347] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  386.035987] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  386.036180] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  390.857775] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  390.857967] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  395.730137] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  395.730313] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  400.557614] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  400.557783] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  405.387616] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  405.387799] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  410.207645] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  410.207831] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  415.133264] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  415.133449] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  419.968648] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  419.968796] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  424.793317] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  424.793551] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  429.618337] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  429.618513] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  434.497886] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  434.498050] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  439.323967] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  439.324157] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  444.156107] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  444.156260] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  448.989150] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  448.989306] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  453.870160] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  453.870436] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  458.704326] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  458.704508] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  463.528918] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  463.529087] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  468.356053] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  468.356210] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  473.227800] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  473.227939] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  478.061373] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  478.061521] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  482.885108] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  482.885267] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  487.705680] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  487.705869] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  492.582216] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  492.582418] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  497.414343] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  497.414498] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  502.231883] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  502.232033] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  507.055077] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  507.055257] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  511.933111] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  511.933264] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  514.847344] IPv6: ADDRCONF(NETDEV_UP): d656a1a9eb662f0: link is not ready
[  514.847573] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[  514.848612] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  514.848658] IPv6: ADDRCONF(NETDEV_CHANGE): d656a1a9eb662f0: link becomes ready
[  514.871848] device d656a1a9eb662f0 entered promiscuous mode
[  516.769413] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  516.769578] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  518.904352] device d656a1a9eb662f0 left promiscuous mode
[  521.605202] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  521.605348] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  526.433379] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  526.433564] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  531.315076] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  531.315260] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  536.152916] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  536.153075] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  540.972539] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  540.972688] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  542.355328] IPv6: ADDRCONF(NETDEV_UP): 180e8dd1fd0df03: link is not ready
[  542.355598] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[  542.356766] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  542.356811] IPv6: ADDRCONF(NETDEV_CHANGE): 180e8dd1fd0df03: link becomes ready
[  542.379756] device 180e8dd1fd0df03 entered promiscuous mode
[  545.796566] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  545.796766] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  550.691579] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  550.691766] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  555.121595] device 180e8dd1fd0df03 left promiscuous mode
[  555.524360] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  555.524579] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  560.354016] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  560.354179] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  565.176082] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  565.176239] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  570.058337] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  570.058498] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  572.269131] IPv6: ADDRCONF(NETDEV_UP): cc31a2f7bee6e72: link is not ready
[  572.269345] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[  572.270238] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  572.270299] IPv6: ADDRCONF(NETDEV_CHANGE): cc31a2f7bee6e72: link becomes ready
[  572.296163] device cc31a2f7bee6e72 entered promiscuous mode
[  574.888687] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  574.888883] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  579.717008] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  579.717196] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  584.557387] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  584.557575] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  585.286690] device cc31a2f7bee6e72 left promiscuous mode
[  589.445553] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  589.445750] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  594.281578] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  594.281732] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  599.105589] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  599.105759] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  603.925390] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  603.925565] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  608.800921] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  608.801099] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  613.626755] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  613.626902] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  618.461852] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  618.462012] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  623.293073] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  623.293274] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  628.164887] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  628.165048] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  632.996457] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  632.996613] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  637.819289] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  637.819443] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  642.641366] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  642.641569] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  647.522474] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  647.522644] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  652.356293] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  652.356460] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  657.176828] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  657.176969] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  661.996094] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  661.996259] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  666.873168] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  666.873349] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  671.706888] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  671.707045] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  676.536067] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  676.536197] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  681.368069] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  681.368194] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  686.249049] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  686.249213] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  691.083042] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  691.083218] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  695.903521] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  695.903659] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  700.727173] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  700.727324] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  705.605454] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  705.605610] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  710.441220] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  710.441444] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  715.267436] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  715.267587] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  720.101389] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  720.101523] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  724.976461] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  724.976632] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  729.802131] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  729.802277] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  734.635145] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  734.635292] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  739.466182] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  739.466344] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  744.352998] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  744.353227] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  749.188022] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  749.188165] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  754.012637] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  754.012825] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  758.835796] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  758.835952] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  763.709743] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  763.709900] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  768.542717] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  768.542857] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  773.365307] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  773.365466] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  778.185177] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  778.185342] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  783.064707] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  783.064875] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  787.898145] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  787.898330] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  792.718082] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  792.718262] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  797.538904] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  797.539121] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  802.423305] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  802.423443] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  807.261379] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  807.261540] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  812.081005] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  812.081162] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  815.123686] IPv6: ADDRCONF(NETDEV_UP): 7fc3f4f1c555460: link is not ready
[  815.123931] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[  815.125179] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  815.125233] IPv6: ADDRCONF(NETDEV_CHANGE): 7fc3f4f1c555460: link becomes ready
[  815.148414] device 7fc3f4f1c555460 entered promiscuous mode
[  816.913428] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  816.913594] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  818.519336] device 7fc3f4f1c555460 left promiscuous mode
[  821.793226] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  821.793407] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  826.628198] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  826.628353] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  831.448484] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  831.448635] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  836.271634] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  836.271803] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  841.149784] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  841.149949] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  845.987671] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  845.987827] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  850.809646] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  850.809799] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  855.629298] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  855.629497] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  860.506707] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  860.506855] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  865.341517] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  865.341670] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  870.162250] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  870.162422] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  874.985429] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  874.985604] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  879.876579] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  879.876719] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  884.711615] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  884.711775] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  889.535916] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  889.536067] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  894.356745] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  894.356936] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  899.239835] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  899.239982] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  904.074859] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  904.075035] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  908.909637] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  908.909792] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  913.738576] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  913.738713] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  918.602510] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  918.602684] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  923.425596] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  923.425738] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  928.250288] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  928.250496] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  933.073236] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  933.073363] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  937.953349] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  937.953524] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  942.790538] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  942.790721] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  947.618174] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  947.618338] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  952.441848] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  952.442037] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  957.315268] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  957.315423] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  962.150144] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  962.150308] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  966.972566] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  966.972782] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  971.797635] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  971.797823] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  976.673522] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  976.673708] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  981.508102] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  981.508286] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[  986.329375] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  986.329551] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  991.151193] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[  991.151358] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[  996.033502] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[  996.033676] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1000.866666] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1000.866845] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1005.687011] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1005.687176] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1010.509375] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1010.509529] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1015.382964] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1015.383186] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1020.215087] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1020.215235] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1025.035803] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1025.035955] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1029.855010] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1029.855169] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1034.729095] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1034.729243] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1039.563980] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1039.564152] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1044.384330] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1044.384467] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1049.205586] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1049.205767] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1054.073704] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1054.073859] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1058.895334] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1058.895498] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1063.727768] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1063.727921] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1068.561156] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1068.561304] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1073.436490] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1073.436647] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1078.275129] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1078.275309] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1083.102930] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1083.103078] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1087.936183] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1087.936325] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1092.815397] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1092.815543] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1097.650222] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1097.650377] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1102.469870] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1102.470052] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1107.289575] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1107.289726] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1112.163623] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1112.163774] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1114.993489] IPv6: ADDRCONF(NETDEV_UP): dddd1c3ccc54a5b: link is not ready
[ 1114.993797] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 1114.994809] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 1114.994848] IPv6: ADDRCONF(NETDEV_CHANGE): dddd1c3ccc54a5b: link becomes ready
[ 1115.021348] device dddd1c3ccc54a5b entered promiscuous mode
[ 1117.008104] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1117.008302] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1119.112380] device dddd1c3ccc54a5b left promiscuous mode
[ 1121.850825] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1121.850974] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1126.682635] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1126.682773] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1131.565603] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1131.565779] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1136.397516] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1136.397658] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1141.218128] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1141.218298] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1146.043916] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1146.044062] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1150.925410] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1150.925575] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1155.759295] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1155.759442] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1160.583066] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1160.583440] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1165.405023] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1165.405196] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1170.274046] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1170.274217] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1175.096899] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1175.097062] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1179.928336] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1179.928491] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1184.762795] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1184.762947] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1189.637342] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1189.637530] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1194.469415] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1194.469566] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1199.291424] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1199.291570] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1204.112875] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1204.113025] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1209.002644] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1209.002809] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1213.837568] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1213.837725] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1218.659102] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1218.659258] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1223.485062] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1223.485190] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1228.370141] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1228.370302] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1233.205059] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1233.205213] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1238.021808] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1238.021974] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1242.842786] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1242.842928] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1247.723236] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1247.723386] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1252.557721] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1252.557882] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1257.379699] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1257.379893] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1262.199656] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1262.199847] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1267.070673] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1267.070832] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1271.906547] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1271.906755] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1276.727309] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1276.727456] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1281.550275] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1281.550463] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1286.434568] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1286.434717] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1291.270437] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1291.270585] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1296.089910] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1296.090079] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1300.911068] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1300.911233] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1305.790272] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1305.790427] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1310.647298] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1310.647485] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1315.470846] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1315.471025] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1320.294933] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1320.295086] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1325.175891] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1325.176044] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1330.012549] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1330.012717] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1334.835523] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1334.835670] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1339.655463] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1339.655651] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1344.543709] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1344.543854] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1349.378796] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1349.378961] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1354.200592] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1354.200754] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1359.023416] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1359.023570] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1363.900608] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1363.900757] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1368.735359] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1368.735501] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1373.554082] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1373.554268] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1378.375795] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1378.375952] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1383.257104] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1383.257294] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1388.089871] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1388.090032] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1392.910600] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1392.910755] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1397.735663] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1397.735877] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1402.618884] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1402.619036] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1407.453820] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1407.453983] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1412.007047] IPv6: ADDRCONF(NETDEV_UP): ddce5abe3252efd: link is not ready
[ 1412.007277] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 1412.008165] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 1412.008205] IPv6: ADDRCONF(NETDEV_CHANGE): ddce5abe3252efd: link becomes ready
[ 1412.029390] device ddce5abe3252efd entered promiscuous mode
[ 1412.274397] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1412.274563] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1413.983859] FS-Cache: Duplicate cookie detected
[ 1413.984627] FS-Cache: O-cookie c=0000000046342fde [p=0000000042ad2ba5 fl=222 nc=0 na=1]
[ 1413.985785] FS-Cache: O-cookie d=00000000caff5027 n=000000001a46cbbe
[ 1413.986676] FS-Cache: O-key=[16] '04000000020000000200080134748e41'
[ 1413.987531] FS-Cache: N-cookie c=000000007ebed363 [p=0000000042ad2ba5 fl=2 nc=0 na=1]
[ 1413.988619] FS-Cache: N-cookie d=00000000caff5027 n=0000000006c192bf
[ 1413.989493] FS-Cache: N-key=[16] '04000000020000000200080134748e41'
[ 1414.189123] IPv6: ADDRCONF(NETDEV_UP): d0e40dabc1b5665: link is not ready
[ 1414.189366] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 1414.190182] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 1414.190226] IPv6: ADDRCONF(NETDEV_CHANGE): d0e40dabc1b5665: link becomes ready
[ 1414.213747] device d0e40dabc1b5665 entered promiscuous mode
[ 1414.476117] IPv6: ADDRCONF(NETDEV_UP): 7bcb3b75dac1d18: link is not ready
[ 1414.477372] IPv6: ADDRCONF(NETDEV_CHANGE): 7bcb3b75dac1d18: link becomes ready
[ 1414.504628] device 7bcb3b75dac1d18 entered promiscuous mode
[ 1416.709623] device 7bcb3b75dac1d18 left promiscuous mode
[ 1417.105419] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1417.105603] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1417.720093] device d0e40dabc1b5665 left promiscuous mode
[ 1421.981313] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1421.981506] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1424.883619] device ddce5abe3252efd left promiscuous mode
[ 1426.817695] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1426.817873] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1431.655492] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1431.655627] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1436.492386] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1436.492533] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1441.372027] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1441.372239] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1446.210149] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1446.210315] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1451.033335] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1451.033516] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1455.855141] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1455.855294] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1460.734662] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1460.734808] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1465.567686] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1465.567878] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1470.387643] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1470.387839] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1472.999688] IPv6: ADDRCONF(NETDEV_UP): 08fdd04b3ee7057: link is not ready
[ 1472.999962] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 1473.000762] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 1473.000809] IPv6: ADDRCONF(NETDEV_CHANGE): 08fdd04b3ee7057: link becomes ready
[ 1473.026337] device 08fdd04b3ee7057 entered promiscuous mode
[ 1475.220143] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1475.220328] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1480.100644] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1480.100790] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1484.930952] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1484.931122] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1486.015749] device 08fdd04b3ee7057 left promiscuous mode
[ 1489.754396] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1489.754583] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1494.576333] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1494.576543] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1499.457411] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1499.457572] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1504.283413] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1504.283544] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1509.117370] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1509.117590] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1513.948359] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1513.948504] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1518.812298] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1518.812462] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1523.633068] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1523.633202] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1528.473171] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1528.473333] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1533.310013] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1533.310182] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1538.186061] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1538.186212] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1543.020879] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1543.021056] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1547.841698] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1547.841853] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1552.664862] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1552.665051] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1557.538763] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1557.538967] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1562.361497] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1562.361629] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1567.194492] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1567.194652] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1572.024658] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1572.024799] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1576.900215] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1576.900371] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1581.737560] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1581.737729] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1586.558069] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1586.558256] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1589.306653] IPv6: ADDRCONF(NETDEV_UP): 431e3df24d381aa: link is not ready
[ 1589.306853] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 1589.307939] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 1589.307988] IPv6: ADDRCONF(NETDEV_CHANGE): 431e3df24d381aa: link becomes ready
[ 1589.332359] device 431e3df24d381aa entered promiscuous mode
[ 1591.392543] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1591.392690] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1596.272241] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1596.272489] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1601.102970] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1601.103125] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1602.243882] device 431e3df24d381aa left promiscuous mode
[ 1604.855462] IPv6: ADDRCONF(NETDEV_UP): 42962c9cd0f5868: link is not ready
[ 1604.855686] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 1604.856762] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 1604.856823] IPv6: ADDRCONF(NETDEV_CHANGE): 42962c9cd0f5868: link becomes ready
[ 1604.883551] device 42962c9cd0f5868 entered promiscuous mode
[ 1605.939132] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1605.939301] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1610.771691] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1610.771861] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1615.648781] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1615.648916] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1617.290419] device 42962c9cd0f5868 left promiscuous mode
[ 1620.483741] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1620.483877] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1625.307783] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1625.307960] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1630.130520] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1630.130674] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1635.004294] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1635.004514] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1639.839305] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1639.839538] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1644.667092] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1644.667254] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1649.509256] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1649.509399] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1654.389295] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1654.389459] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1659.219367] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1659.219501] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1664.053310] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1664.053444] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1668.885942] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1668.886116] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1673.769986] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1673.770147] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1678.603399] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1678.603581] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1683.425759] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1683.425947] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1688.251646] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1688.251783] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1693.131088] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1693.131276] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1697.964819] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1697.964982] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1702.796744] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1702.796885] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1707.628524] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1707.628654] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1712.503515] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1712.503670] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1714.125801] FS-Cache: Duplicate cookie detected
[ 1714.126841] FS-Cache: O-cookie c=00000000a909bd80 [p=0000000042ad2ba5 fl=222 nc=0 na=1]
[ 1714.128566] FS-Cache: O-cookie d=00000000caff5027 n=000000007bcb309e
[ 1714.129892] FS-Cache: O-key=[16] '04000000020000000200080134748e41'
[ 1714.131214] FS-Cache: N-cookie c=000000008d496858 [p=0000000042ad2ba5 fl=2 nc=0 na=1]
[ 1714.132870] FS-Cache: N-cookie d=00000000caff5027 n=00000000d6515ba3
[ 1714.134201] FS-Cache: N-key=[16] '04000000020000000200080134748e41'
[ 1714.361207] IPv6: ADDRCONF(NETDEV_UP): 68c42f3687c28f2: link is not ready
[ 1714.361491] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 1714.362437] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 1714.362488] IPv6: ADDRCONF(NETDEV_CHANGE): 68c42f3687c28f2: link becomes ready
[ 1714.385591] device 68c42f3687c28f2 entered promiscuous mode
[ 1714.677711] IPv6: ADDRCONF(NETDEV_UP): 4b5dce27970e0e1: link is not ready
[ 1714.678798] IPv6: ADDRCONF(NETDEV_CHANGE): 4b5dce27970e0e1: link becomes ready
[ 1714.703214] device 4b5dce27970e0e1 entered promiscuous mode
[ 1717.348108] device 4b5dce27970e0e1 left promiscuous mode
[ 1717.349499] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1717.349732] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1718.364261] device 68c42f3687c28f2 left promiscuous mode
[ 1722.191079] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1722.191252] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1727.015846] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1727.016028] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1731.882236] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1731.882371] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1736.701725] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1736.701855] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1741.531234] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1741.531441] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1746.368240] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1746.368374] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1751.240914] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1751.241084] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1756.074145] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1756.074356] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1760.897503] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1760.897683] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1765.729949] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1765.730060] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1770.610562] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1770.610742] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1775.446495] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1775.446633] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1780.267374] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1780.267509] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1785.092392] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1785.092539] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1789.957957] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1789.958080] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1794.781135] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1794.781356] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1799.614161] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1799.614316] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1804.447422] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1804.447566] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1809.309956] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1809.310116] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1814.134067] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1814.134216] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1817.564925] IPv6: ADDRCONF(NETDEV_UP): 44288b5121dab81: link is not ready
[ 1817.565280] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 1817.567765] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 1817.567817] IPv6: ADDRCONF(NETDEV_CHANGE): 44288b5121dab81: link becomes ready
[ 1817.593690] device 44288b5121dab81 entered promiscuous mode
[ 1818.969257] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1818.969398] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1823.806859] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1823.807001] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1828.683025] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1828.683219] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1831.233936] device 44288b5121dab81 left promiscuous mode
[ 1833.526984] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1833.527140] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1838.348635] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1838.348803] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1843.173826] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1843.174061] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1848.040411] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1848.040563] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1852.876232] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1852.876412] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1857.699447] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1857.699608] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1862.523030] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1862.523181] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1867.394087] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1867.394278] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1872.215707] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1872.215887] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1877.047931] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1877.048120] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1881.877889] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1881.878035] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1886.740948] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1886.741106] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1891.564683] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1891.564855] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1896.399664] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1896.399828] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1901.229589] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1901.229749] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1906.094772] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1906.094942] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1910.915649] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1910.915801] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1915.749855] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1915.749999] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1920.583487] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1920.583650] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1925.447255] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1925.447407] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1930.267177] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1930.267341] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1935.104218] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1935.104344] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1939.933173] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1939.933296] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1944.796437] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1944.796575] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1949.626257] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1949.626411] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1954.446780] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1954.446970] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1959.266911] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1959.267085] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1964.147765] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1964.147953] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1968.983087] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1968.983249] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0
[ 1973.806403] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1973.806657] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1978.625186] NVRM: GPU 0000:04:02.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1978.625366] NVRM: GPU 0000:04:02.0: rm_init_adapter failed, device minor number 1
[ 1983.497852] NVRM: GPU 0000:04:01.0: RmInitAdapter failed! (0x11:0x45:2525)
[ 1983.498026] NVRM: GPU 0000:04:01.0: rm_init_adapter failed, device minor number 0

Is there anything that can be done?

Mar 15 '24 10:03 lugi0

@lugi0 I would recommend opening a support ticket. They can definitely help.

Mar 15 '24 12:03 darrellschrag

@darrellschrag you mean to IBM?

Mar 15 '24 14:03 lugi0

@lugi0 yes

Mar 15 '24 15:03 darrellschrag