Azure
[BUG] NVIDIA A10 GPU GRID driver licensing fix
It was discovered that Azure GPU VM sizes using 550 GRID driver (NVIDIA A10 series) installed by AKS on Ubuntu nodes, may be affected by license status issues caused by a change in driver installation components. In particular, the nvidia-gridd systemd unit is not part of the GRID GPU driver version 550.X installation, which may cause workload failures on A10 GPU-enabled node pools.
NVIDIA CUDA drivers and Azure Linux and Windows are not impacted by this issue.
AKS Information
The Ubuntu node image versions 20250226 and 20250302 contain the affected GRID driver version.
AKS has released Ubuntu node image update to revert the GRID driver version installed on impacted A10 GPU node pools.
- Upgrade your Ubuntu node image version, estimated to roll out to all global Azure regions by 3/21/2025.
Note: The AKS Ubuntu node image, version 20250317, will be deployed in March and contain the GRID driver update by default. Customers can monitor the status of this deployment by using AKS Release Tracker.
The Ubuntu node image versions
20250209and20250302contain the affected GRID driver version.
Should include 20250203 and 20250226? The "imageReference" in my json view says differently:
/subscriptions/109a5e88-712a-48ae-9078-9ca8b3c81345/resourceGroups/AKS-Ubuntu/providers/Microsoft.Compute/galleries/AKSUbuntu/images/2204gen2containerd/versions/202502.03.0
/subscriptions/109a5e88-712a-48ae-9078-9ca8b3c81345/resourceGroups/AKS-Ubuntu/providers/Microsoft.Compute/galleries/AKSUbuntu/images/2204gen2containerd/versions/202502.26.0
I am not sure if this is pure "lucky" so just accidentally avoid the issue.
Meantime, in AKS Ubuntu OS release page:
Thanks for pointing this out @JoeyC-Dev, the impacted node image versions are now updated in the description above.
Action required from @aritraghosh, @julia-yin, @AllenWen-at-Azure
![microsoft-github-policy-service[bot] avatar](https://avatars.githubusercontent.com/in/95686?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue has been automatically marked as stale because it has not had any activity for 30 days. It will be closed if no further activity occurs within 7 days of this comment. Please review @sdesai345.
This issue will now be closed because it hasn't had any activity for 7 days after stale. @sdesai345 feel free to comment again on the next 7 days to reopen or open a new issue after that time if you still have a question/issue or suggestion.