Ubuntu 24.04 image not compatible with Azure DevOps self-hosted VMSS

[asbiin]

Description

When deploying the Ubuntu 24.04 image within an Azure virtual machine scale set for a VMSS pool of Azure DevOps, the Microsoft.Azure.DevOps.Pipelines.Agent extension is added, which is using python to run scripts.

Ubuntu 24.04 default python is now 3.12, which does not have the imp module anymore. When a VM starts and tries to run the extension, it fails with this error:

Traceback (most recent call last):
  File "/var/lib/waagent/Microsoft.VisualStudio.Services.TeamServicesAgentLinux-1.23.0.1/./AzureRM.py", line 9, in <module>
    import Utils.HandlerUtil as Util
  File "/var/lib/waagent/Microsoft.VisualStudio.Services.TeamServicesAgentLinux-1.23.0.1/Utils/HandlerUtil.py", line 58, in <module>
    import imp
ModuleNotFoundError: No module named 'imp'

The issue is more on vsts-azurevm-extension: https://github.com/microsoft/vsts-azurevm-extension/issues/233 but I add it here as a reference

Platforms affected

• [X] Azure DevOps
• [ ] GitHub Actions - Standard Runners
• [ ] GitHub Actions - Larger Runners

Runner images affected

• [ ] Ubuntu 20.04
• [ ] Ubuntu 22.04
• [X] Ubuntu 24.04
• [ ] macOS 11
• [ ] macOS 12
• [ ] macOS 13
• [ ] macOS 13 Arm64
• [ ] macOS 14
• [ ] macOS 14 Arm64
• [ ] Windows Server 2019
• [ ] Windows Server 2022

Image version and build link

N/A

Is it regression?

no

Expected behavior

The VM can register to Azure DevOps host

Actual behavior

The extension starts and register the VM under the Azure DevOps pool

Repro steps

Create a self-hosted Azure DevOps pool, of type Azure virtual machine scale set, connected to your VMSS. Start a pipeline job using this pool.

[vidyasagarnimmagaddi]

Hi @asbiin , We will be analysing the issue and provide the solution. thanks

[chirangaalwis]

Hi folks,

Any update on this? We have faced the same issue when using a custom built VM image based on Ubuntu 24.04 LTS for a DevOps self hosted agent VMSS and are currently, unable to proceed.

We assume that downgrading the Python version is not the best approach as the currently installed Python version is installed by default within an Ubuntu distribution and could cause issues as it may already be in use.

[tkasparek]

The issue has been fixed and the fix has been rolled out everywhere, more in https://github.com/microsoft/vsts-azurevm-extension/issues/233

[markDmullins]

The issue has been fixed and the fix has been rolled out everywhere, more in microsoft/vsts-azurevm-extension#233

Unfortunately we are still experiencing this issue. When you create a new Azure virtual machine scale set agent pool in Azure DevOps it adds the extension with version 1.23 of TeamServicesAgentLinux that still imports imp. rather than version 1.25.2 from the fix you referenced.

This is from a brand new Azure VMSS and Agent Pool created today.

Is there some lead time before this propagates through to Azure DevOps or is an additional change needed?

[DevOpsAzurance]

Any updates?

We have a cron tab that rebuilds our images and we have had to roll back the latest version to avoid errors.

[vidyasagarnimmagaddi]

Hi @tkasparek , the issue is fixed in (https://github.com/microsoft/vsts-azurevm-extension/issues/233). Hence we are closing it . thank you

[gavintoddchurch]

Was there ever a solution to this? My extension is still the 1.23 version and I can't seem to upgrade to the version with the fix.

[DevOpsAzurance]

Still having this issue, building both off the latest release and main. This is my VMSS Extension version per 'az vmss extension list':

{ "autoUpgradeMinorVersion": false, "enableAutomaticUpgrade": null, "forceUpdateTag": null, "id": null, "name": "Microsoft.Azure.DevOps.Pipelines.Agent", "protectedSettings": null, "protectedSettingsFromKeyVault": null, "provisionAfterExtensions": null, "provisioningState": null, "publisher": "Microsoft.VisualStudio.Services", "settings": { "agentDownloadUrl": "https://vstsagentpackage.azureedge.net/agent/3.241.0/vsts-agent-linux-x64-3.241.0.tar.gz", "agentFolder": "/agent", "enableScriptDownloadUrl": "https://vstsagenttools.blob.core.windows.net/tools/ElasticPools/Linux/15/enableagent.sh", "isPipelinesAgent": true }, "suppressFailures": null, "type": null, "typeHandlerVersion": "1.23", "typePropertiesType": "TeamServicesAgentLinux" }

[DevOpsAzurance]

Since this extension does not support automatic upgrades, if I need to upgrade, how would I go about doing this without having to rebuild the VMSS resource? I am getting this when running 'az vmss extension upgrade -g MyResourceGroup -n MyScaleSet':

(OperationNotAllowed) Operation 'VirtualMachineScaleSets.ExtensionRollingUpgrade.POST' is not allowed on Virtual Machine Scale Set 'MyScaleSet'. Code: OperationNotAllowed Message: Operation 'VirtualMachineScaleSets.ExtensionRollingUpgrade.POST' is not allowed on Virtual Machine Scale Set 'E MyScaleSet'.

[aharden]

I opened an Azure DevOps Services support issue about this a few weeks ago but they confirmed that there was no current support for Ubuntu 24.04 LTS. Yesterday, the support technician reached out to me to let me know there was a new 1.26 version recently released of the VM extension available that provided that support now. I tested this successfully on an Ubuntu-24.04-based VMSS. I used a VM image created with the github/actions/runner-images Packer template:

• Create agent pool with the U2404 VMSS
• Let Azure DevOps register the Microsoft.Azure.DevOps.Pipelines.Agent VM Extension - it is deployed at v1.23, but it shows the latest version is 1.26 and is manually upgradable
• After the upgrade of the VM extension to 1.26, upgrade or replace the deployed VM instances to pick up the new extension version

[galszabina026]

Szabina Gál is inviting you to a scheduled Zoom meeting.

Topic: [actions/runner-images] Ubuntu 24.04 image not compatible with Azure DevOps self-hosted VMSS (Issue #10122) Time: Oct 29, 2024 09:00 AM Pacific Time (US and Canada)

Join Zoom Meeting https://us05web.zoom.us/j/84135306482?pwd=8u5APoaK9cycMOunNiKgCmoz8Rq6en.1

Meeting ID: 841 3530 6482 Passcode: TOa2029D

[shinkathe]

Why exactly is Devops forcing the version to 1.23? We are also facing this issue.

[gavintoddchurch]

Within the last week an update to the extension page was released. It shows available update version and the update button actually updates the extension now (from my experience). I was able to update from 1.23 > 1.26 and 1.26 works with Ubuntu 24.04 now.

[aharden]

After trying to move ahead with this, I'm seeing Azure DevOps removing the Pipelines VM Extension and then re-adding it with the v1.23 version, which then needs to be manually updated to v1.26 again. This is not yet a stable self-hosted configuration.

[gavintoddchurch]

Ok, yes i'm seeing the same thing. The UI shows it is back to 1.23 with an update available to 1.26

[shinkathe]

Our tests also show that devops downgrades it back to 1.23 even if you manually update it. You can see a message about this in the provisioning logs.

The only solution right now is to downgrade the image to an earlier ubuntu version.