cloud-init icon indicating copy to clipboard operation
cloud-init copied to clipboard
verified publisher icon canonical

ds-identify not finding NoCloud, causes generator to disable service

Open IsSkyfalls opened this issue 5 months ago • 4 comments

Bug report

cloud-init should run on boot, but is disabled by the generator.

Steps to reproduce the problem

this happens after every boot, including first boot.

Environment details

  • Cloud-init version: /usr/bin/cloud-init 25.1.2
  • Operating System Distribution: Archlinux
  • Cloud provider, platform or installer type: VM on Proxmox, barebones archlinux install, with systemd-boot

logs

this is right after booting and logging in. I modified the generator to make it log the time. you can see that when I rerun ds-identify it worked fine. it seems to me that the generator is running too early when /dev/sr0 still haven't been loaded? sounds also kinda weird to me but it is the only explaination I have after quite a long time of debugging. workaround: manually set /etc/cloud/ds-identify.cfg. the actual cloud-init service runs late enough when sr0 is available.

[root@pve-arch ~]# cat /run/cloud-init/ds-identify.log 
[up 1.87s] ds-identify 
policy loaded: mode=search report=false found=all maybe=all notfound=disabled
WARN: no datasource_list found, using default: MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean Vultr AliYun Ec2 GCE OpenNebula OpenStack VMware OVF SmartOS Scaleway Hetzner IBMCloud Oracle Exoscale RbxCloud UpCloud LXD NWCS Akamai WSL CloudCIX
DMI_PRODUCT_NAME=Standard PC (i440FX + PIIX, 1996)
DMI_SYS_VENDOR=QEMU
DMI_PRODUCT_SERIAL=
DMI_PRODUCT_UUID=dbe6a96f-fba5-44fc-828d-f0957ab14651
PID_1_PRODUCT_NAME=unavailable
DMI_CHASSIS_ASSET_TAG=
DMI_BOARD_NAME=unavailable
FS_LABELS=root,BOOT,BOOT
ISO9660_DEVS=
KERNEL_CMDLINE=initrd=initramfs-linux-lts.img root="LABEL=root" console=ttyS0,115200 console=tty0
VIRT=kvm
UNAME_KERNEL_NAME=Linux
UNAME_KERNEL_VERSION=#1 SMP PREEMPT_DYNAMIC Thu, 10 Jul 2025 15:17:11 +0000
UNAME_MACHINE=x86_64
DSNAME=
DSLIST=MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean Vultr AliYun Ec2 GCE OpenNebula OpenStack VMware OVF SmartOS Scaleway Hetzner IBMCloud Oracle Exoscale RbxCloud UpCloud LXD NWCS Akamai WSL CloudCIX
MODE=search
ON_FOUND=all
ON_MAYBE=all
ON_NOTFOUND=disabled
pid=198 ppid=180
is_container=false
is_ds_enabled(IBMCloud) = true.
ec2 platform is 'Unknown'.
is_ds_enabled(IBMCloud) = true.
No ds found [mode=search, notfound=disabled]. Disabled cloud-init [1]
[up 1.92s] returning 1

[root@pve-arch ~]# cat /run/cloud-init/cloud-init-generator.log 
/usr/lib/systemd/system-generators/cloud-init-generator normal=/run/systemd/generator early=/run/systemd/generator.early late=/run/systemd/generator.late
checking for datasource
Sat Jul 12 11:14:11 PM UTC 2025
ds-identify rc=1
cloud-init is enabled but no datasource found, disabling
already disabled: no change needed [no /run/systemd/generator.early/multi-user.target.wants/cloud-init.target]

[root@pve-arch ~]# journalctl -b | grep sr0
Jul 12 23:14:14 pve-arch kernel: sr 0:0:0:0: [sr0] scsi3-mmc drive: 16x/50x cd/rw xa/form2 cdda tray
Jul 12 23:14:14 pve-arch kernel: sr 0:0:0:0: Attached scsi CD-ROM sr0

[root@pve-arch ~]# blkid
/dev/sr0: BLOCK_SIZE="2048" UUID="2025-07-12-21-53-56-00" LABEL="cidata" TYPE="iso9660"
/dev/sda2: LABEL="root" UUID="cdd3dcd9-727f-4191-a282-ec4590ee1cdb" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="root" PARTUUID="44cd91cb-0d05-45d4-9cc5-5b14b1d97779"
/dev/sda1: LABEL_FATBOOT="BOOT" LABEL="BOOT" UUID="4FDD-93C5" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="boot" PARTUUID="3a49608a-c031-44ca-a9e3-b14ea5cbb052"

[root@pve-arch ~]# /usr/lib/cloud-init/ds-identify --force
WARN: no datasource_list found, using default: MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean Vultr AliYun Ec2 GCE OpenNebula OpenStack VMware OVF SmartOS Scaleway Hetzner IBMCloud Oracle Exoscale RbxCloud UpCloud LXD NWCS Akamai WSL CloudCIX

[root@pve-arch ~]# /usr/lib/systemd/system-generators/cloud-init-generator
[root@pve-arch ~]# cat /run/cloud-init/cloud-init-generator.log 
/usr/lib/systemd/system-generators/cloud-init-generator normal= early= late=
checking for datasource
Sat Jul 12 11:16:47 PM UTC 2025
ds-identify rc=0
enabled via /multi-user.target.wants/cloud-init.target -> /lib/systemd/system/cloud-init.target
removing /run/cloud-init/disabled and creating /run/cloud-init/enabled

IsSkyfalls avatar Jul 12 '25 23:07 IsSkyfalls

Thanks for those timestamps and for filing a bug to make cloud-init better.

On first glance the symptoms appear to agree with your analysis. Your first boot ds-identify.log shows no ISO9660 devices:

FS_LABELS=root,BOOT,BOOT

ISO9660_DEVS=

Your addition of the timestamp in ds-identify logging appears to confirm a 12 hour delay before /dev/sr0 is mounted. I'd have to agree that systemd generator timeframe isn't going to block for that amount of time awaiting a potential device mount labelled 'cidata'. Something else is going on with when QEMU is presenting that attached device that isn't going to help cloud-init get across the line in datasource detection, and no other QEMU artifacts such as kernel command line providing ci.ds=nocloud or dmi product serial set to ds=nocloud.

I'm uncertain how we can help here at the moment. Do you have a set of reproducer steps that seem to work before cloud-init version 25.1.2 but appear broken on 25.1.2?

Thank you for any further context.

On Sat, Jul 12, 2025 at 5:39 PM IsSkyfalls @.***> wrote:

IsSkyfalls created an issue (canonical/cloud-init#6304) https://github.com/canonical/cloud-init/issues/6304 Bug report

cloud-init should run on boot, but is disabled by the generator. Steps to reproduce the problem

this happens after every boot, including first boot. Environment details

  • Cloud-init version: /usr/bin/cloud-init 25.1.2
  • Operating System Distribution: Archlinux
  • Cloud provider, platform or installer type: VM on Proxmox, barebones archlinux install, with systemd-boot

logs

this is right after booting and logging in. I modified the generator to make it log the time. you can see that when I rerun ds-identify it worked fine. it seems to me that the generator is running too early when /dev/sr0 still haven't been loaded? sounds also kinda weird to me but it is the only explaination I have after quite a long time of debugging. workaround: manually set /etc/cloud/ds-identify.cfg. the actual cloud-init service runs late enough when sr0 is available.

@.*** ~]# cat /run/cloud-init/ds-identify.log [up 1.87s] ds-identify policy loaded: mode=search report=false found=all maybe=all notfound=disabled WARN: no datasource_list found, using default: MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean Vultr AliYun Ec2 GCE OpenNebula OpenStack VMware OVF SmartOS Scaleway Hetzner IBMCloud Oracle Exoscale RbxCloud UpCloud LXD NWCS Akamai WSL CloudCIX DMI_PRODUCT_NAME=Standard PC (i440FX + PIIX, 1996) DMI_SYS_VENDOR=QEMU DMI_PRODUCT_SERIAL= DMI_PRODUCT_UUID=dbe6a96f-fba5-44fc-828d-f0957ab14651 PID_1_PRODUCT_NAME=unavailable DMI_CHASSIS_ASSET_TAG= DMI_BOARD_NAME=unavailable FS_LABELS=root,BOOT,BOOT ISO9660_DEVS= KERNEL_CMDLINE=initrd=initramfs-linux-lts.img root="LABEL=root" console=ttyS0,115200 console=tty0 VIRT=kvm UNAME_KERNEL_NAME=Linux UNAME_KERNEL_VERSION=#1 SMP PREEMPT_DYNAMIC Thu, 10 Jul 2025 15:17:11 +0000 UNAME_MACHINE=x86_64 DSNAME= DSLIST=MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean Vultr AliYun Ec2 GCE OpenNebula OpenStack VMware OVF SmartOS Scaleway Hetzner IBMCloud Oracle Exoscale RbxCloud UpCloud LXD NWCS Akamai WSL CloudCIX MODE=search ON_FOUND=all ON_MAYBE=all ON_NOTFOUND=disabled pid=198 ppid=180 is_container=false is_ds_enabled(IBMCloud) = true. ec2 platform is 'Unknown'. is_ds_enabled(IBMCloud) = true. No ds found [mode=search, notfound=disabled]. Disabled cloud-init [1] [up 1.92s] returning 1

@.*** ~]# cat /run/cloud-init/cloud-init-generator.log /usr/lib/systemd/system-generators/cloud-init-generator normal=/run/systemd/generator early=/run/systemd/generator.early late=/run/systemd/generator.late checking for datasource Sat Jul 12 11:14:11 PM UTC 2025 ds-identify rc=1 cloud-init is enabled but no datasource found, disabling already disabled: no change needed [no /run/systemd/generator.early/multi-user.target.wants/cloud-init.target]

@.*** ~]# journalctl -b | grep sr0 Jul 12 23:14:14 pve-arch kernel: sr 0:0:0:0: [sr0] scsi3-mmc drive: 16x/50x cd/rw xa/form2 cdda tray Jul 12 23:14:14 pve-arch kernel: sr 0:0:0:0: Attached scsi CD-ROM sr0

@.*** ~]# blkid /dev/sr0: BLOCK_SIZE="2048" UUID="2025-07-12-21-53-56-00" LABEL="cidata" TYPE="iso9660" /dev/sda2: LABEL="root" UUID="cdd3dcd9-727f-4191-a282-ec4590ee1cdb" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="root" PARTUUID="44cd91cb-0d05-45d4-9cc5-5b14b1d97779" /dev/sda1: LABEL_FATBOOT="BOOT" LABEL="BOOT" UUID="4FDD-93C5" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="boot" PARTUUID="3a49608a-c031-44ca-a9e3-b14ea5cbb052"

@.*** ~]# /usr/lib/cloud-init/ds-identify --force WARN: no datasource_list found, using default: MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean Vultr AliYun Ec2 GCE OpenNebula OpenStack VMware OVF SmartOS Scaleway Hetzner IBMCloud Oracle Exoscale RbxCloud UpCloud LXD NWCS Akamai WSL CloudCIX

@.*** ~]# /usr/lib/systemd/system-generators/cloud-init-generator @.*** ~]# cat /run/cloud-init/cloud-init-generator.log /usr/lib/systemd/system-generators/cloud-init-generator normal= early= late= checking for datasource Sat Jul 12 11:16:47 PM UTC 2025 ds-identify rc=0 enabled via /multi-user.target.wants/cloud-init.target -> /lib/systemd/system/cloud-init.target removing /run/cloud-init/disabled and creating /run/cloud-init/enabled

— Reply to this email directly, view it on GitHub https://github.com/canonical/cloud-init/issues/6304, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADLZJRVXREAF6O76WAV45CT3IGMDTAVCNFSM6AAAAACBMI4FD2VHI2DSMVQWIX3LMV43ASLTON2WKOZTGIZDKOJTGEYTGOI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

blackboxsw avatar Jul 14 '25 02:07 blackboxsw

In the absence of further reproducible procedure, I'll mark this incomplete for the time being. If we are able to see steps to reproduce in proxmox or some other platform, we'll be able to better debug this issue. One thing I'd be wondering about is are the proper device drivers loaded to allow the kernel to see the mounted device early in boot? If necessary device drivers are not pre-compile into the kernel, one can run into situations where the disk may not be seen until a later driver load.

blackboxsw avatar Jul 15 '25 14:07 blackboxsw

Hello, thanks for the pointer. I can confirm that it works as expected after loading isofs in initramfs. I did read the systemd man page stating that generators are ran "very early at bootup", but somehow it didn't occur to me just how early that is. Is this documented anywhere? I only opened the issue because github is more searchable and I couldn't find any existing discussions about this. It would be somewhat rare because it depends on the distro and most people probably don't build their own images, but I can't be the first one to run into this and not immediately see the issue? Thanks again.

IsSkyfalls avatar Jul 15 '25 23:07 IsSkyfalls

Is this documented anywhere? ...but I can't be the first one to run into this and not immediately see the issue?

Cloud-init upstream tries to avoid documenting every downstream distro detail that might be required for building an image that works for every platform. Users generally shouldn't need to deal with this kind of issue if they use a distro-built cloud image. Generally cloud images are built with modules required either built into the kernel or loaded by initramfs.

The arch wiki would likely be the most appropriate location to document this for your distro.

holmanb avatar Aug 15 '25 17:08 holmanb