canonical
cloud init is not able to identify data source
Using the AWS console, I'm launching an instance using a custom AMI. A few XML files need to be updated as part of the system initialization. Although I entered the user data, it doesn't appear to be working.
Upon checking ds-idenify.log and cloud-init-generator.log, I can see that cloud-init is unable to obtain the data source.
cat ds-identify.log
[up 1.17s] ds-identify
policy loaded: mode=search report=false found=all maybe=all notfound=disabled
no datasource_list found, using default: MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack OVF SmartOS Scaleway Hetzner IBMCloud Oracle Exoscale RbxCloud
DMI_PRODUCT_NAME=t3.small
DMI_SYS_VENDOR=Amazon EC2
DMI_PRODUCT_SERIAL=ec2972df-81d1-c72e-f54d-0da7741efa46
DMI_PRODUCT_UUID=DF7229EC-D181-2EC7-F54D-0DA7741EFA46
PID_1_PRODUCT_NAME=unavailable
DMI_CHASSIS_ASSET_TAG=Amazon EC2
FS_LABELS=any:scandir,any:scandir,any:scandir,any:scandir,var,config,deferred,logs,wk,boot,opt,root,notused,rescue
ISO9660_DEVS=
KERNEL_CMDLINE=BOOT_IMAGE=/vmlinuz-x.x.xx-x.xx..0.x86_64 ro root=LABEL=root quiet nmi_watchdog=2 fsck.repair=yes fips=1 elevator=noop rootdelay=300 console=ttyS0,115200n8 aws nvme_core.io_timeout=255 rootfstype=ext4
VIRT=kvm
UNAME_KERNEL_NAME=Linux
UNAME_KERNEL_RELEASE=x.x.xx-x.xx..0.x86_64
UNAME_KERNEL_VERSION=#1 SMP Thu Mar 21 13:43:44 UTC 2024
UNAME_MACHINE=x86_64
UNAME_NODENAME=mlos-tiny
UNAME_OPERATING_SYSTEM=GNU/Linux
DSNAME=
DSLIST=MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack OVF SmartOS Scaleway Hetzner IBMCloud Oracle Exoscale RbxCloud
MODE=search
ON_FOUND=all
ON_MAYBE=all
ON_NOTFOUND=disabled
pid=734 ppid=722
is_container=false
is_ds_enabled(IBMCloud) = true.
ec2 platform is 'Unknown'.
is_ds_enabled(IBMCloud) = true.
No ds found [mode=search, notfound=disabled]. Disabled cloud-init [1]
[up 1.44s] returning 1
cat cloud-init-generator.log
/usr/lib/systemd/system-generators/cloud-init-generator normal=/run/systemd/generator early=/run/systemd/generator.early late=/run/systemd/generator.late
kernel command line (/proc/cmdline): BOOT_IMAGE=/vmlinuz-x.x.xx-x.xx.xx.x.x86_64 ro root=LABEL=root quiet nmi_watchdog=2 fsck.repair=yes fips=1 elevator=noop rootdelay=300 console=ttyS0,115200n8 aws nvme_core.io_timeout=255 rootfstype=ext4
kernel_cmdline found unset
etc_file found unset
default found enabled
checking for datasource
ds-identify rc=1
ds-identify _RET=notfound
**cloud-init is enabled but no datasource found, disabling**
already disabled: no change needed [no /run/systemd/generator.early/multi-user.target.wants/cloud-init.target]
[root@ip-xx-xx-xx-xx cloud-init]#
Using the AWS console, I'm launching an instance using a custom AMI. A few XML files need to be updated as part of the system initialization. Although I entered the user data, it doesn't appear to be working.
Upon checking ds-idenify.log and cloud-init-generator.log, I can see that cloud-init is unable to obtain the data source.
**cat ds-identify.log**
[up 1.17s] ds-identify
policy loaded: mode=search report=false found=all maybe=all notfound=disabled
no datasource_list found, using default: MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack OVF SmartOS Scaleway Hetzner IBMCloud Oracle Exoscale RbxCloud
DMI_PRODUCT_NAME=t3.small
DMI_SYS_VENDOR=Amazon EC2
DMI_PRODUCT_SERIAL=ec2972df-81d1-c72e-f54d-0da7741efa46
DMI_PRODUCT_UUID=DF7229EC-D181-2EC7-F54D-0DA7741EFA46
PID_1_PRODUCT_NAME=unavailable
DMI_CHASSIS_ASSET_TAG=Amazon EC2
FS_LABELS=any:scandir,any:scandir,any:scandir,any:scandir,var,config,deferred,logs,wk,boot,opt,root,notused,rescue
ISO9660_DEVS=
KERNEL_CMDLINE=BOOT_IMAGE=/vmlinuz-x.x.xx-x.xx..0.x86_64 ro root=LABEL=root quiet nmi_watchdog=2 fsck.repair=yes fips=1 elevator=noop rootdelay=300 console=ttyS0,115200n8 aws nvme_core.io_timeout=255 rootfstype=ext4
VIRT=kvm
UNAME_KERNEL_NAME=Linux
UNAME_KERNEL_RELEASE=x.x.xx-x.xx..0.x86_64
UNAME_KERNEL_VERSION=#1 SMP Thu Mar 21 13:43:44 UTC 2024
UNAME_MACHINE=x86_64
UNAME_NODENAME=mlos-tiny
UNAME_OPERATING_SYSTEM=GNU/Linux
DSNAME=
DSLIST=MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack OVF SmartOS Scaleway Hetzner IBMCloud Oracle Exoscale RbxCloud
MODE=search
ON_FOUND=all
ON_MAYBE=all
ON_NOTFOUND=disabled
pid=734 ppid=722
is_container=false
is_ds_enabled(IBMCloud) = true.
ec2 platform is 'Unknown'.
is_ds_enabled(IBMCloud) = true.
No ds found [mode=search, notfound=disabled]. Disabled cloud-init [1]
[up 1.44s] returning 1
**cat cloud-init-generator.log**
/usr/lib/systemd/system-generators/cloud-init-generator normal=/run/systemd/generator early=/run/systemd/generator.early late=/run/systemd/generator.late
kernel command line (/proc/cmdline): BOOT_IMAGE=/vmlinuz-x.x.xx-x.xx.xx.x.x86_64 ro root=LABEL=root quiet nmi_watchdog=2 fsck.repair=yes fips=1 elevator=noop rootdelay=300 console=ttyS0,115200n8 aws nvme_core.io_timeout=255 rootfstype=ext4
kernel_cmdline found unset
etc_file found unset
default found enabled
checking for datasource
ds-identify rc=1
ds-identify _RET=notfound
**cloud-init is enabled but no datasource found, disabling**
already disabled: no change needed [no /run/systemd/generator.early/multi-user.target.wants/cloud-init.target]
[root@ip-xx-xx-xx-xx cloud-init]#
Version of the cloud init used : cloud-init-19.4-12
I have added the datasource_list in /etc/cloud/cloud.cfg
[root@ip-xx-xx-xx-xx ~]# cat /etc/cloud/cloud.cfg
users:
- default
disable_root: 0
ssh_pwauth: 0
mount_default_fields: [~, ~, 'auto', 'defaults,nofail,x-systemd.requires=cloud-init.service', '0', '2']
resize_rootfs_tmp: /dev
ssh_deletekeys: 1
ssh_genkeytypes: ~
syslog_fix_perms: null
disable_vmware_customization: false
datasource_list: [Ec2]
datasource:
Ec2:
metadata_urls: ["http://169.254.169.254:80"]
max_wait: 120
timeout: 50
strict_id: false
#/etc/hosts file will be replaced with the one in /etc/cloud/templates.
manage_etc_hosts: true
cloud_init_modules:
- disk_setup
- migrator
- bootcmd
- write-files
- growpart
- resizefs
- set_hostname
- update_hostname
- update_etc_hosts
- rsyslog
- users-groups
- ssh
cloud_config_modules:
- mounts
- locale
- set-passwords
- rh_subscription
- yum-add-repo
- package-update-upgrade-install
- timezone
- puppet
- chef
- salt-minion
- mcollective
- runcmd
cloud_final_modules:
- rightscale_userdata
- scripts-per-once
- scripts-per-boot
- scripts-per-instance
- scripts-user
- ssh-authkey-fingerprints
- keys-to-console
- phone-home
- final-message
- power-state-change
system_info:
default_user:
name: centos
lock_passwd: true
gecos: Cloud User
groups: [adm, systemd-journal]
sudo: ["ALL=(ALL) NOPASSWD:ALL"]
shell: /bin/bash
distro: rhel
paths:
cloud_dir: /var/lib/cloud
templates_dir: /etc/cloud/templates
ssh_svcname: sshd
# vim:syntax=yaml
I'm using the following user data to test the same.
Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0
--//
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config.txt"
#cloud-config
cloud_final_modules:
- [scripts-user, always]
runcmd:
- [ mkdir, /test-cloudinit ]
write_files:
- path: /test-cloudinit/cloud-init.txt
content: Created by cloud-init
--//
Content-Type: text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="userdata.txt"
#!/bin/bash
mkdir test-userscript
touch /test-userscript/userscript.txt
echo "Created by bash shell script" >> /test-userscript/userscript.txt
--//--
Using the console to run "cloud-init init" yields this.
2024-03-27 07:44:40,631 - warnings.py[WARNING]: **************************************************************************
# This system is using the EC2 Metadata Service, but does not appear to #
# be running on Amazon EC2 or one of cloud-init's known platforms that #
# provide a EC2 Metadata service. In the future, cloud-init may stop #
# reading metadata from the EC2 Metadata Service unless the platform can #
# be identified. #
# #
# If you are seeing this message, please file a bug against #
# cloud-init at #
# https://bugs.launchpad.net/cloud-init/+filebug?field.tags=dsid #
# Make sure to include the cloud provider your instance is #
# running on. #
# #
# For more information see #
# https://bugs.launchpad.net/bugs/1660385 #
# #
# After you have filed a bug, you can disable this warning by #
# launching your instance with the cloud-config below, or #
# putting that content into #
# /etc/cloud/cloud.cfg.d/99-ec2-datasource.cfg #
# #
# #cloud-config #
# datasource: #
# Ec2: #
# strict_id: false #
**************************************************************************
2024-03-27 07:44:40,632 - warnings.py[WARNING]: **************************************************************************
# A new feature in cloud-init identified possible datasources for #
# this system as: #
# [] #
# However, the datasource used was: Ec2 #
# #
# In the future, cloud-init will only attempt to use datasources that #
# are identified or specifically configured. #
# For more information see #
# https://bugs.launchpad.net/bugs/1669675 #
# #
# If you are seeing this message, please file a bug against #
# cloud-init at #
# https://bugs.launchpad.net/cloud-init/+filebug?field.tags=dsid #
# Make sure to include the cloud provider your instance is #
# running on. #
# #
# After you have filed a bug, you can disable this warning by launching #
# your instance with the cloud-config below, or putting that content #
# into /etc/cloud/cloud.cfg.d/99-warnings.cfg #
# #
# #cloud-config #
# warnings: #
# dsid_missing_source: off #
**************************************************************************
Thanks, @ShivChittora, for reporting this.
It looks like ds-identify / cloud-init is not able to see the instance metadata service, but I'm unable to reproduce this.
Could you please attach the logs from sudo cloud-init collect-logs -u, making sure there is no sensitive information?
cloud-init 19.4 is pretty old, is it not possible for you to upgrade could-init in that ami and check if the issue still persist?
@ShivChittora Thanks for reporting. How did you build the custom AMI?
It looks like this happened because ds-identify expects DMI_PRODUCT_UUID to also start with EC2, yet your instance has DMI_PRODUCT_UUID=DF7229EC-D181-2EC7-F54D-0DA7741EFA46. I'm curious what causes instance UUID to be set.
There are a couple of other identifying fields that we could potentially key off of, such as
DMI_CHASSIS_ASSET_TAG or DMI_SYS_VENDOR, but those aren't explicitly listed in the EC2 userguide for identifying ec2 instances.
Maybe @nmeyerhans has some extra context on this?
What's in /sys/devices/virtual/dmi/id/product_uuid? What's the output of sudo dmidecode --string system-uuid? (you may need to install the dmidecode package)
@nmeyerhans The dmidecode package is already installed. PFB the details asked.
[root@ip-xx-xx-xx-xx ~]# cat /sys/devices/virtual/dmi/id/product_uuid D58E20EC-225A-8316-3691-2830F954777C [root@ip-xx-xx-xx-xx ~]# sudo dmidecode --string system-uuid ec208ed5-5a22-1683-3691-2830f954777c
[root@ip-xx-xx-xx-xx ~]# rpm -qa | grep "dmidecode" dmidecode-3.2-6.xxxx.x86_64
@nmeyerhans The method identify_aws (/usr/lib/python2.7/site-packages/cloudinit/sources/DataSourceEc2.py) was searching for data['uuid'] to begin with Ec2.
The exception "uuid = util.read_dmi_data('system-uuid')" was occurring in the subroutine def _collect_platform_data() because `util.load_file("/sys/hypervisor/uuid").strip()" was not present. Instead of supplying system-uuid, I changed this code line by passing system-serial-number.
Like except Exception: uuid = util.read_dmi_data('system-serial-number') #system-uuid data['uuid_source'] = 'dmi'
However, the same datasource not found error were still being reported by "ds-indentity.log".
@holmanb We have an on-premises solution . Installing an ISO file is what we do. Our goal is to offer support on AWS. I created an AMI from that ISO using "docs.aws.amazon.com/vm-import/latest/userguide/…".I rebuilt the iso by adding the cloud-init rpm and created the AMI because the original iso did not have the cloud-init package.
Apart from the code changes mentioned earlier, I also made the changes to the service file (/usr/lib/systemd/system/cloud-*) that were suggested in "https://forums.opensuse.org/t/cloud-init-does-not-run/131727/4". With the help of the user data supplied, system was able to write the contents of the file and execute the script that is located at /run/scripts/test-script.sh.
user data :
cloud_final_modules:
- [scripts-user, always]
write_files:
- path: /run/scripts/test-script.sh
content: |
#!/bin/bash
echo 'Script executed successfully!' >> /run/testing1.txt
permissions: '0755'
runcmd:
- [ sh, "/run/scripts/test-script.sh" ]
However, the "no datasource found" error is still reported by ds-identify.log and cloud-init-generator.log. is this fine ?
I am unable to update the cloud-init package because AMI is based on CentOs 7 and only supports versions of Cloud-inti up to 19.4.
This is what I see in journal.txt, but I won't be able to provide the entire logs because of some restrictions.
[root@ip-xx-xx-xx-xx cloud-init-logs-2024-04-04]# grep -inr "cloud-init" journal.txt
614:Apr 04 04:08:29.418365 mlos-tiny systemd[1]: Starting Initial cloud-init job (pre-networking)...
629:Apr 04 04:08:31.065749 mlos-tiny cloud-init[1325]: Cloud-init v. 19.4 running 'init-local' at Thu, 04 Apr 2024 04:08:30 +0000. Up 6.59 seconds.
757:Apr 04 04:08:33.135632 mlos-tiny cloud-init[1325]: 2024-04-04 04:08:32,894 - DataSourceEc2.py[WARNING]: Calling 'http://169.254.169.254:80/latest/api/token' failed [1/1s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x7fd2f2a39390>, 'Connection to 169.254.169.254 timed out. (connect timeout=1.0)'))]
761:Apr 04 04:08:33.365679 mlos-tiny cloud-init[1325]: 2024-04-04 04:08:32,896 - DataSourceEc2.py[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fd2f2a39410>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))
784:Apr 04 04:08:38.394588 mlos-tiny systemd[1]: Started Initial cloud-init job (pre-networking).
933:Apr 04 04:08:47.657404 ip-xx-xx-xx-xx systemd[1]: Starting Initial cloud-init job (metadata service crawler)...
952:Apr 04 04:08:48.395615 ip-xx-xx-xx-xx cloud-init[3158]: Cloud-init v. 19.4 running 'init' at Thu, 04 Apr 2024 04:08:48 +0000. Up 24.02 seconds.
954:Apr 04 04:08:48.475645 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: +++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++
955:Apr 04 04:08:48.545756 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: +--------+------+------------------------------+---------------+--------+-------------------+
956:Apr 04 04:08:48.615801 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
957:Apr 04 04:08:48.685645 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: +--------+------+------------------------------+---------------+--------+-------------------+
958:Apr 04 04:08:48.795694 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: | eth0 | True | xx-xx-xx-xx | 255.255.240.0 | global | xx-xx-xx-xx |
959:Apr 04 04:08:48.875610 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: | eth0 | True | xx-xx-xx-xx | . | link | xx-xx-xx-xx |
960:Apr 04 04:08:48.955582 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
961:Apr 04 04:08:49.026169 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: | lo | True | ::1/128 | . | host | . |
962:Apr 04 04:08:49.145474 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: +--------+------+------------------------------+---------------+--------+-------------------+
963:Apr 04 04:08:49.235523 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: +++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++
964:Apr 04 04:08:49.355521 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: +-------+-------------+-------------+---------------+-----------+-------+
965:Apr 04 04:08:49.445716 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
966:Apr 04 04:08:49.485597 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: +-------+-------------+-------------+---------------+-----------+-------+
967:Apr 04 04:08:49.485800 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: | 0 | 0.0.0.0 | xx-xx-xx.1 | 0.0.0.0 | eth0 | UG |
968:Apr 04 04:08:49.486020 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: | 1 | xx-xx-xx.0 | 0.0.0.0 | 255.255.240.0 | eth0 | U |
969:Apr 04 04:08:49.745477 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: +-------+-------------+-------------+---------------+-----------+-------+
970:Apr 04 04:08:49.805475 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
971:Apr 04 04:08:49.839726 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: +-------+-------------+---------+-----------+-------+
972:Apr 04 04:08:49.905465 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: | Route | Destination | Gateway | Interface | Flags |
973:Apr 04 04:08:49.965483 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: +-------+-------------+---------+-----------+-------+
974:Apr 04 04:08:50.000328 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: | 9 | fe80::/64 | :: | eth0 | U |
975:Apr 04 04:08:50.040733 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: | 13 | ff00::/8 | :: | eth0 | U |
976:Apr 04 04:08:50.105472 ip-xx-xx-xx-xx cloud-init[3158]: ci-info: +-------+-------------+---------+-----------+-------+
979:Apr 04 04:08:48.810543 ip-xx-xx-xx-xx systemd[1]: Started Initial cloud-init job (metadata service crawler).
986:Apr 04 04:08:50.266263 ip-xx-xx-xx-xx cloud-init[3305]: Cloud-init v. 19.4 running 'modules:config' at Thu, 04 Apr 2024 04:08:50 +0000. Up 25.80 seconds.
1004:Apr 04 04:08:51.685633 ip-xx-xx-xx-xx cloud-init[3345]: Cloud-init v. 19.4 running 'modules:final' at Thu, 04 Apr 2024 04:08:51 +0000. Up 27.12 seconds.
1005:Apr 04 04:08:51.735601 ip-xx-xx-xx-xx cloud-init[3345]: sh: /run/scripts/test-script.sh: No such file or directory
1007:Apr 04 04:08:51.736486 ip-xx-xx-xx-xx cloud-init[3345]: 2024-04-04 04:08:51,637 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/runcmd [127]
1008:Apr 04 04:08:51.933148 ip-xx-xx-xx-xx cloud-init[3345]: 2024-04-04 04:08:51,641 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
1009:Apr 04 04:08:52.045500 ip-xx-xx-xx-xx cloud-init[3345]: 2024-04-04 04:08:51,641 - util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
1764:Apr 04 04:15:15.467062 ip-xx-xx-xx-xx sudo[9062]: root : TTY=pts/0 ; PWD=/root ; USER=root ; COMMAND=/bin/cloud-init collect-logs -u
[root@ip-xx-xx-xx-xx cloud-init-logs-2024-04-04]#