cloud-init icon indicating copy to clipboard operation
cloud-init copied to clipboard

cloud-init not import public key to authorized_keys in ubuntu24.04

Open kengsia opened this issue 8 months ago • 8 comments

Bug report

I download this ubuntu24.04.2 LTS server ISO image from https://ubuntu.com/download/server , and upload to my openstack platform to create a image , then create VM from the image , but I cannot login this VM with ssh key , as checked , the /home/ubuntu/.ssh/authorized_keys is empty , in the cloud-init.log I can find below logs

2025-04-15 07:31:29,361 - util.py[DEBUG]: Reading from /etc/ssh/sshd_config (quiet=False) 2025-04-15 07:31:29,361 - util.py[DEBUG]: Reading 3255 bytes from /etc/ssh/sshd_config 2025-04-15 07:31:29,362 - util.py[DEBUG]: Changing the ownership of /home/ubuntu/.ssh to 1000:1000 2025-04-15 07:31:29,362 - util.py[DEBUG]: Writing to /home/ubuntu/.ssh/authorized_keys - wb: [600] 0 bytes 2025-04-15 07:31:29,362 - util.py[DEBUG]: Changing the ownership of /home/ubuntu/.ssh/authorized_keys to 1000:1000 2025-04-15 07:31:29,362 - util.py[DEBUG]: Reading from /home/ubuntu/.ssh/authorized_keys (quiet=False) 2025-04-15 07:31:29,362 - util.py[DEBUG]: Reading 0 bytes from /home/ubuntu/.ssh/authorized_keys 2025-04-15 07:31:29,363 - handlers.py[DEBUG]: finish: modules-final/config-ssh_authkey_fingerprints: SUCCESS: config-ssh_authkey_fingerprints ran successfully and took 0.002 seconds

it indicates cloud-init wrote 0 bytes to the /home/ubuntu/.ssh/authorized_keys file

Steps to reproduce the problem

Environment details

  • Cloud-init version: 24.4-0ubuntu1~24.04.2
  • Operating System Distribution: ubuntu24.04.2-lts server
  • Cloud provider, platform or installer type: openstack

cloud-init logs

kengsia avatar Apr 17 '25 06:04 kengsia

Thank you for filing a bug and helping improve cloud-init @kengsia. Generally, what is helpful when filing a bug is to provide enough information for us to reproduce this issue, the logs alone do not Since we don't have enough information here in the bug about the platform meta-data or user-data provided to your openstack instance (and maybe you wanted to keep that private) some things you should check. cloud-init gets SSH key information from meta-data, vendor-data or user-data for a given platform. Sometimes if user-data or metadata is not formatted correctly, cloud-init will be unable to process and write out the proper configuration to disk. Any configuration cloud-init obtains from a platform (termed DataSource in cloud-init) can be checked on the instance to confirm the SSH data you expected is actually being presented, in the right format, to the instance.

Some things to check for OpenStack:

  1. Check to make sure cloud-init hasn't emitted warnings or errors to indicate other issues: cloud-init status --format json
  2. assert that the metadata service exposed a "keys" or "public-keys" bit of config within the top-level "meta_data" key scope: sudo cloud-init query --all. Example of an old OpenStack cloud-init query --all example which contains keys
  3. assert that the user-data or vendor-data provided to the instance is expected and the correct schema format: sudo cloud-init schema --system --annotate
  4. Confirm that the SSH key you provided to OpenStack is actually a valid public key and doesn't contain unexpected header, white-space or truncated content: ssh-keygen -l -f your_key.pub If neither meta-data.keys nor meta-data.public-keys nor user-data: #cloud-config : ssh_authorized_keys are provided, then that instance will not setup SSH authorized keys

If these steps above don't help determine why cloud-init is ignoring your SSH keys, you can provide either the steps to reproduce this problem using the openstack CLI, or attach the output of some of the above commands above and the full /var/log/cloud-init log which may give us a better idea of what config is being provided to the system from your OpenStack's instance metadata service (IMDS). Note to output of sudo cloud-init query --all may contain sensitive information such as passwords if provided in vendordata or user-data so please review that content before posting.

blackboxsw avatar Apr 24 '25 14:04 blackboxsw

Marking incomplete until we have more actionable details on this failure here from @kengsia as the excerpt of logs is not enough to triage the issue.

blackboxsw avatar Apr 24 '25 14:04 blackboxsw

I encounter this same issue deploying the same cloud-init scripts that were applied to Ubuntu 20.04 and 22.04. VMs never come online only on Ubuntu 24.04.

This is observed on Azure.

{
  "boot_status_code": "enabled-by-generator",
  "datasource": "azure",
  "detail": "DataSourceAzure [seed=/dev/sr0]",
  "errors": [
    "('ssh_authkey_fingerprints', KeyError(\"getpwnam(): name not found: 'adminuser'\"))"
  ],
  "extended_status": "error - done",
  "init": {
    "errors": [],
    "finished": null,
    "recoverable_errors": {},
    "start": null
  },
  "init-local": {
    "errors": [],
    "finished": 9.63,
    "recoverable_errors": {},
    "start": 7.31
  },
  "last_update": "Thu, 01 Jan 1970 00:00:28 +0000",
  "modules-config": {
    "errors": [],
    "finished": 13.26,
    "recoverable_errors": {},
    "start": 12.58
  },
  "modules-final": {
    "errors": [
      "('ssh_authkey_fingerprints', KeyError(\"getpwnam(): name not found: 'adminuser'\"))"
    ],
    "finished": 28.47,
    "recoverable_errors": {
      "WARNING": [
        "Running module ssh_authkey_fingerprints (<module 'cloudinit.config.cc_ssh_authkey_fingerprints' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ssh_authkey_fingerprints.py'>) failed"
      ]
    },
    "start": 28.19
  },
  "recoverable_errors": {
    "WARNING": [
      "Running module ssh_authkey_fingerprints (<module 'cloudinit.config.cc_ssh_authkey_fingerprints' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ssh_authkey_fingerprints.py'>) failed"
    ]
  },
  "stage": null,
  "status": "error"
}

We struggle with a similar issue, and I am able to provide a full reproducible procedure:

  1. Create a fresh VM (for example with kvm, virt-manager) with Ubuntu Server 24.04.3 iso
  2. Launch a simple http server via "command" and "user-data" config below.
  3. Launch the VM and edit "GRUB menu entry" like below

Http Server with user data

mkdir simple_http_cloud_init
cd simple_http_cloud_init

mkdir test-system
cat > test-system/user-data << EOF
#cloud-config
autoinstall:
  version: 1
  storage:
    layout:
      name: lvm
      sizing-policy: all
  keyboard:
    layout: de
    toggle: null
    variant: ''
  timezone: "Europe/Berlin"
  user-data:
    users:
    - name: admin-linux
      passwd: ""
      ssh-authorized-keys:
        - ""
      shell: /bin/bash
      lock-passwd: false
      groups: sudo
  ssh:
    allow-pw: false
    ssh-authorized-keys:
      - ""
    install-server: true
EOF

ssh-keygen -t ed25519 -f ~/.ssh/testing-ubuntu-auto

export HASHED_PW=$(mkpasswd -m sha-512 -s test )
export PUBKEY=$(cat ~/.ssh/testing-ubuntu-auto.pub)

# yq --version > 4.0.0
yq -i '.autoinstall."user-data".users[0].ssh_authorized_keys[0] = env(PUBKEY)' test-system/user-data
yq -i '.autoinstall."user-data".users[0].passwd = env(HASHED_PW)' test-system/user-data

touch test-system/meta-data

# This is the ip of my default virt-manager network 
IP_OF_VIRBR=10.99.0.1

python -m http.server <IP OF VIRBR>

I must admit that I was not able to access the http server from the VM, but I thougt a complete netboot.xyz setup was a bit overkill for the explanation, either way this is the setup of the reachable http server. Probably the easiest way would be to create two proxmox vms. One providing the http and the other is the vm we want to install. The below entry is based on our netboot.xyz menu entry:

GRUB menu entry

# REPLACE THE IP BELOW
linux /casper/vmlinuz root=/dev/ram0 ramdisk_size=3500000 cloud-config-url=/dev/null autoinstall ds=nocloud-net;s=http://10.99.0.1/test-system initrd=initrd.magic
initrd /casper/initrd

I try tomorrow to set up the proxmox approach and update after success, but I hope the guideline helps already to reproduce.

MaKaNu avatar Sep 08 '25 14:09 MaKaNu

@MaKaNu please include complete cloud-init logs as shown here.

Also, I haven't looked further but this looks suspicious:

      ssh-authorized-keys:
        - ""

What do you expect this to do?

holmanb avatar Sep 08 '25 16:09 holmanb

@MaKaNu please include complete cloud-init logs as shown here.

I will provide.

Also, I haven't looked further but this looks suspicious:

      ssh-authorized-keys:
        - ""

What do you expect this to do?

Look further I had issues with directly adding the hashed password via EOF so decided to replace them via yq afterwards, which also allows to create the key via ssh-keygen and read the content of the pub file

MaKaNu avatar Sep 08 '25 16:09 MaKaNu

@holmanb

The following logs are created on a fresh test installation via netbootxyz with the config described above, while using this command:

sudo cloud-init collect-logs --redact-sensitive

cloud-init.tar.gz

MaKaNu avatar Sep 09 '25 13:09 MaKaNu

While working on a workaround, I found a small but important issue.

Which would not appear in my MWE from above, but was inside our implementation. The key type information was malformed for a reason I cannot explain anymore:

- ssh_ed25519 A****h
+ ssh-ed25519 A****h

At this point, the Output on the DEBUG is really not helpful. Writing to /home/ubuntu/.ssh/authorized_keys - wb: [600] 0 bytes doesn't help that the key might be malformed. At some point, the installer is probably executing a ssh-key-type check and fails silently, and later Zero Bytes are provided.

For me, it seems somewhere somebody wrote and try...except but did not catch the expected Error and instead just continued.

For further testing, here is my user-data which worked on my test machines with PW test:

PRIVATE-KEY (Not used on any system):

-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW
QyNTUxOQAAACDwmq3kHCdmIo3srUeVYYuGuCIFwN4JXT9dN/MLvV8UqgAAALDkQ6yS5EOs
kgAAAAtzc2gtZWQyNTUxOQAAACDwmq3kHCdmIo3srUeVYYuGuCIFwN4JXT9dN/MLvV8Uqg
AAAECzD12jRwG8nlifDgkcE3QIcHjBJn7g35+z7v/US0qnmvCareQcJ2YijeytR5Vhi4a4
IgXA3gldP1038wu9XxSqAAAAJm1hdHRpLmthdXBlbmpvaGFubkBsYXRpdHVkZTc0MzAtbG
l0LTAxAQIDBAUGBw==
-----END OPENSSH PRIVATE KEY-----

user-data:

#cloud-config
autoinstall:
 version: 1
 storage:
   layout:
     name: lvm
     sizing-policy: all
 keyboard:
   layout: de
   toggle: null
   variant: ''
 timezone: "Europe/Berlin"
 user-data:
   users:
     - name: admin-linux
       passwd: "$6$lYyPMAKv6oUF5/z1$yvYMZuulKYtDFXEdCVBr4tauWPV3ocWkC.LK6iOVLgiiLjr/XI1yqaNKTHYouBvMp6kXZqAMPtEO68zFZ9DlA1"
       ssh_authorized_keys:
         - "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPCareQcJ2YijeytR5Vhi4a4IgXA3gldP1038wu9XxSq"
       shell: /bin/bash
       lock_passwd: false
       groups: sudo
 ssh:
   authorized-keys:
     - "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPCareQcJ2YijeytR5Vhi4a4IgXA3gldP1038wu9XxSq"
   install-server: true
   allow-pw: true

MaKaNu avatar Sep 09 '25 15:09 MaKaNu