The Virtualization.framework does not provide an RTC device as `/dev/rtc` when a VM is booted using a Linux kernel image.

[norio-nomura]

Description

The Virtualization.framework does not provide an RTC device as /dev/rtc when a VM is booted using a Linux kernel image.

When booted from a disk image, rtc-efi is available as /dev/rtc0:

$ limactl create --name debug template://ubuntu-24.04 \
      --set '.mounts |= [{"location":"{{.Dir}}/log","mountPoint":"/var/log","writable": true}] + .' \
      --tty=false --log-level=warn
3.50 GiB / 3.50 GiB [---------------------------------------] 100.00% 1.63 GiB/s
$ limactl start debug --log-level=warn
$ head -1 ~/.lima/debug/log/kern.log 
2024-11-14T02:26:21.879392+00:00 lima-debug kernel: Booting Linux on physical CPU 0x0000000000 [0x610f0000]
$ grep rtc ~/.lima/debug/log/kern.log
2024-11-14T02:26:21.879850+00:00 lima-debug kernel: rtc-efi rtc-efi.0: registered as rtc0
2024-11-14T02:26:21.879850+00:00 lima-debug kernel: rtc-efi rtc-efi.0: setting system clock to 2024-11-14T02:26:16 UTC (1731551176)

However, when booting with a kernel image, no RTC devices (such as rtc-efi) are present, so the Linux kernel begins counting from the time configured at build:

$ hack/inject-cmdline-to-template.sh ~/.lima/debug/lima.yaml console=hvc0
kernel_location=https://cloud-images.ubuntu.com/releases/24.04/release-20241004/unpacked/ubuntu-24.04-server-cloudimg-arm64-vmlinuz-generic
kernel_digest=sha256:88e11de35134f64ca401ce11674b8480e416555255191b8dc95af69ec03cdd80
cmdline=root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyAMA0 console=hvc0
initrd_location=https://cloud-images.ubuntu.com/releases/24.04/release-20241004/unpacked/ubuntu-24.04-server-cloudimg-arm64-initrd-generic
initrd_digest=sha256:5b380b89003c96bab04d43dc95ecb69eb25e46478fb3058efc6beea9ce5caf2b
$ limactl factory-reset debug --log-level=warn; rm -rf ~/.lima/debug/log
ERRO[0000] remove /Users/norio/.lima/debug/log: directory not empty 
$ limactl start debug --log-level=warn
WARN[0000] vmType vz: ignoring images[1]: [Kernel Initrd] 
17.39 MiB / 17.39 MiB [------------------------------------------] 100.00% ? p/s
3.50 GiB / 3.50 GiB [---------------------------------------] 100.00% 1.61 GiB/s
$ head -1 ~/.lima/debug/log/kern.log
2024-08-08T14:51:13.862963+00:00 lima-debug kernel: Booting Linux on physical CPU 0x0000000000 [0x610f0000]
$ grep rtc ~/.lima/debug/log/kern.log

Reference: https://developer.apple.com/forums/thread/760344

The system clock remains inaccurate until systemd-timesyncd.service synchronizes with an NTP server.

Known Issues Resulting from Lack of /dev/rtc0

1. Delayed NTP Synchronization: The time required for systemd-timesyncd.service to sync with an NTP server in the background can be lengthy—up to 30 seconds in some cases. If curl or apt-get install is executed in the foreground before synchronization completes, certificate validation fails, resulting in errors:

   [    6.489211] cloud-init[770]: + curl -fsSL https://get.docker.com
   [    6.558633] cloud-init[770]: curl: (60) SSL certificate problem: certificate is not yet valid
   [    6.558703] cloud-init[770]: More details here: https://curl.se/docs/sslcerts.html
   [    6.558761] cloud-init[770]: curl failed to verify the legitimacy of the server and therefore could not
   [    6.558834] cloud-init[770]: establish a secure connection to it. To learn more about this situation and
   [    6.558926] cloud-init[770]: how to fix it, please visit the web page mentioned above.
   [    6.559955] cloud-init[770]: LIMA 2024-08-08T23:51:16+09:00| WARNING: Failed to execute /mnt/lima-cidata/provision.system/00000002
   [    6.561851] cloud-init[770]: LIMA 2024-08-08T23:51:16+09:00| Executing /mnt/lima-cidata/provision.user/00000003 (as user norio)
   [�[0;32m  OK  �[0m] Started �[0;1;39msession-c1.scope�[0m - Session c1 of User norio.
   
   [    6.584176] cloud-init[770]: + command -v jq
   [    6.584344] cloud-init[770]: + sudo apt-get install --assume-yes jq
   [    6.723919] cloud-init[770]: Reading package lists...
   [    6.725890] cloud-init[770]: Building dependency tree...
   [    6.726179] cloud-init[770]: Reading state information...
   [    6.726716] cloud-init[770]: E: Unable to locate package jq
   [    6.728593] cloud-init[770]: LIMA 2024-08-08T23:51:16+09:00| WARNING: Failed to execute /mnt/lima-cidata/provision.user/00000003 (as user norio)
   

2. Guest Agent Time Synchronization Failure: fixSystemTimeSkew() in pkg/guestagent/guestagent_linux.go fails, preventing the guest OS from adjusting the time.

   $ limactl shell debug sudo journalctl -u lima-guestagent
   Nov 14 11:31:28 lima-debug systemd[1]: Started lima-guestagent.service - lima-guestagent.
   Nov 14 11:31:28 lima-debug systemd[1]: Stopping lima-guestagent.service - lima-guestagent...
   Nov 14 11:31:28 lima-debug systemd[1]: lima-guestagent.service: Deactivated successfully.
   Nov 14 11:31:28 lima-debug systemd[1]: Stopped lima-guestagent.service - lima-guestagent.
   Nov 14 11:31:28 lima-debug systemd[1]: Started lima-guestagent.service - lima-guestagent.
   Nov 14 11:31:28 lima-debug lima-guestagent[1727]: time="2024-11-14T11:31:28+09:00" level=info msg="event tick: 3s"
   Nov 14 11:31:28 lima-debug lima-guestagent[1727]: time="2024-11-14T11:31:28+09:00" level=info msg="serving the guest agent on vsock port: 2222"
   Nov 14 11:31:28 lima-debug lima-guestagent[1727]: time="2024-11-14T11:31:28+09:00" level=warning msg="fixSystemTimeSkew: error: stat /dev/rtc: no such file or directory"
   

Using a kernel image for booting allows specifying kernel command-line arguments, which has proven beneficial, e.g., in QEMU-based workarounds (#2541). Therefore, ensuring the system works as expected under these configurations is essential.

[norio-nomura]

I created PR #2894 to address the first issue mentioned above.

[nirs]

Using a kernel image for booting allows specifying kernel command-line arguments, which has proven beneficial, e.g., in QEMU-based workarounds (#2541). Therefore, ensuring the system works as expected under these configurations is essential.

So using kernel image is required only for qemu, but this issue is a about vz. Do we have a use case for using kernel image with vz?

[nirs]

2. Guest Agent Time Synchronization Failure: fixSystemTimeSkew() in pkg/guestagent/guestagent_linux.go fails, preventing the guest OS from adjusting the time.

Do you know why this fails? Can we fix this instead of waiting for NTP?

[nirs]

In the apple forum there is an interesting idea on how to synchronise time after sleep: https://developer.apple.com/forums/thread/760344?answerId=805675022#805675022

Maybe we can add this to the guest agent, and synchronize the time during start and each time when the host wakes up from sleep?

This will work in all case, even if the vm does not have access to the network.

[norio-nomura]

Do you know why this fails?

The guest agent checks for the existence of /dev/rtc before attempting to read the time, and it fails because /dev/rtc does not exist.

Can we fix this instead of waiting for NTP?

If the guestagent retrieves the time from the hostagent and is installed and started earlier (currently done in 25-guestagent-base.sh), it would eliminate the need to wait for NTP synchronization.

[norio-nomura]

Workaround by installing an additional module to the Minimal.

################################################################################
# Linux Modules Extra for /dev/rtc0 on Ubuntu minimal
################################################################################
- mode: system
  # This script install linux-modules-extra-$(uname -r) package if /dev/rtc0 is not a character device.
  script: |
    #!/bin/bash
    set -eux -o pipefail
    test -c /dev/rtc0 && exit 0
    modules_extra=linux-modules-extra-$(uname -r)
    dpkg -s "${modules_extra}" &>/dev/null && exit 0
    apt-get --update install --assume-yes "${modules_extra}"
    modprobe rtc_pl031

[afbjorklund]

This is a quirk in the Apple code, normally you would just add a symlink.

For the scenario where rtc0 is available, that is. Otherwise, maybe NTP?