libtimeSync syncs time from hypervisor to vm, even when disabled

[Gunni]

Hey,

So about four weeks ago i was setting up a vm on a new hypervisor in a different country. The clock on it kept going wrong, all my systems run on UTC+0 and it was jumping to the local time (wrong too since the host was unsynchronized, which i discovered later), even ntpd was panic exiting (panic threshold, not crash).

After a while of debugging and watching journal and more, we found that when vmtoolsd was left running or restarted, that's when the time jumped. So I verified that time syncing was indeed disabled in the system settings.

But ... apparently libtimeSync doesn't care, and syncs on start and then regularily after that, overriding the good work of ntpd and setting the clock wrong.

Oct 03 19:52:41 x vmtoolsd[17119]: [Oct 03 18:55:19.393] [   debug] [timeSync] Using BDOOR_CMD_GETTIMEFULL_WITH_LAG
Oct 03 19:52:41 x vmtoolsd[17119]: [Oct 03 18:55:19.393] [   debug] [timeSync] Using BDOOR_CMD_GETTIMEFULL_WITH_LAG
Oct 03 19:52:41 x vmtoolsd[17119]: [Oct 03 18:55:19.393] [   debug] [timeSync] One time synchronization: stepping time.

Obviously the hypervisor clock should be correct, but to me it's irrelevant, ntpd syncs time and every server i know of has it, why force enable a feature THAT'S DISABLED IN THE UI!

Now i was onto something, and confirmed that, not only was it affecting that VM, but all VMs, heck, even the ntpd anycast VMs servicing the friggin hypervisors. In those cases the jumps were small, but noticable when looking into the data.

My solution: Delete libtimeSync.so, confirm that the clock is unmolested, add to ansible role, deploy to all systems.

Oct 03 19:20:16 x vmtoolsd[19517]: [Oct 03 19:20:11.531] [   debug] [vmsvc] RpcChannel: Unknown Command 'Time_Synchronize': Handler not registered.

Result: Offsets are all zero, or as close as they are going to get.

Suggestion: Respect the sync time flag from the VM settings page (I have not found a SINGLE system with it checked!)

Alternate solution: Don't check time difference to hypervisor at-all to try to correct it, just notify ntpd that a possible time altering event (suspend/resume, snapshot, disk, etc...) has occurred and it should verify that its time is correct and jump to the correct time if required. For example, a signal to ntpd...

[PaTHml]

Can you provide version information for the products and guest OS involved?

Guest OS: linux distro version and kernel version open-vm-tools: vmware-toolbox-cmd -v or vmtoolsd -v hypervisor: vmware -v and uname -a Is there a vCenter present? versions for that too

If you can specify "which UI" was used for the configuration (ESX, VI client, vCenter), that will help.

From the guest OS Get time sync status: vmware-toolbox-cmd timesync status

Disable time sync: vmware-toolbox-cmd timesync disable

or "enable" to enable time sync

Have a look at: https://kb.vmware.com/s/article/1189;

If you can provide the relevant .vmx settings mention in the article we can check that too.

[Gunni]

The feature causing me issues is time.synchronize.tools.startup which defaults to 1 in vCenter:

RpcIn: received 42 bytes, content:"Set_Option time.synchronize.tools.enable 1"
RpcIn: received 42 bytes, content:"Set_Option time.synchronize.guest.resync 0"
RpcIn: received 50 bytes, content:"Set_Option time.synchronize.guest.resync.timeout 0"
RpcIn: received 52 bytes, content:"Set_Option time.synchronize.tools.startup.backward 0"
RpcIn: received 43 bytes, content:"Set_Option time.synchronize.tools.startup 1"

It needs to be possible to configure these on the VM itself.

Can I override these in tools.conf?

# cat /etc/os-release | grep -i pretty
PRETTY_NAME="CentOS Linux 7 (Core)"
# vmware-toolbox-cmd -v
10.2.5.3619 (build-8068406)
# vmtoolsd -v
VMware Tools daemon, version 10.2.5.3619 (build-8068406)
# vmware-toolbox-cmd timesync status
Disabled

[stanguturi]

Can I override these in tools.conf?

There is no way to do this in tools.conf

vmware-toolbox-cmd timesync status Disabled

Check the following URLs: (Copied from https://github.com/vmware/open-vm-tools/issues/277#issuecomment-510982557)

• VMware KB - https://kb.vmware.com/kb/1189
• VMware Blog - https://blogs.vmware.com/apps/2016/01/completely-disable-time-synchronization-for-your-vm.html

All the options mentioned in the above KB articles need to be set to FALSE to completely disable the time synchronization inside the guest VM. Please reopen if you encounter the issue even after ALL the settings are set to FALSE.

[Gunni]

So if i understand you right, if i can't change the vCenter settings, then, it just sucks to be me?

And the client has no way to override that, except by DELETING a part of the program. Or by not running it in the first place?

If that's what you think, here's what i'm doing, for use by future annoyed sysadmins like me.

- name: Install VMware tools
  yum:
    name: open-vm-tools
    state: present
  when:
  - ansible_virtualization_role is defined
  - ansible_virtualization_role == "guest"
  - ansible_virtualization_type == "VMware"
  tags:
  - vm

- name: Ensure VMware timeSync will never happen
  file:
    path: /usr/lib64/open-vm-tools/plugins/vmsvc/libtimeSync.so
    state: absent
  notify:
  - restart vmtoolsd
  when:
  - ansible_virtualization_role is defined
  - ansible_virtualization_role == "guest"
  - ansible_virtualization_type == "VMware"
  tags:
  - latest  # to ensure this task runs even when the package is updated
  - vm

- name: Start and enable VMware tools
  service:
    name: vmtoolsd
    state: started
    enabled: true
  when:
  - ansible_virtualization_role is defined
  - ansible_virtualization_role == "guest"
  - ansible_virtualization_type == "VMware"
  tags:
  - vm

[stanguturi]

Had a discussion with other team members about this issue. Logged an internal bug for tracking this issue. Reopening the issue.

[Firefishy]

@stanguturi Any progress?

[Firefishy]

Here is how to completely disable this from the Guest in chef: https://github.com/openstreetmap/chef/blob/ce7189b3fd44901074b45fe76e143ce548393a75/cookbooks/hardware/recipes/default.rb#L97-L108

[stanguturi]

@stanguturi Any progress?

We are working internally with the respective team. We will update this issue when we have a specific update about the release.

[sbueringer]

@stanguturi Any updates?

[stanguturi]

@stanguturi Any updates?

No update yet. Will keep you posted. Thanks.

[Gunni]

Over 4 years since I reported this issue, over 2 years since your last response.

How is it going @stanguturi?

[Gunni]

Had a discussion with other team members about this issue. Logged an internal bug for tracking this issue. Reopening the issue.

Hey @stanguturi how is this issue going? Any update on the internal ticket? This issue is starting school soon, since it's reaching 6 years old...

[jonathanvmw]

@Gunni - Thanks for the reminder. We pinged the responsible development team and it looks like this will be getting some attention. Will post here when there is further information.

[Gunni]

Here's an alternative for disabling this using a systemd path unit, making it rather impossible to accidentally have libtimeSync.so running. Even when replaced or added when upgrading using package managers.

Obviously an upstream fix would be better, but they haven't cared for almost 6 years now.

File: /etc/systemd/system/timesync-disable.path

[Unit]
Description=Check if open-vm-tools libtimeSync.so exists
Documentation=https://github.com/vmware/open-vm-tools/issues/302

[Path]
PathExists=/usr/lib64/open-vm-tools/plugins/vmsvc/libtimeSync.so

[Install]
WantedBy=paths.target

File: /etc/systemd/system/timesync-disable.service

[Unit]
Description=Ensure open-vm-tools libtimeSync.so does not work
Documentation=https://github.com/vmware/open-vm-tools/issues/302

[Service]
Type=oneshot
ExecStart=/usr/bin/rm -fv /usr/lib64/open-vm-tools/plugins/vmsvc/libtimeSync.so
ExecStart=/usr/bin/systemctl restart vmtoolsd.service