amazon-eks-ami icon indicating copy to clipboard operation
amazon-eks-ami copied to clipboard

Published 20 hours ago • verified publisher icon awslabs

Make TSC the default system clock source

Open cwhatley opened this issue 5 years ago • 7 comments

What would you like to be added:

Make the default system clock source "tsc" instead of "xen".

Why is this needed:

Docker containers that do frequent system time lookups can be badly affected by the overhead around the xen system clock source.

AWS's own tuning recommendations suggest this switch.

Is there any reason not to switch to tsc as the default for the EKS images?

cwhatley avatar Apr 19 '19 21:04 cwhatley

I have a soon pending PR for this. I need to test it first.

https://github.com/JasonSwindle/amazon-eks-ami/commit/9d57d76229bd21ab78f827c5668e03b7d6a60621

JasonSwindle avatar Apr 30 '19 18:04 JasonSwindle

This issue can be closed.

JasonSwindle avatar May 30 '19 20:05 JasonSwindle

Closed in #272

micahhausler avatar May 30 '19 23:05 micahhausler

This doesn't appear to be working anymore. See comments in #321

cwhatley avatar May 08 '20 00:05 cwhatley

seeing the same behavior in v1.15.11-eks-065dce

redbelow avatar Aug 06 '20 16:08 redbelow

Has anyone continued to experience this issue on currently-supported AMI's?

cartermckinnon avatar Dec 27 '21 19:12 cartermckinnon

I don't know if its related, but I usually do not have to open NTP/123 on my NACLs/security groups but with the EKS AMI I am seeing a lot of REJECTs in flow logs

Per this https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html

The Amazon Time Sync Service is available through NTP at the 169.254.169.123 IPv4 address or the fd00:ec2::123 IPv6 address for any instance running in a VPC. The IPv6 address is only accessible on Instances built on the Nitro System. Your instance does not require access to the internet, and you do not have to configure your security group rules or your network ACL rules to allow access. The latest versions of Amazon Linux 2 and Amazon Linux AMIs synchronize with the Amazon Time Sync Service by default.

Is there any way to use this service at 169.254.169.123 instead of having to open up NTP/123 to a public route?

bryantbiggs avatar Feb 24 '22 16:02 bryantbiggs

Looking into this. I think we should remove our chrony config, because the base AL2 AMI is already doing what's necessary for TSS.

As for #272 -- that PR was a no-op. It's setting the current clocksource during the AMI build process, and the default is reset upon reboot. I think we're fine to set clocksource=tsc on the kernel command line, but I need to verify this doesn't cause issues if/when that clocksource isn't available. I'm also not sure if tsc is considered stable on newer Intels versus kvm-clock (the recommended clocksource on Nitro-based instance types).

cartermckinnon avatar Jun 15 '23 22:06 cartermckinnon

@ybstaked are you using Xen or Nitro-based instance types? FWICT, tsc is what's being used on Nitro-based instance types with our AMI, and the recommendation is to use kvm-clock; whether this could be responsible for the drift is an open question.

cartermckinnon avatar Jun 15 '23 23:06 cartermckinnon

We are known to have issues on specifically at leastc6i.4xlarge and m6i2xlarge. These are:

  • https://github.com/prysmaticlabs/prysm/issues/11415
  • https://www.coincashew.com/coins/overview-eth/guide-or-how-to-setup-a-validator-on-eth2-mainnet/part-i-installation/synchronizing-time-with-chrony
  • https://www.reddit.com/r/ethstaker/comments/ilr1ma/keeping_systemtime_accurate/

ybstaked avatar Jun 16 '23 13:06 ybstaked

Interesting; those are Nitro instance types. If you want to give kvm-clock a try, it'd be a good data point.

I'll go ahead and update #1328 to use kvm-clock on Nitro, it's the recommendation from EC2 anyway 👍

cartermckinnon avatar Jun 16 '23 17:06 cartermckinnon