amazon-eks-ami
amazon-eks-ami copied to clipboard
awslabs
eBPF filesystem mounted when in use
systemd unit for mounting eBPF, when used:
needed by some tools like Calico:
Latest calico versions allow using eBPF networking, replacing all the kube-proxy functionality. But eBPF filesystem must be mounted on worker node. Can be achieved by creating simple systemd unit (installed by custom startup script, but why not to have it in distro?).
Sample systemd unit: `[Unit] Description=BPF mounts DefaultDependencies=no Before=local-fs.target umount.target After=swap.target
[Mount] What=bpffs Where=/sys/fs/bpf Type=bpf Options=rw,nosuid,nodev,noexec,relatime,mode=700
[Install] WantedBy=multi-user.target`
just place it in /etc/systemd/system/sys-fs-bpf.mount, and enable it.
@cartermckinnon , is there any appetite for this ? I'd be happy to open up a PR to get this change in
I think enabling eBPF has fairly wide implications, I'm not sure it's the right default for all of our users. Are there any other use-cases (besides Calico) for which you'd like to see this support?
I know we've been looking at Pixie, though not sure if using that requires AMI changes.
Many of the monitoring tools are starting to offer features only through eBPF, in particular for our use case this would be DataDog and Sysdig Secure / Aquasec. From the trends that I've been seeing, its eBPF is starting to become important for observability in general. If the implications for enabling eBPF are wide and we don't want to make that the default, then maybe we can take the approach of having a seperate AMI with eBPF enabled , similar to how there is a seperate AMI for the GPU support ?
maybe we can take the approach of having a seperate AMI with eBPF enabled , similar to how there is a seperate AMI for the GPU support ?
Generally, we aren't willing to add AMI variants for things like this. All of our current variants are on the compute dimension; either CPU architecture or GPU support. We'd need to publish a eBPF AMI for each of the existing variants, add testing for each, and then offer support for them -- it just isn't a reasonable change to make in our AMI offerings.
I'll reach out to our security team to get their input on this, but my understanding is that eBPF is not yet hardened enough for us to enable in our existing AMI's. Examples of this are CVE-2021-3490 and CVE-2021-33909.
If you have workloads that need this support, you can always build a custom AMI (or as you suggested, enable it yourself during the bootstrap process). This type of scenario is precisely why custom AMI support exists. 😄
maybe we can take the approach of having a seperate AMI with eBPF enabled , similar to how there is a seperate AMI for the GPU support ?
Generally, we aren't willing to add AMI variants for things like this. All of our current variants are on the compute dimension; either CPU architecture or GPU support. We'd need to publish a eBPF AMI for each of the existing variants, add testing for each, and then offer support for them -- it just isn't a reasonable change to make in our AMI offerings.
I'll reach out to our security team to get their input on this, but my understanding is that eBPF is not yet hardened enough for us to enable in our existing AMI's. Examples of this are CVE-2021-3490 and CVE-2021-33909.
If you have workloads that need this support, you can always build a custom AMI (or as you suggested, enable it yourself during the bootstrap process). This type of scenario is precisely why custom AMI support exists. 😄
So on one side you hate eBPF and Calico, on second you are recommending it to your custommers (https://aws.amazon.com/blogs/containers/turbocharging-eks-networking-with-bottlerocket-calico-and-ebpf//)? Something is really wrong in Amazon. Maybe time to switch to Azure. Our local AWS team will be happy with your answer.
That blog post is discussing eBPF usage on Bottlerocket, which is another option if you need this support. The steps linked in the post also detail how to use a custom AMI to enable Calico's eBPF data plane.
I'm not opposed to the change; but need the opinion of the Amazon Linux security team. I expect an update soon.
@cartermckinnon any updates from the security team on having ebpf enabled?
@Chili-Man was just circling back to this -- they gave me the green-light!
For future readers:
- The
systemdunit suggested here is equivalent to the one Cilium recommends. There's been ample discussion around the proper permissions. -
systemdmounts this automatically in version 238+.
Yeah, I'll take a stab at getting the PR opened. thanks @cartermckinnon !
@aquam8 Unfortunately, I have not started on this yet and I'm not sure I'll be able to start on it until past halfway November.