AWS Outpost support for eksctl

[santosh-at-github]

AWS Supports creating EKS Worker nodes on AWS Outpost, however eksctl doesn't support that yet. I think it will be good to add this capability into eksctl.

I opened a feature request for this earlier (#3441 ) and it was closed due to inactivity and no update was provided on it. I am opening it again to see if there is any possibility of eksctl supporting resource provisioning on AWS Outpost.

[Callisto13]

Hi @santosh07bec, sorry the old issue got closed.

Would you be able to help us understand what is missing from eksctl right now? The team is not hugely familiar with Outposts (we don't have access to any), but eksctl does provide pretty flexible configuration; after a chat the team feel at least pieces should be possible. Could you tell us what is not working for you at the moment?

[santosh-at-github]

At this time to create Node Group on Outpost Subnet, we need to create a Subnet manually on Outpost and then use that in NodeGroup config to create the nodes in the Outpost Subnet which basically launches the instances on Outpost. In order for the eksctl to support nodeGroup creation on Outpost, I think we need the ability to specify the Outpost ARN in "create nodegroup" command which can be used to create Subnet on Outpost and then instances can be launched in that. The only difference between crating subnet on Outpost and in region is that we need to specify Outpost ARN while creating Subnet On Outpost. We can have an "Outpost Only" Option with Outpost ARN in "create nodegroup" subcommand which will create nodegroup on the Outpost.

Create Subnet API: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CreateSubnet.html Run Instances API: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_RunInstances.html -> This doesn't need any Outpost info explicitly, it picks up from the Subnet setting.

Also we can have default volume type as gp2 when creating the NodeGroup on Outpost as Outpost do not support any other volume type as of now.

Please let me know if you need any other information or examples, I will be able to help you with that.

[Callisto13]

Thanks @santosh07bec.

Can you confirm whether create cluster without nodegroups works?

Also we can have default volume type as gp2 when creating the NodeGroup on Outpost as Outpost do not support any other volume type as of now.

This is configurable via volumeType.

[santosh-at-github]

Sorry for the delay.

Can you confirm whether create cluster without nodegroups works?

Yes it works because as of now EKS cluster can't be created in Outpost, we can only create worker node in Outpost and connect that back to In Region EKS Cluster.

[santosh-at-github]

Callisto13, wanted to check if there is plan to implement this. I do not see it in roadmap yet.

[Callisto13]

Hi @santosh07bec thanks for your patience, we have not made a decision on this yet.

@mikestef9 did you find time to sync up with the Outpost team on their request?

[santosh-at-github]

@Callisto13 @mikestef9, any progress on this issue which you would like to share here?

[Callisto13]

Hi @santosh07bec thanks for your patience.

At this time we don't plan to introduce an outpost or equivalent flag to eksctl. The create nodegroup command does not create subnets, (this happens in create cluster) this capability is too much of a niche case for us to add so much functionality.

Furthermore, the open source team has no way to test this, which means we cannot guarantee that it will work/be maintained.

This may change in the future.

As you have said, it is possible to use eksctl to create nodegroups in Outpost with a custom set of subnets. eksctl supports bringing VPCs not created by eksctl to the configuration and launching nodegroups within them: Outpost's usecase fits this feature. We are currently exploring how flexible our subnet configuration is and how we can make it easier for users to add more subnets for their nodegroups after cluster create. If there is anything you have discovered missing from this feature, please let us know.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[santosh-at-github]

Is it possible to provide more control over how eksctl creates subnet and pass more parameters to it? If this can be enabled, we can use that feature to create subnet on Outpost or even on Wavelength/LocalZones by passing parameters such as Outpost ARN or WL/LZ specific AZ to create subnets on it?

[Callisto13]

Is it possible to provide more control over how eksctl creates subnet and pass more parameters to it?

Right now you have 2 options: eksctl creates everything for you, or you bring your own VPC+subnets. We don't currently have any plans to add more "dials" than that, but feel free to write up a feature proposal.

[hsdp-smulford]

+1 eksctl should support outposts

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.

[aclevername]

Attempting to re-summarise what has been said across the two issues by @Callisto13 @santosh07bec @FireballDWF

Overview

What is AWS outpost?

AWS outpost is the name of a product from AWS that allows you to run AWS services on your on-prem infrastructure. For example you could instruct through the AWS API for an EC2 instaces to be created on your on-prem datacenter, instead on one of amazons.

How is this related to eksctl?

Since users can create the majority of AWS resources on there outposts, some users want to be able to create unmanaged nodegroup on there outpost using eksctl. This issue and #3441 both are asking for the option to create unmanaged nodegroups (often referred to as just nodegroups in our config, AWS specifically don't support managed nodegroup) using eksctl.

What stops a user of eksctl from doing this today?

As stated by folks above and in the previous issues the current limitations are:

• you need to supply additional configuration when creating the subnets, specifically providing outpostArn
  • A limitation of EKS specifically states "You can't pass Outposts subnets in when creating a cluster. For more information, see Creating an Amazon EKS cluster.". Currently eksctl is always creating nodegroups in the same set of subnets as the cluster, so this is a significant deviation from our current default behaviour.
• ~~You need to use gp2 volume types for the nodegroup volumes~~ Update: this is already possible through the volumeType config setting

Workaround

eksctl supports creating nodegroups on cluster it didn't create itself. In theory a user could today:

1. Manually create an EKS cluster in Subnets A, B & C
2. Create subnets D, E and F in there outpost
3. Follow our documentation for creating nodegroups for clusters not created by eksctl, making sure to also speciffically set volumeType to gp2.

I haven't tested this so I don't have access to an outpost environment, but in theory this should work.

Paths forward

If we want to be serious about eksctl supporting outpost we could/should:

1. Test & document the workaround if its works

and then if we find the demand is strong: 2. Add full support for creating clusters with nodegroups in outpost. As outlined above this will required a lot of changes to how we create subnets and what the subnets relation to the cluster is.

Any other issues?

From the perspective of the team testing this is difficult. We would have to have access to an AWS outpost environment, which seems costly/unlikely.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.