Add support for infra nodes

[romdalf]

Infra nodes are not supported based on the documentation: https://docs.microsoft.com/en-us/azure/openshift/openshift-faq#are-infrastructure-nodes-available

Three questions:

• Is this from a provisioning standpoint or full deployment one? Can worker nodes be configured as infra node post-deployment?
• Can we consider supporting infra nodes? This is a common pattern for customers to dedicate nodes for infrastructure related operators to separate and guarantee resources but also to remove the license (in case of Azure) cost of these nodes.

[jaideepcodes]

Is there an ETA for this . We could have this addressed by Tagging the as #infra nodes and remove the OCP license cost

[viecili]

+1 This is really important for customers running bundled solutions (ACM, ACS) on ARO.

[0kashi]

Thanks for the input. This is currently doable by following the instructions here: https://docs.openshift.com/container-platform/4.11/machine_management/creating-infrastructure-machinesets.html#machineset-yaml-azure_creating-infrastructure-machinesets

Though in terms of the subscription cost for these nodes we have added this to our roadmap targeting early next year.

[marcusne]

I would love to see that feature as well, as we're currently using custom labelled worker to run core extensions for OpenShift and additional operators to avoid to much noisy neighbor behavior with customer workloads running on dedicated workers.

[viecili]

@0kashi

re: "Though in terms of the subscription cost for these nodes we have added this to our roadmap targeting early next year."

can you share the link to this issue?

[0kashi]

This is now GA: https://learn.microsoft.com/en-us/azure/openshift/howto-infrastructure-nodes

[viecili]

Thanks @0kashi , this is a great step forward.

I see there are some limitations on the number and types of infra nodes. Is there any other issue tracking those limitations to make it easier to run bundled solutions (ACM, ACS) on ARO infra nodes?

[BenjaminNeale-Heritage]

Hi @0kashi, I agree with @viecili . While I welcome the introduction of infrastructure nodes, limiting an ARO cluster to only three infrastructure nodes is concerning. It makes it hard to achieve certain use cases involving infrastructure nodes which are valid under the OpenShift Subscription Guide.

For example, I have seen recommendations from Red Hat to have a HUB cluster that runs RCACM Hub and RHACS Central and Red Hat Quay. All of these can run on infrastructure nodes and the comment from Red Hat stated that the HUB cluster could potentially contain only infrastructure nodes with no app worker nodes.

Are there plans in the future to allow more than three infrastructure nodes on ARO clusters?

Thanks, Ben

[0kashi]

Hi @BenjaminNeale-Heritage and @viecili. Truly appreciate your input here. So, the number and type of nodes marked as infrastructure is not limited. You may label as many and whichever instance types you prefer as "infra". Though only the nodes meeting the criteria in the docs will not incur an OpenShift fee, while the rest will be billed as any other worker node. Ultimately, this feature is a billing construct to allow for workloads required by the cluster to be run free of OpenShift fees, as it exists on other managed OpenShift solutions.

Hope this helps clarify.

[BenjaminNeale-Heritage]

Hi @0kashi , thanks for the clarification as I got myself confused and my post above was incorrect.

Infra nodes do not require OpenShift subscriptions according to the OpenShift Subscription Guide which states that there is no upper limit on the number of infra nodes that can be used within a cluster as long as those nodes are only being used for workloads that are valid for infra nodes (e.g. RHACM, RHACS, Quay, etc). ARO adds a cap of three infra nodes that do not incur OpenShift fees while any additional infra nodes will incur OpenShift fees. However, I can understand that ARO is a managed service and there is a need to cover those costs. If we were truly concerned about OpenShift fees on additional infra nodes then we could choose to host OCP clusters in Azure ourselves rather than using ARO. 😅