Automatically add/remove master nodes from plan based on deployment size

[arjanelolong2022]

Resource definition

{    
    version: '8.6.1',
    deploymentTemplateId: 'aws-storage-optimized-v3',
    elasticsearch: {
      autoscale: 'true',
      topologies: [
        {
          id: 'hot_content',
          size: '8g',
          zoneCount: 2,
          autoscaling: {
            maxSize: '15g',
            maxSizeResource: 'memory'
          },
        },
        {
          id: 'warm',
          size: '0g',
          zoneCount: 1
        }
      ],
    }
}

One tie breaker master was automatically provisioned.

Expected Behavior

Changes will still be applied even without a master node definition.

Current Behavior

On apply the error below is thrown.

* api error: clusters.cluster_invalid_plan: Cluster must contain at least a master topology element and a data topology element. 'master' node type is missing,'master' node type exists in more than one topology element (resources.elasticsearch[0].cluster_topology)
* api error: clusters.cluster_invalid_plan: Instance configuration [aws.es.master.c5d] does not allow usage of node types [data,ingest]. You must either change instance configuration or use only allowed node types [master]. (resources.elasticsearch[0].cluster_topology[4].instance_configuration_id)

We are also not able to define/add a master node since we have less than 6 data nodes.

Context

After we have provisioned our deployment, we are no longer able to apply updates to our deployment.

Your Environment

We provision via Pulumi.

• Version used: v0.5.1

Might be related

https://github.com/elastic/terraform-provider-ec/issues/343

[pascal-hofmann]

I tested around with v 0.7.0 and different node counts and found out the following:

1. Starting from 6 nodes, you need to specify at least one master via terraform.
2. Below 6 nodes, you must not specify any masters via terraform.
3. When switching from 5 nodes to 6 nodes with master, terraform apply will fail (a subsequent plan will show no drift related to this change though):

| When applying changes to module.[deployment_1.ec](http://deployment_1.ec/)_deployment.this, provider "provider[\"[registry.terraform.io/elastic/ec](http://registry.terraform.io/elastic/ec)\"]" produced an unexpected new value: .elasticsearch.hot.node_roles: planned set element cty.StringVal("master") does not correlate with any element in actual.

4. Switching back from 6 nodes and master to only 5 nodes is not possible at all via terraform (it does work via UI though):

* clusters.cluster_invalid_plan: Cluster must contain at least a master topology element and a data topology element. 'master' node type is missing,'master' node type exists in more than one topology element (resources.elasticsearch[0].cluster_topology)

[dimuon]

Just from a quick look at the issue - I think we need to change node_roles plan modifier to fix item 3 from that list.