Unable to do resize operation on EMR instance group

[stormic-nomad-nishant]

The tf code doesn't allow to resize the cluster. Instead it destroys all the nodes and rebuilds them when instance_count is increased . ex: 1-->2

[keerthi-penmatsa]

Hi, I am facing a similar issue. For my requirement, I wanted to set instance_count = "0" in the { name = "CoreInstanceGroup" instance_role = "CORE" instance_type = "m3.xlarge" instance_count = "0" bid_price = "0.30" }, ]

and terraform errors during apply. It says "* aws_emr_cluster.cluster: ValidationException: An instance group must have at least one instance." Would like to know the reason behind this error.

[jeancochrane]

Hey @keerthi-penmatsa! I'm guessing that Terraform is confused why you would specify a configuration block for an instance group but then not deploy any instances to it. What behavior were you expecting? If you were trying to delete the instance group, I'd try removing the map definition you posted from your instance_group list entirely, and see if Terraform understands that you're trying to delete the existing instance group.

[keerthi-penmatsa]

Hi @jeancochrane , Thank you for your response. We wanted to use EMR as a module and enable Users to provide their preference through datafile. Few use cases might require CORE/TASK nodes while the others might not. The below is the way we variablized in our code:

  instance_group = [
    {
      name           = "MasterInstanceGroup"
      instance_role  = "MASTER"
      instance_type  = "${var.emr_master_instance_type}"
      instance_count = "${var.emr_master_instance_count}"
    },
    {
      name           = "CoreInstanceGroup"
      instance_role  = "CORE"
      instance_type  = "${var.emr_core_instance_type}"
      instance_count = "${var.emr_core_instance_count}"

      ebs_config {
        size                 = "${var.emr_core_instance_size}"
        type                 = "${var.emr_core_ebs_type}"
        volumes_per_instance = "${var.emr_core_volumes}"
      }
        autoscaling_policy  = "${var.asg_policy ? data.template_file.autoscaling_policy.rendered : "" }"
    },
    {
      name           = "TaskInstanceGroup"
      instance_role  = "TASK"
      instance_type  = "${var.emr_task_instance_type}"
      instance_count = "${var.emr_task_instance_count}"

      ebs_config {
        size                 = "${var.emr_task_instance_size}"
        type                 = "${var.emr_task_ebs_type}"
        volumes_per_instance = "${var.emr_task_volumes}"
      }
        autoscaling_policy  = "${var.asg_policy ? data.template_file.autoscaling_policy.rendered : "" }"
    },
  ]

I cannot supply instance_groups as a single variable as is the case with your github code because, I gotto to be able to use autoscaling_policy rendered o/p. In order to get past this limitation, I tried to use formatstring with the variable but it did not work as Terraform was unable to read the rendered o/p. This might be a seperate issue but appreciate any inputs on resolving the same.

[jeancochrane]

@keerthi-penmatsa Are you using our module as part of your module? It sounds like maybe not -- if that's the case, let's continue this conversation elsewhere so that we can keep this issue on track. If you can point me to your repo I'd be happy to continue the discussion there.

[keerthi-penmatsa]

@jeancochrane - I did try to use the module and got stuck with using autoscaling with your module. Is there a way we can use auto-scaling with instance groups using your module?

and yeah, I am with you.. the above examples are not directly related to your module, apologies for posting on the wrong thread.