aws_grafana_workspace service managed permission type does not automatically provision the permissions

[tanvp112]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform 1.1.9, AWS Provider 4.10.0

Affected Resource(s)

aws_grafana_workspace

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

resource "aws_grafana_workspace" "example" {
  account_access_type      = "CURRENT_ACCOUNT"
  authentication_providers = ["SAML"]
  permission_type          = "SERVICE_MANAGED"
  role_arn                 = aws_iam_role.assume.arn
}

resource "aws_iam_role" "assume" {
  name = "grafana-assume"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Sid    = ""
        Principal = {
          Service = "grafana.amazonaws.com"
        }
      },
    ]
  })
}

Debug Output

NA

Panic Output

NA

Expected Behavior

As per the documentation stated:

Permission_type - (Required) The permission type of the workspace. If SERVICE_MANAGED is specified, the IAM roles and IAM policy attachments are generated automatically.

Actual Behavior

No permission was provisioned, resulted to no data sources can be added in AMG workspace console.

Only the assume role is created correctly.

Steps to Reproduce

1. terraform apply

Important Factoids

NA

References

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/grafana_workspace

[justinretzolk]

Hey @tanvp112 👋 Thank you for taking the time to raise this! It looks like the documentation for this resource could use a little help to make things a bit more clear. I noticed when looking at the API documentation for the underlying API call that there's a couple of additional bits of configuration that'll be needed to allow AWS to automatically generate the correct role/policy attachments:

• workspaceDataSources: Specify the AWS data sources that you want to be queried in this workspace. Specifying these data sources here enables Amazon Managed Grafana to create IAM roles and permissions that allow Amazon Managed Grafana to read data from these sources. You must still add them as data sources in the Grafana console in the workspace.
• workspaceNotificationDestinations: Specify the AWS notification channels that you plan to use in this workspace. Specifying these data sources here enables Amazon Managed Grafana to create IAM roles and permissions that allow Amazon Managed Grafana to use these channels.

In the resource, these translate to the data_sources and notification_destinations, respectively. Can you add these arguments with the appropriate values for your configuration and verify that that resolves the issue you're running into?

[tanvp112]

Hi @justinretzolk , added as suggested but the outcome is the same :(

resource "aws_grafana_workspace" "example" {
  account_access_type      = "CURRENT_ACCOUNT"
  authentication_providers = ["SAML"]
  permission_type          = "SERVICE_MANAGED"
  role_arn                 = aws_iam_role.assume.arn
  data_sources              = ["PROMETHEUS"]
  notification_destinations = ["SNS"]
}

resource "aws_iam_role" "assume" {
  name = "grafana-assume"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Sid    = ""
        Principal = {
          Service = "grafana.amazonaws.com"
        }
      },
    ]
  })
}

An additional finding is on the console the service-managed policy seems to have enabled but somehow no permission was added to the role:

Still cannot list nor add any managed Prometheus data source on the Grafana console.

[sollie]

I'm seeing the same issues as @tanvp112. When creating a managed grafana from the AWS Console, policies are created and attached. When creating from Terraform (with data_sources specified), the UI says data sources are attached, but not policies are created or attached.

[bryantbiggs]

For reference - https://github.com/terraform-aws-modules/terraform-aws-managed-service-grafana/pull/3

[philicious]

Quote from AWS CLI doc, which sounds like this is unfortunately on purpose

"If you specify SERVICE_MANAGED on AWS Grafana console, Amazon Managed Grafana automatically creates the IAM roles and provisions the permissions that the workspace needs to use Amazon Web Services data sources and notification channels. In CLI mode, the permissionType SERVICE_MANAGED will not create the IAM role for you."

to make things worse, the AWS doc reads the same but misses this crucial hint regarding CLI ( and API most likely)

[ScottGuymer]

So it seems like SERVICE_MANAGED will not work from terraform in its current state.

Which means that the data_sources and notification_destinations are also not really functioning as expected.

[ScottGuymer]

To make things slightly more confusing with this config

resource "aws_grafana_workspace" "example" {
  account_access_type      = "CURRENT_ACCOUNT"
  authentication_providers = ["AWS_SSO"]
  permission_type          = "CUSTOMER_MANAGED"
  role_arn                 = aws_iam_role.grafana.arn
}

you get the following error

╷
│ Error: error updating Grafana Workspace (g-xxxx): ValidationException: When the permissionType is CUSTOMER_MANAGED a Workspace Role ARN should be provided.
│ {
│   RespMetadata: {
│     StatusCode: 400,
│     RequestID: "xxxxac64"
│   },
│   Message_: "When the permissionType is CUSTOMER_MANAGED a Workspace Role ARN should be provided."
│ }
│
│   with aws_grafana_workspace.example,
│   on grafana.tf line 7, in resource "aws_grafana_workspace" "example":
│    7: resource "aws_grafana_workspace" "example" {
│
╵

[centur]

If anyone is still interested in getting this working before official fixes - I managed to find a workaround of the current state of bugged aws_grafana_workspace by manually creating and attaching policy to a role and adjusting STS assume a bit. And now my grafana workspace, created with permission_type = "SERVICE_MANAGED" can actually read and access data.

Trick is 2 parts fixes:

1. Manually create policy and attach it to the "managed" role. This part is obvious from the mentioned aws docs. This covers the missing policies - we just creating them manually and attaching to a known role, even if permission type is set to service_managed. Referenced PR https://github.com/terraform-aws-modules/terraform-aws-managed-service-grafana/pull/3 does it for Current account, but for Service_managed - still need to manually craft them.

2. Update Assume statement and add assume role principal to it. This is a bit hacky, but the hint to it is error message from grafana when one tries to explore available metrics - metric request error: "AccessDenied: User: arn:aws:sts::<accountID>:assumed-role/<ROLE>/amg-session is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::<accountID>:role/<ROLE>

indicates that we need to extend allowed principals in AssumeRole policy. Default is not enough:

...
   actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["grafana.amazonaws.com"]
    }
...

So the end result looks something like this (It's hacky but it works with quite old TF and module version):

// Extended document with AWS amg-session principal in it.
data "aws_iam_policy_document" "grafana_assume" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["grafana.amazonaws.com"]
    }
    // Trick part 2
    principals {
      type        = "AWS"
      // NOTE : no wildcards here so we have to manually hack togethers the STS ID
      identifiers = ["arn:aws:sts::${local.account_id}:assumed-role/grafana-${local.name}/amg-session"]
    }
  }
}

// Trick part 1, statements from https://docs.aws.amazon.com/grafana/latest/userguide/AMG-manage-permissions.html
data "aws_iam_policy_document" "grafana_access" {
  statement {
    sid    = "AllowReadingMetricsFromCloudWatch"
    ...
  }
  statement {
    sid    = "AllowReadingLogsFromCloudWatch"
   ...
  }
  statement {
    sid    = "AllowSNSAlerts"
    effect = "Allow"
    actions = ["sns:Publish"]
    resources = ["arn:aws:sns:*:${local.account_id}:grafana*" ]
  }
}

resource "aws_iam_policy" "grafana_role_access" {
  name        = "grafana-${local.name}"
  policy      = data.aws_iam_policy_document.grafana_access.json
}

// This role is assumed by Workspace when accessing resources
resource "aws_iam_role" "grafana_role" {
  name               = "grafana-${local.name}"
  assume_role_policy = data.aws_iam_policy_document.grafana_assume.json // Trick part 2
  managed_policy_arns = [ aws_iam_policy.grafana_role_access.arn ]
}

resource "aws_grafana_workspace" "grafana_main" {
  name        = local.name
  account_access_type      = "CURRENT_ACCOUNT"
  permission_type          = "SERVICE_MANAGED"
  authentication_providers = ["AWS_SSO"]
  data_sources              = ["CLOUDWATCH"]
  notification_destinations = ["SNS"]
  role_arn = aws_iam_role.grafana_role.arn
}

Hope this helps and 🤞 for patched version where we don't need to hack around this quirk.

[vHanda]

I also worked around this bug by creating the grafana workspace via the AWS Console, looking at the permissions that are added by AWS, and manually adding those to my terraform configuration.

Here is mine for Cloudwatch + Prometheus -

const role = new iam.IamRole(stack, "grafana-role", {
  name: "grafana-role",
  assumeRolePolicy: JSON.stringify({
    Version: "2012-10-17",
    Statement: [
      {
        Effect: "Allow",
        Sid: "",
        Principal: {
          Service: "grafana.amazonaws.com",
        },
        Action: "sts:AssumeRole",
      },
    ],
  }),
});

// Workaround for "https://github.com/hashicorp/terraform-provider-aws/issues/24342"
// I'm having to create the policies manually for now.
const cloudWatchPolicy = new iam.IamPolicy(
  stack,
  "grafana-cloudwatch-policy",
  {
    name: "grafana-cloudwatch-policy",
    policy: JSON.stringify({
      Version: "2012-10-17",
      Statement: [
        {
          Sid: "AllowReadingMetricsFromCloudWatch",
          Effect: "Allow",
          Action: [
            "cloudwatch:DescribeAlarmsForMetric",
            "cloudwatch:DescribeAlarmHistory",
            "cloudwatch:DescribeAlarms",
            "cloudwatch:ListMetrics",
            "cloudwatch:GetMetricStatistics",
            "cloudwatch:GetMetricData",
            "cloudwatch:GetInsightRuleReport",
          ],
          Resource: "*",
        },
        {
          Sid: "AllowReadingLogsFromCloudWatch",
          Effect: "Allow",
          Action: [
            "logs:DescribeLogGroups",
            "logs:GetLogGroupFields",
            "logs:StartQuery",
            "logs:StopQuery",
            "logs:GetQueryResults",
            "logs:GetLogEvents",
          ],
          Resource: "*",
        },
        {
          Sid: "AllowReadingTagsInstancesRegionsFromEC2",
          Effect: "Allow",
          Action: [
            "ec2:DescribeTags",
            "ec2:DescribeInstances",
            "ec2:DescribeRegions",
          ],
          Resource: "*",
        },
        {
          Sid: "AllowReadingResourcesForTags",
          Effect: "Allow",
          Action: "tag:GetResources",
          Resource: "*",
        },
      ],
    }),
  }
);

new iam.IamRolePolicyAttachment(
  stack,
  "grafana-cloudwatch-policy-attachment",
  {
    role: role.name,
    policyArn: cloudWatchPolicy.arn,
  }
);

const promPolicy = new iam.IamPolicy(stack, "grafana-prom-policy", {
  name: "grafana-prom-policy",
  policy: JSON.stringify({
    Version: "2012-10-17",
    Statement: [
      {
        Effect: "Allow",
        Action: [
          "aps:ListWorkspaces",
          "aps:DescribeWorkspace",
          "aps:QueryMetrics",
          "aps:GetLabels",
          "aps:GetSeries",
          "aps:GetMetricMetadata",
        ],
        Resource: "*",
      },
    ],
  }),
});

new iam.IamRolePolicyAttachment(stack, "grafana-prom-policy-attachment", {
  role: role.name,
  policyArn: promPolicy.arn,
});

const grafana = new GrafanaWorkspace(stack, "medl-grafana", {
  name: "medl-grafana",
  accountAccessType: "CURRENT_ACCOUNT",
  authenticationProviders: ["AWS_SSO"],
  permissionType: "SERVICE_MANAGED",
  dataSources: ["CLOUDWATCH", "PROMETHEUS"],
  roleArn: role.arn,
});

new TerraformOutput(stack, "grafana_endpoint", {
  value: grafana.endpoint,
});

[lacleung]

As issue is still outstanding, I'd additionally like to mention that the SERVICE_MANAGED permission_type should also automatically generate the iam role according to the documentation:

permission_type - (Required) The permission type of the workspace. If SERVICE_MANAGED is specified, the IAM roles and IAM policy attachments are generated automatically. If CUSTOMER_MANAGED is specified, the IAM roles and IAM policy attachments will not be created.

However, it is not possible to deploy a configuration without a role_arn specified (and account_access_type set to CURRENT_ACCOUNT), as such there is no autogenerated service role when deploying the workspace. It is my understanding that the entire SERVICE_MANAGED permission type is not functioning as intended.

│ Error: creating Grafana Workspace: ValidationException: When the accountAccessType is CURRENT_ACCOUNT a Workspace Role ARN should be provided.
│ {
│   RespMetadata: {
│     StatusCode: 400,
│     RequestID: "9e4311e1-0a15-4ae5-85ed-10153e19866a"
│   },
│   Message_: "When the accountAccessType is CURRENT_ACCOUNT a Workspace Role ARN should be provided."
│ }

[ktibi]

Hello,

Same issue when we choose organization for the account_access_type variable. With the AWS console, aws provide a stackset to deploy role in each account from the org.

So if I understand for now, the workarround is to deploy the stackset directly with a ressource ?

[elliottkopp]

Also running into this issue. Does not work from terraform. Had to manually create my own roles, policies, and trust relationships to get this to work. Works great from the management console.