[BLUEPRINT] Cost optimization / FinHack pattern

[askulkarni2]

To perform cost allocation calculations for your Amazon EKS cluster, Kubecost retrieves the public pricing information of AWS services and AWS resources from the AWS Price List API. You can also integrate Kubecost with AWS Cost and Usage Report to enhance the accuracy of the pricing information specific to your AWS account. This information includes enterprise discount programs, reserved instance usage, savings plans, and spot usage. To learn more about how the AWS Cost and Usage Report integration works, see AWS Cloud Integration in the Kubecost documentation.

[florentio]

Hello @askulkarni2 I am going to work on this blueprint. My company is very interested by the feature. I will raise a PR soon

[askulkarni2]

@florentio that would be awesome! Let us discuss here on what you are planning to do.

[florentio]

Hello @askulkarni2 In my company, we are are running multiple EKS clusters in different accounts different from from the master payer account.

For the blueprint my proposal will be focused on kubernetes clusters all run in the same account as the master payer account. Having a blueprint for multiple EKS clusters run in different accounts different from the master payer account required more configurations such as terraform for multi account deployment, etc... which may be too complex for the scope. We can still have a workaround and if it is needed, we can also work on that one and propose all the step to achieve it

The documentation provide by kubecost https://guide.kubecost.com/hc/en-us/articles/4407595928087-AWS-Cloud-Integration is quite difficult to automate with terraform within a single terraform apply. First I will propose piece of terraform code for each step and we will see how to bundle them together even if there are some them which required manual action before

Step 1: Setting up the CUR

This can be created by aws_cur_report_definition resource. A terraform apply against this configuration in the main.tf file will set up the CUR

locals {
  report_name = "cur-name"                                      # change this or use a variable
  cur_bucket_name = "kubecost-cur-us-west-2"                    # change this or use a variable
}
## CUR Report
resource "aws_cur_report_definition" "cur" {
  report_name                = local.report_name
  time_unit                  = "DAILY"
  format                     = "Parquet"
  compression                = "Parquet"
  additional_schema_elements = ["RESOURCES"]
  s3_bucket                  = module.cur_bucket.s3_bucket_id
  s3_prefix                  = "reports"
  s3_region                  = local.region
  additional_artifacts       = ["ATHENA"]
}

## CUR bucket policy
data "aws_iam_policy_document" "bucket_policy" {
  statement {
    principals {
      type        = "Service"
      identifiers = ["billingreports.amazonaws.com"]
    }
    actions = [
      "s3:GetBucketAcl",
      "s3:GetBucketPolicy"
    ]
    resources = [module.cur_bucket.s3_bucket_arn]
  }
  statement {
    principals {
      type        = "Service"
      identifiers = ["billingreports.amazonaws.com"]
    }
    actions = [
      "s3:PutObject"
    ]
    resources = [
      "${module.cur_bucket.s3_bucket_arn}/*",
    ]
  }
}

## CUR bucket 
module "cur_bucket" {
   source  = "terraform-aws-modules/s3-bucket/aws"
   version = "~> 3.0"

   bucket = local.cur_bucket_name
   acl    = "private"

   force_destroy = true

   # Bucket policies
   attach_policy           = true
   policy                  = data.aws_iam_policy_document.bucket_policy.json

   block_public_acls       = true
   block_public_policy     = true
   ignore_public_acls      = true
   restrict_public_buckets = true

   tags = local.tags
 }

We must wait at least 24hours to proceed the next terraform operation because AWS may take several hours to publish data(up to 24 hours) to the bucket and we need to download crawler-cfn.yml which will be generated before moving forward

Step 2: Setting up Athena

• Create a folder athena-integration in the root folder at the same level as main.tf (here you can give any name to the folder but let's keep in mind to use the same name later for terraform code)
• crawler-cfn.yml generated by AWS has to be downloaded from the cur_bucket bucket and save into the athena-integration folder. The path where to download the file is described in the documentation https://guide.kubecost.com/hc/en-us/articles/4407595928087-AWS-Cloud-Integration#step-2-setting-up-athena
• The configuration below can be appended to the main.tf in order to deploy the athena integration and terraform apply with -target can be used to apply this step

locals {
  athena_bucket_name = "aws-athena-query-results-kubecost-cur"  # change this or use a variable
  athena_stack_name = "athena-integration"                      # change this or use a variable
}

resource "aws_cloudformation_stack" "athena_integration" {
  name = local.athena_stack_name
  template_body = file("athena-integration/crawler-cfn.yml")
  tags = local.tags
}

module "athena_result_bucket" {
   source  = "terraform-aws-modules/s3-bucket/aws"
   version = "~> 3.0"

   bucket = local.athena_bucket_name

   # For example only - please evaluate for your environment
   force_destroy = true

   tags = local.tags
}

There is 3 last action for that step which, at this moment the only way to achieve them is manual action within the console ( I am opened to any automation solution within terraform or something else) :

• Navigate to https://console.aws.amazon.com/athena
• Click Settings
• Set Query result location to the S3 bucket you just created (bucket which name is module.athena_result_bucket.s3_bucket_id or local.athena_bucket_name)

Step 3: Setting up IAM permissions

Lets' remember here that we are going to focus only on the option My kubernetes clusters all run in the same account as the master payer account

Below is the config to be appended to main.tf and to be apply for setup

# Policy for cur and athena
data "aws_iam_policy_document" "athena_cur_policy" {
  statement {
    sid       = "AthenaAccess"
    effect    = "Allow"
    actions   = ["athena:*"]
    resources = ["*"]
  }
  statement {
    sid    = "ReadAccessToAthenaCurDataViaGlue"
    effect = "Allow"
    actions = [
      "glue:GetDatabase*",
      "glue:GetTable*",
      "glue:GetPartition*",
      "glue:GetUserDefinedFunction",
      "glue:BatchGetPartition"
    ]
    resources = [
      "arn:aws:glue:*:*:catalog",
      "arn:aws:glue:*:*:database/athenacurcfn*",
      "arn:aws:glue:*:*:table/athenacurcfn*/*"
    ]
  }
  statement {
    sid    = "AthenaQueryResultsOutput"
    effect = "Allow"
    actions = [
      "s3:GetBucketLocation",
      "s3:GetObject",
      "s3:ListBucket",
      "s3:ListBucketMultipartUploads",
      "s3:ListMultipartUploadParts",
      "s3:AbortMultipartUpload",
      "s3:CreateBucket",
      "s3:PutObject"
    ]
    resources = [
      "arn:aws:s3:::aws-athena-query-results-*"
    ]
  }

  statement {
    sid    = "S3ReadAccessToAwsBillingData"
    effect = "Allow"
    actions = [
      "s3:Get*",
      "s3:List*"
    ]
    resources = [
      "arn:aws:s3:::${module.cur_bucket.s3_bucket_id}*"
    ]
  }
}

# policy for spot feed
data "aws_iam_policy_document" "spot_feed_policy" {
  statement {
    sid    = "SpotDataAccess"
    effect = "Allow"
    actions = [
      "s3:ListAllMyBuckets",
      "s3:ListBucket",
      "s3:HeadBucket",
      "s3:HeadObject",
      "s3:List*",
      "s3:Get*"
    ]
    resources = [
      "arn:aws:s3:::${local.spot_data_bucket_name}*"
    ]
  }
}

resource "aws_iam_policy" "athena_cur_policy" {
  description = "IAM Policy for IRSA"
  name_prefix = "kubecost-policy"
  policy      = data.aws_iam_policy_document.athena_cur_policy.json
}

resource "aws_iam_policy" "spot_feed_policy" {
  description = "IAM Policy for IRSA"
  name_prefix = "spot-feed-policy"
  policy      = data.aws_iam_policy_document.spot_feed_policy.json
}

Step 4: Attaching IAM permissions to Kubecost

locals {
  namespace        = "kubecost"
  service_account  = "cost-analyzer"
}

# irsa for kubecost
module "kubecost_irsa" {
  source = "../../modules/irsa"

  eks_cluster_id             = module.eks_blueprints.eks_cluster_id
  eks_oidc_provider_arn      = module.eks_blueprints.eks_oidc_provider_arn
  irsa_iam_policies          = [aws_iam_policy.athena_cur_policy.arn, aws_iam_policy.spot_feed_policy.arn]
  kubernetes_namespace       = local.namespace
  kubernetes_service_account = local.service_account
}

Kubecost addons configuration

here is the values file to be used to be save under kubecost-values.yaml file

global:
  grafana:
    enabled: false
    proxy: false

imageVersion: prod-1.96.0
kubecostFrontend:
  image: public.ecr.aws/kubecost/frontend

kubecostModel:
  image: public.ecr.aws/kubecost/cost-model

serviceAccount:
  create: false
  name: ${service-account}
  annotations:
    eks.amazonaws.com/role-arn: ${iam-role-arn}
  # name: kc-test
kubecostMetrics:
  emitPodAnnotations: true
  emitNamespaceAnnotations: true

prometheus:
  server:
    image:
      repository: public.ecr.aws/kubecost/prometheus
      tag: v2.35.0

  configmapReload:
    prometheus:
      image:
        repository: public.ecr.aws/bitnami/configmap-reload
        tag: 0.7.1

reporting:
  productAnalytics: false

kubecostProductConfigs:
  projectID: ${project-id}
  spotLabel: ${spot-label}
  spotLabelValue: ${spot-label-value}
  awsSpotDataRegion: ${spot-region}
  awsSpotDataBucket:  ${spot-bucket}
  awsSpotDataPrefix: ${spot-bucket-prefix}
  athenaProjectID: ${athena-project-id}
  athenaBucketName: s3://${athena-bucket-name}
  athenaRegion: ${athena-region}
  athenaDatabase: ${athena-db-name}
  athenaTable: ${athena-table-name}

then the terrafom code will be

locals {
  spot_label       = "lifecycle"
  spot_label_value = "spot"
  athena_db        = "athenacurcfn_test"
  athena_table     = "athenacurcfn_test_"
}

data "aws_caller_identity" "current" {}

module "eks_blueprints_kubernetes_addons" {
  source = "../../modules/kubernetes-addons"

  eks_cluster_id       = module.eks_blueprints.eks_cluster_id
  eks_cluster_endpoint = module.eks_blueprints.eks_cluster_endpoint
  eks_oidc_provider    = module.eks_blueprints.oidc_provider
  eks_cluster_version  = module.eks_blueprints.eks_cluster_version

  # EKS Managed Add-ons
  enable_amazon_eks_vpc_cni            = true
  enable_amazon_eks_coredns            = true
  enable_amazon_eks_kube_proxy         = true
  enable_amazon_eks_aws_ebs_csi_driver = true

  # Add-ons
  enable_kubecost = true
  kubecost_helm_config = {
    namespace        = local.namespace
    create_namespace = false
    values = [
      templatefile("kubecost-values.yaml", {
        service-account    = local.service_account
        iam-role-arn       = module.kubecost_irsa.irsa_iam_role_arn
        project-id         = data.aws_caller_identity.current.account_id # the Account ID of the AWS Account on which the spot nodes are running.
        spot-label         = local.spot_label                            # optional Kubernetes node label name designating whether a node is a spot node. Used to provide pricing estimates until exact spot data becomes available from the CUR
        spot-label-value   = local.spot_label_value                      # optional Kubernetes node label value designating a spot node. Used to provide pricing estimates until exact spot data becomes available from the CUR
        spot-region        = local.region                                # region of your spot data bucket
        spot-bucket        = local.spot_data_bucket_name                 # the configured bucket for the spot data feed
        bucket-prefix      = local.spot_data_bucket_prefix               # optional configured prefix for your spot data feed bucket
        athena-project-id  = data.aws_caller_identity.current.account_id # The AWS AccountID where the Athena CUR is.
        athena-bucket-name = local.athena_bucket_name                    # The s3 bucket to store Athena query results that you’ve created that Kubecost has permission to access
        athena-region      = local.region                                # The aws region athena is running in
        athena-db-name     = local.athena_db                             # the athena database name is available as the value (physical id) of AWSCURDatabase in the CloudFormation stack created above
        athena-table-name  = local.spot_data_bucket_prefix               # the name of the table created by the Athena setup
      })
    ]
  }

  tags = local.tags
}

What do you think :) ?

[florentio]

Looking forward your inputs in order to elaborate it better and raise a PR. thanks

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days