opencost icon indicating copy to clipboard operation
opencost copied to clipboard

Published 20 hours ago • verified publisher icon opencost

"QueryAthenaPaginated: start query error: not found, ResolveEndpointV2" error when enabling aws cloud cost

Open driosalido opened this issue 10 months ago • 6 comments

Describe the bug

When trying to enable aws cloud cost integration the Athena queries fail with GetCloudCost: error getting Athena columns: QueryAthenaPaginated: start query error: not found, ResolveEndpointV2 error.

To Reproduce Steps to reproduce the behavior:

  1. Follow kubecost documentation to enable CUR and Athena Reports

  2. Install via helm chart with the following values

helm upgrade --install opencost opencost-charts/opencost --namespace opencost -f values.yaml

values.yaml content

loglevel: debug

serviceAccount:
  create: false
  name: opencost

opencost:
  cloudCost:
    enabled: true

  customPricing:
    enabled: true
    provider: aws
    costModel:
      description: AWS Provider Configuration. Provides default values used if instance type or spot information is not found.
      CPU: 0.031611
      spotCPU: 0.006655
      RAM: 0.004237
      spotRAM: 0.000892
      GPU: 0.95
      storage: 0.00005479452
      zoneNetworkEgress: 0.01
      regionNetworkEgress: 0.01
      internetNetworkEgress: 0.143

      #Spot Data
      projectID: "<REDACTED>"
      spotLabel: karpenter.sh/capacity-type
      spotLabelValue: spot
      awsSpotDataRegion: eu-west-1
      awsSpotDataBucket: <REDACTED>-spot-datafeed-subscription
      awsSpotDataPrefix: spot_datafeed_subscription

      #Cloud Cost
      athenaBucketName: "s3://<REDACTED>-kubecost-report-athena-query-results"
      athenaDatabase: athenacurcfn_kubecost_report
      athenaProjectID: "<REDACTED>"
      athenaRegion: eu-west-1
      athenaTable: kubecost_report
      athenaWorkgroup: primary

  prometheus:
    internal:
      enabled: true
      serviceName: prometheus-operated
      namespaceName: monitoring
      port: 9090
  ui:
    enabled: true
  metrics:
    serviceMonitor:
      enabled: true
      namespace: monitoring

All the Athena queries fails with the same ResolveEndopointV2 error

opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.537741921Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build[jQvAe]: Starting build back to 2024-03-25 00:00:00 +0000 UTC in blocks of 7d
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.629201778Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: building window [2024-04-02T00:00:00+0000, 2024-04-09T00:00:00+0000)
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.630094366Z INF AthenaIntegration[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: GetCloudCost: [2024-04-02T00:00:00+0000, 2024-04-09T00:00:00+0000)
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.630974887Z DBG AthenaQuerier[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: Performing Query: SELECT column_name FROM information_schema.columns WHERE table_schema = 'athenacurcfn_kubecost_report' AND table_name = 'kubecost_report'
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.632299221Z ERR CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build failed for window [2024-04-02T00:00:00+0000, 2024-04-09T00:00:00+0000): GetCloudCost: error getting Athena columns: QueryAthenaPaginated: start query error: not found, ResolveEndpointV2
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.632336757Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build[jQvAe]:  [2024-04-02T00:00:00+0000, 2024-04-09T00:00:00+0000) in 3.164226ms
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.632362701Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: building window [2024-03-26T00:00:00+0000, 2024-04-02T00:00:00+0000)
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.63238635Z INF AthenaIntegration[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: GetCloudCost: [2024-03-26T00:00:00+0000, 2024-04-02T00:00:00+0000)
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.641906507Z DBG AthenaQuerier[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: Performing Query: SELECT column_name FROM information_schema.columns WHERE table_schema = 'athenacurcfn_kubecost_report' AND table_name = 'kubecost_report'
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.64239071Z ERR CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build failed for window [2024-03-26T00:00:00+0000, 2024-04-02T00:00:00+0000): GetCloudCost: error getting Athena columns: QueryAthenaPaginated: start query error: not found, ResolveEndpointV2
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.64247167Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build[jQvAe]:  [2024-03-26T00:00:00+0000, 2024-04-02T00:00:00+0000) in 10.110034ms
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.642554702Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: building window [2024-03-25T00:00:00+0000, 2024-03-26T00:00:00+0000)
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.64259483Z INF AthenaIntegration[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: GetCloudCost: [2024-03-25T00:00:00+0000, 2024-03-26T00:00:00+0000)
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.642627046Z DBG AthenaQuerier[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: Performing Query: SELECT column_name FROM information_schema.columns WHERE table_schema = 'athenacurcfn_kubecost_report' AND table_name = 'kubecost_report'
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.64287647Z ERR CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build failed for window [2024-03-25T00:00:00+0000, 2024-03-26T00:00:00+0000): GetCloudCost: error getting Athena columns: QueryAthenaPaginated: start query error: not found, ResolveEndpointV2
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.643310785Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build[jQvAe]:  [2024-03-25T00:00:00+0000, 2024-03-26T00:00:00+0000) in 755.261µs
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.644030854Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build[jQvAe]: completed in 106.316763ms
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.537787095Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: Run[jQvAe]: running month-to-date update starting at 2024-04-01 00:00:00 +0000 UTC
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.645031307Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: building window [2024-04-01T00:00:00+0000, 2024-04-08T00:00:00+0000)
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.645542925Z INF AthenaIntegration[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: GetCloudCost: [2024-04-01T00:00:00+0000, 2024-04-08T00:00:00+0000)
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.646072828Z DBG AthenaQuerier[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: Performing Query: SELECT column_name FROM information_schema.columns WHERE table_schema = 'athenacurcfn_kubecost_report' AND table_name = 'kubecost_report'
opencost-59975969d8-lc97j opencost 2024-04-09T10:19:34.66812276Z ERR CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build failed for window [2024-04-01T00:00:00+0000, 2024-04-08T00:00:00+0000): GetCloudCost: error getting Athena columns: QueryAthenaPaginated: start query error: not found, ResolveEndpointV2

Expected behavior Opencost being able to query Athena to get cloud cost data

Screenshots Not applicable

Which version of OpenCost are you using? Helm Chart 1.33.2 OpenCost 1.109.0

Additional context The same configuration and Athena setup works fine with Kubecost. The ServiceAccount IRSA created has the same permissions that the IRSA created for kubecost.

driosalido avatar Apr 09 '24 10:04 driosalido

Logs of kubecost getting data from Athena using the same athena isntance, configuration and permissions

kubecost-cost-analyzer-86d5f66487-ls7bd cloud-cost 2024-04-09T10:51:33.476703066Z INF AthenaIntegration[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: GetCloudCost: [2024-04-08T00:00:00+0000, 2024-04-10T00:00:00+0000)
kubecost-cost-analyzer-86d5f66487-ls7bd cloud-cost 2024-04-09T10:51:34.568892453Z INF CloudCost: IngestionManager: deleting integration with key: <REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results
kubecost-cost-analyzer-86d5f66487-ls7bd cloud-cost 2024-04-09T10:51:38.360051004Z INF CloudCost: IngestionManager: creating integration with key: <REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results
kubecost-cost-analyzer-86d5f66487-ls7bd cloud-cost 2024-04-09T10:51:38.360145526Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build[aBYcC]: Starting build back to 2024-01-09 00:00:00 +0000 UTC in blocks of 7d
kubecost-cost-analyzer-86d5f66487-ls7bd cloud-cost 2024-04-09T10:51:38.360339902Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build[aBYcC]:  [2024-04-02T00:00:00+0000, 2024-04-09T00:00:00+0000) in 99.079µs
kubecost-cost-analyzer-86d5f66487-ls7bd cloud-cost 2024-04-09T10:51:38.360402412Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: Run[aBYcC]: running month-to-date update starting at 2024-04-01 00:00:00 +0000 UTC
kubecost-cost-analyzer-86d5f66487-ls7bd cloud-cost 2024-04-09T10:51:38.360441757Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: building window [2024-04-01T00:00:00+0000, 2024-04-08T00:00:00+0000)
kubecost-cost-analyzer-86d5f66487-ls7bd cloud-cost 2024-04-09T10:51:38.360445353Z INF CloudCost[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: ingestor: build[aBYcC]:  [2024-03-26T00:00:00+0000, 2024-04-02T00:00:00+0000) in 81.531µs
kubecost-cost-analyzer-86d5f66487-ls7bd cloud-cost 2024-04-09T10:51:38.360460343Z INF AthenaIntegration[<REDACTED>/s3://<REDACTED>-kubecost-report-athena-query-results]: GetCloudCost: [2024-04-01T00:00:00+0000, 2024-04-08T00:00:00+0000)

driosalido avatar Apr 09 '24 10:04 driosalido

Hi @driosalido this should have been fixed-- what version are you using?

AjayTripathy avatar Apr 09 '24 14:04 AjayTripathy

image: ghcr.io/opencost/opencost:1.109.0

driosalido avatar Apr 10 '24 07:04 driosalido

Per Slack discussion, is https://github.com/aws/aws-sdk-go-v2/issues/2370 related? OpenCost 1.109.0 appears to have github.com/aws/aws-sdk-go-v2 v1.24.1

mattray avatar Apr 10 '24 07:04 mattray

I'm also facing same issue and with same version of opencost v1.109.0 As an workaround i used opencost v2.2.0 [ which has resolved ResolveEndpointV2 error ] but seems like v2.2.0 is removed as per this https://github.com/opencost/opencost/issues/2683

asuprabhat avatar Apr 15 '24 02:04 asuprabhat

@asuprabhat that build was an accidental release, we should be putting out OpenCost 1.110.0 soon. The 2.2 builds were inadvertent Kubecost-tagged builds

mattray avatar Apr 16 '24 05:04 mattray