pinot icon indicating copy to clipboard operation
pinot copied to clipboard

Published 20 hours ago • verified publisher icon apache

Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: DFM25B81QM9RZ2WX, Extended Request ID: 7vSTw22PgYLApWtVYc4blcMDFPRq1NOUG/5tKNqvH/YWCQwyhEO87rv6OOL0AFRf8Ia/+M4voOtLHW656U10Ng==)

Open talhakhan119 opened this issue 2 years ago • 8 comments

I am unable to start my pinot controller pod in AWS EKS environment.

For that I have added configurations in values.template.yaml for pinot to connect it from aws s3

As I have confirmed my serviceaccount have correct permissions and policies to access AWS S3

As I have tested it while exec into another pod using the same service account:

in the pinot pod
apt-get update
apt-get install awscli
# aws s3 ls s3://perceptdata-pinot
              PRE controller-data/
              PRE pinot-data/

Configurations:

  # Extra configs will be appended to pinot-controller.conf file
  extra:
    configs: |-
      pinot.set.instance.id.to.hostname=true
      controller.task.scheduler.enabled=true
      controller.disable.ingestion.groovy=false
      pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
      pinot.controller.storage.factory.s3.region=eu-west-1
      pinot.controller.segment.fetcher.protocols=file,http,s3
      pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher

The data directory for aws s3 bucket is:

data:
    dir: s3://perceptdata-pinot/controller-data

The following Error logs I am facing:

Initializing ControllerFilePathProvider
Data directory: s3://perceptdata-pinot/controller-data/
Failed to start a Pinot [CONTROLLER] at 17.271 since launch
java.lang.RuntimeException: Caught exception while initializing ControllerFilePathProvider
        at org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:569) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.controller.BaseControllerStarter.setUpPinotController(BaseControllerStarter.java:380) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.controller.BaseControllerStarter.start(BaseControllerStarter.java:328) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.service.PinotServiceManager.startController(PinotServiceManager.java:118) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:87) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.lambda$startBootstrapServices$0(StartServiceManagerCommand.java:251) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:304) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startBootstrapServices(StartServiceManagerCommand.java:250) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:196) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:187) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.Command.call(Command.java:33) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.Command.call(Command.java:29) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at picocli.CommandLine.executeUserObject(CommandLine.java:1953) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at picocli.CommandLine.access$1300(CommandLine.java:145) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at picocli.CommandLine.execute(CommandLine.java:2078) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:171) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:202) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
Caused by: org.apache.pinot.controller.api.resources.InvalidControllerConfigException: Caught exception while initializing file upload path provider
        at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.<init>(ControllerFilePathProvider.java:107) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.init(ControllerFilePathProvider.java:49) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:567) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        ... 20 more
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: TMMVB33627KABQ7N, Extended Request ID: VOWG6S7PM21PAQ6FNeGPIMFeMFiz/CnWf+MoJ7pyGTktnYuOEV7mzjNGnXe8DXx4ZP1jCDqXnco=)
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:106) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:84) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:42) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:94) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$5(BaseClientHandler.java:229) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:128) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:154) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:107) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:162) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:91) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at software.amazon.awssdk.services.s3.DefaultS3Client.listObjectsV2(DefaultS3Client.java:5614) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.plugin.filesystem.S3PinotFS.isDirectory(S3PinotFS.java:572) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.plugin.filesystem.S3PinotFS.exists(S3PinotFS.java:437) ~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.spi.filesystem.NoClosePinotFS.exists(NoClosePinotFS.java:74) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.<init>(ControllerFilePathProvider.java:71) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.init(ControllerFilePathProvider.java:49) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:567) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        ... 20 more
Shutting down Pinot Service Manager with all running Pinot instances...
Shutting down Pinot Service Manager admin application...
Deregistering service status handler

As I am using a service account to access aws in my pinot-controller pod as the specs of service account are as:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::333870179331:role/pinot
    meta.helm.sh/release-name: pinot
    meta.helm.sh/release-namespace: pinot-new
  creationTimestamp: "2023-08-25T06:54:37Z"
  labels:
    app: pinot
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/version: 0.2.7
    helm.sh/chart: pinot-0.2.7
    heritage: Helm
    release: pinot
  name: pinot
  namespace: pinot-new
  resourceVersion: "389804078"
  uid: 488a7bf2-15fe-4aac-a3ad-e82c5515b3c9

This is the policy for pinot :

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PinotAccessPolicy",
      "Effect": "Allow",
      "Action": [
        "*"
      ],
      "Resource": [
        "arn:aws:s3:::perceptdata-pinot/*",
        "arn:aws:s3:::perceptdata-pinot"
      ]
    }
  ]
}

talhakhan119 avatar Aug 28 '23 13:08 talhakhan119

Can you post this to the troubleshooting slack channel so that people running into similar problems might be able to help?

Jackie-Jiang avatar Aug 30 '23 20:08 Jackie-Jiang

Can you post this to the troubleshooting slack channel so that people running into similar problems might be able to help?

Sure, Can you please provide the relevant channel link?

talhakhan119 avatar Aug 31 '23 06:08 talhakhan119

https://communityinviter.com/apps/apache-pinot/apache-pinot

Jackie-Jiang avatar Sep 01 '23 04:09 Jackie-Jiang

Running into the similar issue. Is controller supposed to use serviceaccount when making calls to S3?

wahab-io avatar Apr 28 '24 10:04 wahab-io

@wahab-io - Could you try the workaround suggested here ? https://apache-pinot.slack.com/archives/C011C9JHN7R/p1694409197833749

swaminathanmanish avatar Apr 29 '24 23:04 swaminathanmanish

@swaminathanmanish the above link is not working, can you please share the workaround here?

wahab-io avatar Apr 30 '24 18:04 wahab-io

@swaminathanmanish the above link is not working, can you please share the workaround here?

@xiangfu0 had suggested the following in that slack thread to the problem described in this issue -

<<Can you check IAM Role Trust Relationship: Make sure that your EKS nodes (or the EKS service itself) can assume the IAM role arn:aws:iam::333870179331:role/pinot. If the trust relationship is not configured correctly, then your nodes won’t be able to assume this role.

Also check the Annotations: You’ve annotated your service account with eks.amazonaws.com/role-arn. Make sure that this service account is actually being used by your pod. You can check this by describing the pod or statefulset (kubectl describe pod <POD_NAME> -n pinot-new) and looking at the Service Account field >>

swaminathanmanish avatar May 02 '24 05:05 swaminathanmanish

@wahab-io and @talhakhan119 does this hint solve your issue? Or did you find another solution you can share? If so we’d like to close this issue.

hpvd avatar Oct 04 '24 07:10 hpvd