Failed to get object lambda-promtail

[asanz-mrmilu]

Describe the bug I'm trying to use lamdba-promtail to put s3 logs from my LB into loki. However it fails at getting the object from S3, saying it cannot get the object and that's all

To Reproduce Steps to reproduce the behavior:

1. Created bucket
2. Created ECR and uploaded image
3. Did terraform apply to create lambda and stuff
4. Tried to test the function with a proper s3-put event (with real filename, principal ID, region & bucket) as if it were a real execution
5. Failed saying it cannot get the object

Expected behavior Got file successfully and see logs in loki+grafana (it never reaches the loki part)

Environment:

• AWS
• Lambda with ECR image build as per README
• Terraform 1.1.8
• Go 1.18.1

Screenshots, Promtail config, or terminal output Tried to debug this locally, but I get an error saying "context deadline exceded" which does not look like a problem related to this.

Here's the error itself (replaced values for security reasons):

Failed to get object beta/AWSLogs/acc-id/elasticloadbalancing/us-west-2/2022/05/10/my-log.log.gz from bucket my-bucket on account

As said, there's not much context on the error itself

Here's the event I've tried as well (with replaced values as well):

{
    "Records": [
      {
        "eventVersion": "2.0",
        "eventSource": "aws:s3",
        "awsRegion": "us-west-2",
        "eventTime": "2022-05-10T10:00:00.000Z",
        "eventName": "ObjectCreated:Put",
        "userIdentity": {
          "principalId": "acc-id"
        },
        "requestParameters": {
          "sourceIPAddress": "127.0.0.1"
        },
        "responseElements": {
          "x-amz-request-id": "EXAMPLE123456789",
          "x-amz-id-2": "EXAMPLE123/5678abcdefghijklambdaisawesome/mnopqrstuvwxyzABCDEFGH"
        },
        "s3": {
          "s3SchemaVersion": "1.0",
          "configurationId": "testConfigRule",
          "bucket": {
            "name": "my-bucket",
            "ownerIdentity": {
              "principalId": "acc-id"
            },
            "arn": "arn:aws:s3:::my-bucket"
          },
          "object": {
            "key": "beta/AWSLogs/acc-id/elasticloadbalancing/us-west-2/2022/05/10/my-log.log.gz",
            "size": 22000,
            "eTag": "0123456789abcdef0123456789abcdef",
            "sequencer": "0A1B2C3D4E5F678901"
          }
        }
      }
    ]
  }

[asanz-mrmilu]

Let me know if you guys need anything else, I can share most of my config except names and IDs

[RavianXReaver]

Yup, I'm experiencing the same issue.

[icyground]

The same issue for me even though the permission to get objects from the bucket is in place for the Lambda function.

[icyground]

The same issue for me even though the permission to get objects from the bucket is in place for the Lambda function.

I found the root cause for my problem: Make sure to place this Lamba function in the private subnets of your VPC. Additionally, the subnets need to have NAT gateways or an S3 gateway endpoint.

[asanz-mrmilu]

I do have it on a VPC (even tho the lambda originally was not intended to have it), and I have it on a public subnet where it should not need a NAT gateway.

[icyground]

I thought so too, try putting it on a private subnet with a NAT gateway or an S3 endpoint, it resolved the very same issue that you have for me. Lambda functions do not follow common logic in a public subnet:

Lambda functions connected to a VPC public subnet cannot typically access the internet. 
To access the internet from a public subnet you need a public IP or you need to route via a NAT that itself has a public IP. 
You also need an Internet Gateway (IGW).