Put task to an S3 bucket fails even though the actual upload succeeded

[njbennett]

This describes an issue that we found a workaround for, but we felt still merited attention. We had a hard time figuring out how to upload screenshots to an S3 bucket; for a while, we were able to upload the file to the bucket, but then the task failed for reasons we don't entirely understand.

We had a task with the following configuration:

          on_failure: &screenshots
            do:
            - put: capybara-screenshots
              params:
                file: prepared-screenshots/*.tar.gz

With this resource definition:

  - name: capybara-screenshots
    type: s3
    source:
      bucket: capybara-failure-screenshots
      access_key_id: {{bam-aws-access-key-id}}
      secret_access_key: {{bam-aws-secret-access-key}}

As far as we've been able to figure out from the documentation, this should work, and in fact the file is uploaded to Amazon, but the put step fails when it tries to pull the file back down. This is the error thrown:

error running command: InvalidParameter: 1 validation errors:
- field too short, minimum length 1: Key

While we fixed the immediate issue by switching to using a versioning file, we're still not sure why we were seeing this behavior in the first place. We had to get a member of the Concourse team to physically look at our set up in order to fix the issue. It's not clear to us how our configuration was different from the example in the documentation. An updated or more robust example, or an explanation of this error message, would have helped us diagnose and fix this problem on our own.

[concourse-bot]

Hi there!

We use Pivotal Tracker to provide visibility into what our team is working on. A story for this issue has been automatically created.

The current status is as follows:

• [ ] #140235895 Put task to an S3 bucket fails even though the actual upload succeeded
• [ ] #112642505 S3 resource should validate that either regexp or versioned_file is specified

This comment, as well as the labels on the issue, will be automatically updated as the status in Tracker changes.

[vito]

One of regexp or versioned_file must be specified for the resource's get and check to work. This is mentioned here but unfortunately because one of them has to be present, they're both marked "optional".

Should probably find a better way of formatting that, and/or add validations to the resource itself so it tells you what went wrong.

[njbennett]

Thanks. Yeah, we ended up using versioned_file option, and it's now working nicely.

[squeedee]

+1 for validation.

[richarddowner]

This seems to still be a problem when using regexp in the get as per example configuration here: https://github.com/concourse/s3-resource#example-configuration

Errors with: error running command: InvalidParameter: 1 validation error(s) found.

• minimum field size of 1, HeadObjectInput.Key.

While using in the resource get: regexp: directory_on_s3/release-(.*).tgz

It works with versioned_file - but seems you cant have * in that field

[ionphractal]

I got the same error trying to store files with this in Minio. It doesn't support versioned_file so I'm stuck with regexp (and there seems to be no notion of folder in Minio either). Whatever I put into regex seems to make no difference (e.g. a group test-(.*).tgz or even version named group test-(?version.*).tgz).

How is this supposed to be used?

[ionphractal]

Have to correct myself: Minio has folders but I still wasn't able to get it working.

[ishneetdua]

Hello Is there any update on how to solve this ?

I am using this template

terraform {
  backend "s3" {}
}

data "terraform_remote_state" "state" {
  backend = "s3"
  config {
    access_key = "${var.access_key}"
    secret_key = "$(var.secret_key)"
    bucket     = "${var.tf_state_bucket}"
    dynamodb_table = "${var.tf_state_table}"
    region     = "${var.region}"
    key        = "${var.pathtokey}"
  }
}

The terraform init :

terraform init -backend-config "bucket=$tf_state_bucket" -backend-config "key=$pathtokey" -backend-config "region=${var.region}" -backend-config "dynamodb_table=$tf_state_table" -backend-config "access_key=XXXXXXXXXXX"  -backend-config "secret_key=XXXXXXXXXX" -plugin-dir=.

I get this error every time:

- minimum field size of 1, GetObjectInput.Key.
2017/10/19 14:49:19 [DEBUG] [aws-sdk-go] DEBUG: Validate Request s3/GetObject failed, not retrying, error InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, GetObjectInput.Key.
2017/10/19 14:49:19 [DEBUG] [aws-sdk-go] DEBUG: Build Request s3/GetObject failed, not retrying, error InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, GetObjectInput.Key.
2017/10/19 14:49:19 [DEBUG] plugin: waiting for all plugin processes to complete...

My S3 bucket has versioning enabled and the file in the bucket is "terraform.tfstate". How should I be using the regexp or versioned_file? I tried to use them but did not work out well.

Thank you

[keraghu]

I want to get all the files from s3 bucket/folder, how to pass the regex to read all the files