sergeant icon indicating copy to clipboard operation
sergeant copied to clipboard

Add example of accessing S3 files

Open hrbrmstr opened this issue 6 years ago • 3 comments

https://issues.apache.org/jira/browse/DRILL-6662 makes it possible to use non-hardcoded creds so it finally makes sense to add some examples of how to query S3 data.

hrbrmstr avatar Dec 21 '18 11:12 hrbrmstr

If your example could include referencing a specific regional endpoint, that would be A++ good. My first attempt at getting IAM roles and a regionally scoped bucket call to work failed and I've yet to go back and make another attempt.

davidski avatar Jan 22 '19 18:01 davidski

I gave it a quick try the day 1.15.0 came out but didn't go back to it.

hrbrmstr avatar Jan 22 '19 18:01 hrbrmstr

Looks like I just needed to come back to this. Got this working on a us-west-2 S3 endpoint with the following (excessively verbose) storage configuration:

{
  "type": "file",
  "connection": "s3a://cloudy-mccloudface",
  "config": {
    "fs.s3a.aws.credentials.provider": "com.amazonaws.auth.InstanceProfileCredentialsProvider",
    "fs.s3a.endpoint": "s3.us-west-2.amazonaws.com"
  },
  "workspaces": {
    "tmp": {
      "location": "/tmp",
      "writable": true,
      "defaultInputFormat": null,
      "allowAccessOutsideWorkspace": false
    },
    "root": {
      "location": "/",
      "writable": false,
      "defaultInputFormat": null,
      "allowAccessOutsideWorkspace": false
    },
    "csvs": {
      "location": "/csvs",
      "writable": false,
      "defaultInputFormat": null,
      "allowAccessOutsideWorkspace": false
    }
  },
  "formats": {
    "psv": {
      "type": "text",
      "extensions": [
        "tbl"
      ],
      "delimiter": "|"
    },
    "csv": {
      "type": "text",
      "extensions": [
        "csv"
      ],
      "delimiter": ","
    },
    "tsv": {
      "type": "text",
      "extensions": [
        "tsv"
      ],
      "delimiter": "\t"
    },
    "parquet": {
      "type": "parquet"
    },
    "json": {
      "type": "json",
      "extensions": [
        "json"
      ]
    },
    "avro": {
      "type": "avro"
    },
    "sequencefile": {
      "type": "sequencefile",
      "extensions": [
        "seq"
      ]
    },
    "csvh": {
      "type": "text",
      "extensions": [
        "csvh"
      ],
      "extractHeader": true,
      "delimiter": ","
    }
  },
  "enabled": true
}

A bit too shagged out to write this up properly right now, so dumping the config as a reminder for later.

davidski avatar Jan 23 '19 03:01 davidski