aws.s3
aws.s3 copied to clipboard
Retrieving data from an S3 bucket
I'm currently using the aws.s3
package in R to extract data from various CSV's in one of my AWS S3 buckets. The data is stored as seen below.
What I'm trying to do is extract ALL of the data currently in .csv
format. So if I were just doing the first file there it would look as such:
data <- aws.s3::s3read_using(read.csv, object = "s3://test-testpilot/09242021-testpilot-michigan-script_results-09-24-2021-871107619.csv")
But the problem is, I want to extract all of the files and put them into three separate data frames. The first dataframe would be all the files that contain -script_results-
. The second dataframe would be any file with just -results-
in it, and the final dataframe would be any .csv file with -callers-
.
What would be the most effective way to go about doing this?
$Contents
Key: 09242021-testpilot-michigan-script_results-09-24-2021-871107619.csv
LastModified: 2021-09-25T01:36:35.000Z
ETag: "7ee0a56df05121e815b2f1bc2fba819d"
Size (B): 28738
Owner:
Storage class: STANDARD
$Contents
Key: 09242021-testpilot-nevada-callers-09-24-2021-937053503.csv
LastModified: 2021-09-25T07:36:56.000Z
ETag: "3363a2788d9f3c9822a0a634c7c07095"
Size (B): 889
Owner:
Storage class: STANDARD
$Contents
Key: 09242021-testpilot-nevada-results-09-24-2021-704158614.csv
LastModified: 2021-09-25T07:36:57.000Z
ETag: "c69d06426351db42d03a4c0616e526eb"
Size (B): 85399
Owner:
Storage class: STANDARD
$Contents
Key: 09242021-testpilot-nevada-script_results-09-24-2021-780474261.csv
LastModified: 2021-09-25T07:36:57.000Z
ETag: "b6be083391b484a5ab2a2574a1c6fa4a"
Size (B): 19051
Owner:
Storage class: STANDARD
$Contents
Key: 09242021-testpilot-new-hampshire-callers-09-24-2021-779182855.csv
LastModified: 2021-09-25T01:55:30.000Z
ETag: "867c55d68310e9a55d95f1b1b5ef5b96"
Size (B): 1903
Owner:
Storage class: STANDARD
$Contents
Key: 09242021-testpilot-new-hampshire-results-09-24-2021-574695815.csv
LastModified: 2021-09-25T01:55:31.000Z
ETag: "da4026197af765d0ef076f0f61387903"
Size (B): 96827
Owner:
Storage class: STANDARD
$Contents
Key: 09242021-testpilot-new-hampshire-script_results-09-24-2021-830415712.csv
LastModified: 2021-09-25T01:55:31.000Z
ETag: "0b8942e4bb80787cb6e4bbda10dfcc3c"
Size (B): 19509
Owner:
Storage class: STANDARD
$Contents
Key: 09242021-testpilot-north-carolina-callers-09-24-2021-728851613.csv
LastModified: 2021-09-25T01:42:34.000Z
ETag: "b72c362480a1ae171f6edfc3cca861b3"
Size (B): 2316
Owner:
Storage class: STANDARD
$Contents
Key: 09242021-testpilot-north-carolina-results-09-24-2021-690797258.csv
LastModified: 2021-09-25T01:42:35.000Z
ETag: "0ab9a52dff07518668fe5a36e5ca16c7"
Size (B): 636831
Owner:
Storage class: STANDARD
$Contents
Key: 09242021-testpilot-north-carolina-script_results-09-24-2021-552436920.csv
LastModified: 2021-09-25T01:42:35.000Z
ETag: "79a4492084a57ee07fbb0fc4550e6ec9"
Size (B): 88138
Owner:
Storage class: STANDARD
$Contents
Key: 09242021-testpilot-pennsylvania-callers-09-24-2021-695602875.csv
LastModified: 2021-09-25T01:41:51.000Z
ETag: "bc9e2f80431ba40352f86a759b6a697f"
Size (B): 1750
Owner:
Storage class: STANDARD