aws.s3
aws.s3 copied to clipboard
cloudyr
Access Denied on get_bucket command
Hi,
When running the get_bucket command, I get an HTTP 403 Access Denied error:
install.packages("aws.s3")
install.packages("aws.signature")
install.packages("aws.ec2metadata")
library(aws.s3)
library(aws.signature)
library(aws.ec2metadata)
ACCESS_KEY <- "############################"
SECRET_ACCESS_KEY <- "###########################"
my_bucket <- "##############################"
Sys.setenv("AWS_ACCESS_KEY_ID" = ACCESS_KEY,
"AWS_SECRET_ACCESS_KEY" = SECRET_ACCESS_KEY)
get_bucket(bucket = my_bucket,
check_region = FALSE,
verbose = TRUE)
## response
Checking for credentials in user-supplied values
Checking for credentials in Environment Variables
Using Environment Variable 'AWS_ACCESS_KEY_ID' for AWS Access Key ID
Using Environment Variable 'AWS_SECRET_ACCESS_KEY' for AWS Secret Access Key
Using default value for AWS Region ('us-east-1')
S3 Request URL: https://s3.amazonaws.com/dxae-useast1-external-files/
Executing request with AWS credentials
Checking for credentials in user-supplied values
Using user-supplied value for AWS Access Key ID
Using user-supplied value for AWS Secret Access Key
Using user-supplied value for AWS Region ('us-east-1')
Checking for credentials in user-supplied values
Using user-supplied value for AWS Secret Access Key
Using user-supplied value for AWS Region ('us-east-1')
Parsing AWS API response
Client error: (403) Forbidden
List of 4
$ Code : chr "AccessDenied"
$ Message : chr "Access Denied"
$ RequestId: chr "86DD97C0E28AB6DC"
$ HostId : chr "uyWnFkjCnAPOubfZcARR3sPV9ew1gCKakYkLbc6/dIlmBMMq2gsTVIUmB3toi/VSpNOuiSHXqhI="
- attr(*, "headers")=List of 7
..$ x-amz-bucket-region: chr "us-east-1"
..$ x-amz-request-id : chr "86DD97C0E28AB6DC"
..$ x-amz-id-2 : chr "uyWnFkjCnAPOubfZcARR3sPV9ew1gCKakYkLbc6/dIlmBMMq2gsTVIUmB3toi/VSpNOuiSHXqhI="
..$ content-type : chr "application/xml"
..$ transfer-encoding : chr "chunked"
..$ date : chr "Wed, 26 Sep 2018 07:46:45 GMT"
..$ server : chr "AmazonS3"
..- attr(*, "class")= chr [1:2] "insensitive" "list"
- attr(*, "class")= chr "aws_error"
- attr(*, "request_canonical")= chr "GET\n/###############/\n\nhost:s3.amazonaws.com\nx-amz-date:20180926T074522Z\n\nhost;x-amz-date\ne3"| __truncated__
- attr(*, "request_string_to_sign")= chr "AWS4-HMAC-SHA256\n20180926T074522Z\n20180926/us-east-1/s3/aws4_request\n7676acadde0e02b8223e5caf139021d79dcc34c"| __truncated__
- attr(*, "request_signature")= chr "AWS4-HMAC-SHA256 Credential=###################/20180926/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-d"| __truncated__
NULL
Error in parse_aws_s3_response(r, Sig, verbose = verbose) :
Forbidden (HTTP 403).
## session info for your system
sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1
Matrix products: default
locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252 LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C LC_TIME=English_United States.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] aws.ec2metadata_0.1.5 aws.signature_0.4.4 aws.s3_0.3.12
loaded via a namespace (and not attached):
[1] httr_1.3.1 compiler_3.5.1 R6_2.2.2 tools_3.5.1 base64enc_0.1-3 curl_3.2 Rcpp_0.12.18
[8] xml2_1.2.0 jsonlite_1.5 digest_0.6.16
install.packages("aws.s3") etc is installing the versions from CRAN... did you try the versions from drat/github as described here https://github.com/cloudyr/aws.s3#installation ?... this way you get aws.s3 version 0.3.19, which is much newer. I have seen the same issue here and got it resolved by the newer packages.
Hi @dbast
Thank you for replying. I have tried to installed the latest version (apparently it is 0.3.20 now) from cloudyr, and the error still persists.
Any clue why?
install.packages("aws.s3", repos = c("cloudyr" = "http://cloudyr.github.io/drat"), INSTALL_opts = "--no-multiarch")
install.packages("aws.signature")
install.packages("aws.ec2metadata")
library(aws.s3)
library(aws.signature)
library(aws.ec2metadata)
ACCESS_KEY <- "#####################"
SECRET_ACCESS_KEY <- "########################"
my_bucket <- "############"
Sys.setenv("AWS_ACCESS_KEY_ID" = ACCESS_KEY,
"AWS_SECRET_ACCESS_KEY" = SECRET_ACCESS_KEY)
get_bucket(bucket = my_bucket,
check_region = FALSE,
verbose = TRUE)
Result
Checking for credentials in user-supplied values
Checking for credentials in Environment Variables
Using Environment Variable 'AWS_ACCESS_KEY_ID' for AWS Access Key ID
Using Environment Variable 'AWS_SECRET_ACCESS_KEY' for AWS Secret Access Key
Using default value for AWS Region ('us-east-1')
S3 Request URL: https://s3.amazonaws.com/dxae-useast1-external-files/
Executing request with AWS credentials
Checking for credentials in user-supplied values
Using user-supplied value for AWS Access Key ID
Using user-supplied value for AWS Secret Access Key
Using user-supplied value for AWS Region ('us-east-1')
Checking for credentials in user-supplied values
Using user-supplied value for AWS Secret Access Key
Using user-supplied value for AWS Region ('us-east-1')
Parsing AWS API response
Client error: (403) Forbidden
List of 4
$ Code : chr "AccessDenied"
$ Message : chr "Access Denied"
$ RequestId: chr "B0445822B95A3D7E"
$ HostId : chr "YOiSTm6Pl9FuUz3UTSeELqFtftO7FiZ1gENpzBeWCnGd8EV2qalmE7BXVDcJbicgpYAoT2Dg3kA="
- attr(*, "headers")=List of 7
..$ x-amz-bucket-region: chr "us-east-1"
..$ x-amz-request-id : chr "B0445822B95A3D7E"
..$ x-amz-id-2 : chr "YOiSTm6Pl9FuUz3UTSeELqFtftO7FiZ1gENpzBeWCnGd8EV2qalmE7BXVDcJbicgpYAoT2Dg3kA="
..$ content-type : chr "application/xml"
..$ transfer-encoding : chr "chunked"
..$ date : chr "Thu, 27 Sep 2018 06:41:01 GMT"
..$ server : chr "AmazonS3"
..- attr(*, "class")= chr [1:2] "insensitive" "list"
- attr(*, "class")= chr "aws_error"
- attr(*, "request_canonical")= chr "GET\n/##################/\n\nhost:s3.amazonaws.com\nx-amz-date:20180927T063933Z\n\nhost;x-amz-date\ne3"| __truncated__
- attr(*, "request_string_to_sign")= chr "AWS4-HMAC-SHA256\n20180927T063933Z\n20180927/us-east-1/s3/aws4_request\n3488a495ff0a10148672bb8899c1e564d00d12d"| __truncated__
- attr(*, "request_signature")= chr "AWS4-HMAC-SHA256 Credential=##############/20180927/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-d"| __truncated__
NULL
Error in parse_aws_s3_response(r, Sig, verbose = verbose) :
Forbidden (HTTP 403).
Sessioninfo:
R version 3.5.1 (2018-07-02)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1
Matrix products: default
locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252 LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C LC_TIME=English_United States.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] aws.ec2metadata_0.1.5 aws.signature_0.4.4 aws.s3_0.3.20
loaded via a namespace (and not attached):
[1] httr_1.3.1 compiler_3.5.1 R6_2.2.2 tools_3.5.1 base64enc_0.1-3 curl_3.2 Rcpp_0.12.18 xml2_1.2.0
[9] jsonlite_1.5 digest_0.6.16
I am having the same issue on version 0.3.20. I know it is not actually a permission issue because I can use the AWS CLI using the same credentials and get results back.
CLI call that works:
aws s3 ls s3://data-lake/ --profile prod
R call:
get_bucket("data-lake")
I copied my ~/.aws/credentials variables into my ~/.Renviron so it's not a problem of reading my keys correctly.
Same here but I can't access to S3 from AWS CLI. Probably because of MFA enabled.
I have a Rscript that runs on Rstudio AMI on AWS. The script essential has this logic
aws.s3::get_bucket(
bucket = aws_bucket_name,
key = aws_key,
secret = aws_secret
)
and
aws.s3::s3readRDS(object = "Navarro2016_DIA_DIAumpire_input_FragSummary.RDS", bucket = "xxxxxx")
which when ran through R command prompt or local R studio/ Remote studio GUI works just fine.
However, the script executes as part of the after_install script hooked to AWS CodeDeploy service and fails. The script has a shebang #!/usr/bin/env Rscript and I've modified file permission appropriately so that ./script.R executes in the ec2 command line.
The error message is also un-informative - "Internal Error"
List of 4
$ Code : chr "InternalError"
$ Message : chr "We encountered an internal error. Please try again."
$ RequestId: chr "CE4A60DC1824FF83"
$ HostId : chr "7k7WYL7s+yeX+X0lgukjFgtH0R9NDYMgkbiF4PCSrHFIPoNJArsqZIhkzC1fqM+74GMzA+THCMM="
- attr(*, "headers")=List of 7
..$ x-amz-request-id : chr "CE4A60DC1824FF83"
..$ x-amz-id-2 : chr "7k7WYL7s+yeX+X0lgukjFgtH0R9NDYMgkbiF4PCSrHFIPoNJArsqZIhkzC1fqM+74GMzA+THCMM="
..$ content-type : chr "application/xml"
..$ transfer-encoding: chr "chunked"
..$ date : chr "Tue, 03 Nov 2020 04:04:23 GMT"
..$ server : chr "AmazonS3"
..$ connection : chr "close"
..- attr(*, "class")= chr [1:2] "insensitive" "list"
- attr(*, "class")= chr "aws_error"
- attr(*, "request_canonical")= chr "GET\n/xxxxxxxxxxxxxxx/\n\nhost:s3.amazonaws.com\nx-amz-date:20201103T040418Z\n\nhost;x-amz-date\ne3b0c44298f"| __truncated__
- attr(*, "request_string_to_sign")= chr "AWS4-HMAC-SHA256\n20201103T040418Z\n20201103/us-east-1/s3/aws4_request\na7cb4c55ef6922daf58e810e8772517f8ed2754"| __truncated__
- attr(*, "request_signature")= chr "AWS4-HMAC-SHA256 Credential=xxxxxxxxxxxxxxxxx/20201103/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-da"| __truncated__
I've double-checked IAM roles and policies and made sure S3 is accessible via Rstudio under typical circumstances. I'm trying to figure out if aws.s3 not intended to use outside the scope of the R session?
Thank you
