ice
ice copied to clipboard
Teevity
billing file download failed
Hello
I've been trying to configure Ice but It's not easy for me I got failed to download my billing object which is in S3 I used AWS master account and I've already changed the target object by modifying the start_millisecond, but It didn't work also.
Do you have any ideas?
Here are my Error logs
| Error 2017-02-10 02:18:56,136 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.BillingFileProcessor - Error reading from file lastProcessMillis_2016-08 Message: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: CE0491D891CEAEE6) Line | Method ->> 1077 | handleErrorResponse in com.amazonaws.http.AmazonHttpClient
| 725 | executeOneRequest in '' | 460 | executeHelper . . . in '' | 295 | execute in '' | 3710 | invoke . . . . . . in com.amazonaws.services.s3.AmazonS3Client | 1146 | getObject in '' | 1018 | getObject . . . . . in '' | 658 | getLastMillis in com.netflix.ice.processor.BillingFileProcessor | 672 | lastProcessTime . . in '' | 151 | poll in '' | 50 | doWork . . . . . . in com.netflix.ice.common.Poller | 28 | access$000 in '' | 88 | run . . . . . . . . in com.netflix.ice.common.Poller$1 ^ 745 | run in java.lang.Thread 2017-02-10 02:18:56,185 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - trying to download xxxxxxxxxxxx-aws-billing-detailed-line-items-2016-08.csv.zip... | Error 2017-02-10 02:18:56,251 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.BillingFileProcessor - Error polling Message: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: D0E19DD03B5374D0) Line | Method ->> 1077 | handleErrorResponse in com.amazonaws.http.AmazonHttpClient
| 725 | executeOneRequest in '' | 460 | executeHelper . . . . . . in '' | 295 | execute in '' | 3710 | invoke . . . . . . . . . . in com.amazonaws.services.s3.AmazonS3Client | 1010 | getObjectMetadata in '' | 988 | getObjectMetadata . . . . in '' | 275 | downloadFileIfChangedSince in com.netflix.ice.common.AwsUtils | 175 | poll . . . . . . . . . . . in com.netflix.ice.processor.BillingFileProcessor | 50 | doWork in com.netflix.ice.common.Poller | 28 | access$000 . . . . . . . . in '' | 88 | run in com.netflix.ice.common.Poller$1 ^ 745 | run . . . . . . . . . . . in java.lang.Thread
and here is my ice.properties
ice.processor=true
ice.reader=false
ice.reservationCapacityPoller=false
ice.reservationPeriod=threeyear
ice.reservationUtilization=MEDIUM
ice.highstockUrl=http://code.highcharts.com/stock/highstock.js
ice.ondemandCostAlertThreshold=250
ice.billing_s3bucketname=kshbill,
ice.billing_s3bucketprefix=,
ice.startmillis=1467339397000
ice.companyName=<My Company>
ice.work_s3bucketname=ice-result
ice.work_s3bucketprefix=ice/
ice.processor.localDir=/root/ice_processor
ice.reader.localDir=/root/ice_reader
ice.monthlycachesize=12
ice.account.account1=
Have you tried the very last release (v1.1)? It includes PR that solve authentication problem with eu-central-1 (and all the auth v4 only endpoints). Edit: don't forgot to set the billing bucket region with the new parameter.
I got the v1.1, set:
ice.billing_s3bucketregion=eu-west-1,eu-central-1
in ice.properties, but got this message:
ice_1 | 2017-06-14 20:20:21,598 [localhost-startStop-1] WARN s3.AmazonS3Client - Attempting to re-send the request to snakelab-ice-work.s3-external-1.amazonaws.com with AWS V4 authentication. To avoid this warning in the future, please use region-specific endpoint to access buckets located in regions that require V4 signing. ice_1 | | Error com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: E1182119EE427C9E), S3 Extended Request ID: FaVVvsnKdvIKRDYTALRWgbv+U8rSQsR/4VSlpaD8xZM6V0xb7onkVE70BWuvzlKYHoaW8K7TgQU=
What am i doing wrong!?
Here is my ice.properties:
ice.processor=true ice.billing_s3bucketregion=eu-central-1 ice.reader=true ice.reservationCapacityPoller=false ice.reservationPeriod=threeyear ice.reservationUtilization=MEDIUM ice.highstockUrl=https://code.highcharts.com/stock/4.2.1/highstock.js ice.ondemandCostAlertThreshold=250 ice.ondemandCostAlertEmails= ice.billing_s3bucketname=snakelab-ice ice.billing_s3bucketprefix=, ice.startmillis=1364774400000 ice.companyName=Hartmut ice.work_s3bucketname=snakelab-ice-work ice.work_s3bucketprefix=ice/ ice.processor.localDir=/mnt/ice_processor ice.reader.localDir=/mnt/ice_reader ice.monthlycachesize=12 ice.account.testing=xxxxxxxxxxxxxxx
Hi I'm having the same issue. I was really hoping someone can help me with this.
| Error 2017-09-05 05:59:02,742 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.BillingFileProcessor - Error reading from file lastProcessMillis_2015-08
Message: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 75F99787B3B99E9E)
Line | Method
->> 1588 | handleErrorResponse in com.amazonaws.http.AmazonHttpClient$RequestExecutor
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
| 1258 | executeOneRequest in ''
| 1030 | executeHelper . . . in ''
| 742 | doExecute in ''
| 716 | executeWithTimer . in ''
| 699 | execute in ''
| 667 | access$500 . . . . in ''
| 649 | execute in com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl
| 513 | execute . . . . . . in com.amazonaws.http.AmazonHttpClient
| 4221 | invoke in com.amazonaws.services.s3.AmazonS3Client
| 4168 | invoke . . . . . . in ''
| 1378 | getObject in ''
| 1255 | getObject . . . . . in ''
| 659 | getLastMillis in com.netflix.ice.processor.BillingFileProcessor
| 673 | lastProcessTime . . in ''
| 152 | poll in ''
| 50 | doWork . . . . . . in com.netflix.ice.common.Poller
| 28 | access$000 in ''
| 88 | run . . . . . . . . in com.netflix.ice.common.Poller$1
^ 745 | run in java.lang.Thread
2017-09-05 05:59:02,746 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - trying to download 931406743573-aws-billing-detailed-line-items-2015-08.csv.zip...
| Error 2017-09-05 05:59:02,779 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.BillingFileProcessor - Error polling
Message: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 4B72F3CE351F36E4)
What I did find is that in order to get details into bucket I had to go through the process on AWS Billing where I did the following steps.
- My Billing Dashboard >> Preferences >> Enter the bucket and (*GIVES A SAMPLE POLICY TO ATTACH TO THE BUCKET) >> Check all the options.
- Attached the sample policy to the bucket.
"Version": "2008-10-17",
"Id": "Policy1335892530063",
"Statement": [
{
"Sid": "Stmt1335892150622",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::386209384616:root"
},
"Action": [
"s3:GetBucketAcl",
"s3:GetBucketPolicy"
],
"Resource": "arn:aws:s3:::<BUCKET_NAME>"
},
{
"Sid": "Stmt1335892526596",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::386209384616:root"
},
"Action": [
"s3:PutObject"
],
"Resource": "arn:aws:s3:::<BUCKET_NAME>/*"
}
]
}
Note the iam account 386209384616 seems to be some AWS specific account that generates the billing. Note this is not my account number.
After following the above 2 steps I did the whole ice install.sh and as per the logs it says that it cannot download one of the require files. But when I tried to change the permissions in the bucket I noticed that since I am not the owner of the file I cannot change the file permissions. Note this is the FILE PERMISSION I cannot change because I think it is used by the AWS specific account 386209384616. Note this is not the bucker permissions.
I hope I'm not going down the wrong path with this. Any help would be much appreciated.
Regards DD
Hello,
To help identify the problem, can you try to download the 931406743573-aws-billing-detailed-line-items-2015-08.csv.zip file using the AWS CLI and the same credentials you are using with Ice ?
Thanks,
Nicolas
-- Nicolas Fonrose | Teevity | Founder +33.6.61.35.43.31 https://teevity.com - Cloud Costs Analytics built on NetflixOSS twitter - @nfonrose / @teevity
On Tue, Sep 5, 2017 at 8:12 AM, gdd1984 [email protected] wrote:
Hi I'm having the same issue. I was really hoping someone can help me with this.
| Error 2017-09-05 05:59:02,742 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.BillingFileProcessor - Error reading from file lastProcessMillis_2015-08 Message: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 75F99787B3B99E9E) Line | Method ->> 1588 | handleErrorResponse in com.amazonaws.http.AmazonHttpClient$RequestExecutor
| 1258 | executeOneRequest in '' | 1030 | executeHelper . . . in '' | 742 | doExecute in '' | 716 | executeWithTimer . in '' | 699 | execute in '' | 667 | access$500 . . . . in '' | 649 | execute in com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl | 513 | execute . . . . . . in com.amazonaws.http.AmazonHttpClient | 4221 | invoke in com.amazonaws.services.s3.AmazonS3Client | 4168 | invoke . . . . . . in '' | 1378 | getObject in '' | 1255 | getObject . . . . . in '' | 659 | getLastMillis in com.netflix.ice.processor.BillingFileProcessor | 673 | lastProcessTime . . in '' | 152 | poll in '' | 50 | doWork . . . . . . in com.netflix.ice.common.Poller | 28 | access$000 in '' | 88 | run . . . . . . . . in com.netflix.ice.common.Poller$1 ^ 745 | run in java.lang.Thread 2017-09-05 05:59:02,746 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - trying to download 931406743573-aws-billing-detailed-line-items-2015-08.csv.zip... | Error 2017-09-05 05:59:02,779 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.BillingFileProcessor - Error polling Message: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 4B72F3CE351F36E4)
What I did find is that in order to get details into bucket I had to go through the process on AWS Billing where I did the following steps.
- My Billing Dashboard >> Preferences >> Enter the bucket and (*GIVES A SAMPLE POLICY TO ATTACH TO THE BUCKET) >> Check all the options.
- Attached the sample policy to the bucket.
"Version": "2008-10-17", "Id": "Policy1335892530063", "Statement": [ { "Sid": "Stmt1335892150622", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::386209384616:root" }, "Action": [ "s3:GetBucketAcl", "s3:GetBucketPolicy" ], "Resource": "arn:aws:s3:::<BUCKET_NAME>" }, { "Sid": "Stmt1335892526596", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::386209384616:root" }, "Action": [ "s3:PutObject" ], "Resource": "arn:aws:s3:::<BUCKET_NAME>/*" } ] }```
Note the iam account 386209384616 seems to be some AWS specific account that generates the billing. Note this is not my account number.
After following the above 2 steps I did the whole ice install.sh and as per the logs it says that it cannot download one of the require files. But when I tried to change the permissions in the bucket I noticed that since I am not the owner of the file I cannot change the file permissions. Note this is the FILE PERMISSION I cannot change because I think it is used by the AWS specific account 386209384616. Note this is not the bucker permissions.
I hope I'm not going down the wrong path with this. Any help would be much appreciated.
Regards DD
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Teevity/ice/issues/238#issuecomment-327080669, or mute the thread https://github.com/notifications/unsubscribe-auth/AACUpXQsVGIPVDCkpa_mMHWrqj-gBJfUks5sfOZYgaJpZM4L87Rm .
Hi Nicolas,
No I cannot.
aws s3api get-object --bucket <BUCKET-NAME> --key <ACCOUNT NUMBER>-aws-billing-detailed-line-items-2015-08.csv.zip test.txt
An error occurred (AccessDenied) when calling the GetObject operation: Access Denied
I would agree that this is not really an application issue but a S3 bucket object issue. I am trying to figure out how other people get this working. Because I have tried this on 3 seperate account now and when I enable "Receive Billing Report" option via the AWS billing dashboard it asks me to enter in a sample policy which allows access to an account 386209384616. This seems to be some special AWS billing account which when uploading the objects(files) retains ownership of the object. So there after even my root AWS user cannot grant permissions on the object. Is there some trick to assigning permissions to the user who's credentials I'm using to access the bucket. Or do I have to edit the bucker policy itself. Note the user can list all the objects in the bucket. Just can't download.
Thanks again in advance to Nicolas or anyone who can help me with this.
Cheers DD
Hi,
You need to create an IAM user (or IAM role) with the permissions described by the policy below.
You then need to ensure you use this IAM user (or IAM role) when you execute your AWS CLI commands and when you configure Ice. I let you check the AWS docs regarding how to define the credentials used by the CLI.
{ "Version": "2012-10-17", "Statement": [ { "Action": [ "s3:Get*", "s3:List*" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::<BUCKET-NAME>", "arn:aws:s3:::<BUCKET-NAME>/*" ] } ] }
Cheers,
Nicolas
-- Nicolas Fonrose | Teevity | Founder +33.6.61.35.43.31 https://teevity.com - Cloud Costs Analytics built on NetflixOSS twitter - @nfonrose / @teevity
On Wed, Sep 6, 2017 at 7:09 AM, gdd1984 [email protected] wrote:
Hi Nicolas,
No I cannot.
aws s3api get-object --bucket <BUCKET-NAME> --key 931406743573-aws-billing-detailed-line-items-2015-08.csv.zip test.txt
An error occurred (AccessDenied) when calling the GetObject operation: Access Denied
I would agree that this is not really an application issue but a S3 bucket object issue. I am trying to figure out how other people get this working. Because I have tried this on 3 seperate account now and when I enable "Receive Billing Report" option via the AWS billing dashboard it asks me to enter in a sample policy which allows access to an account 386209384616. This seems to be some special AWS billing account which when uploading the objects(files) retains ownership of the object. So there after even my root AWS user cannot grant permissions on the object. Is there some trick to assigning permissions to the user who's credentials I'm using to access the bucket. Or do I have to edit the bucker policy itself. Note the user can list all the objects in the bucket. Just can't download.
Thanks again in advance to Nicolas or anyone who can help me with this.
Cheers DD
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Teevity/ice/issues/238#issuecomment-327376288, or mute the thread https://github.com/notifications/unsubscribe-auth/AACUpW3ZX6w8TqLqjoK1jmHj5Vx7Os0tks5sfikkgaJpZM4L87Rm .
Hi Nicolas,
The user already has most of these permissions.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1421551747000",
"Effect": "Allow",
"Action": [
"ec2:DescribeReservedInstances",
"ec2:DescribeReservedInstancesOfferings"
],
"Resource": [
"*"
]
},
{
"Sid": "Stmt1418665415000",
"Effect": "Allow",
"Action": [
"s3:DeleteObject",
"s3:GetBucketLocation",
"s3:GetObject",
"s3:ListAllMyBuckets",
"s3:ListBucket",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::<WORK_BUCKET_NAME>/*"
]
},
{
"Sid": "Stmt1418665415001",
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::<WORK_BUCKET_NAME>"
]
},
{
"Sid": "Stmt1418665415000a",
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:GetObject",
"s3:ListAllMyBuckets",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::<BILLING_BUCKET_NAME>/*"
]
},
{
"Sid": "Stmt1418665415001a",
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::<BILLING_BUCKET_NAME>"
]
}
]
}
I believe even though the user policy is ok the permissions of the objects in the s3 bucket are not correct. But because the AWS special account is the owner I can't even change it. The following is link to a screenshot of the object permission https://s3-ap-southeast-2.amazonaws.com/tempdd1/Untitled.png
Any other ideas?
Regards DD
Nope
-- Nicolas Fonrose | Teevity | Founder +33.6.61.35.43.31 https://teevity.com - Cloud Costs Analytics built on NetflixOSS twitter - @nfonrose / @teevity
On Wed, Sep 6, 2017 at 2:31 PM, gdd1984 [email protected] wrote:
Hi Nicolas,
The user already has most of these permissions.
{ "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt1421551747000", "Effect": "Allow", "Action": [ "ec2:DescribeReservedInstances", "ec2:DescribeReservedInstancesOfferings" ], "Resource": [ "" ] }, { "Sid": "Stmt1418665415000", "Effect": "Allow", "Action": [ "s3:DeleteObject", "s3:GetBucketLocation", "s3:GetObject", "s3:ListAllMyBuckets", "s3:ListBucket", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::<WORK_BUCKET_NAME>/" ] }, { "Sid": "Stmt1418665415001", "Effect": "Allow", "Action": [ "s3:GetBucketLocation", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::<WORK_BUCKET_NAME>" ] }, { "Sid": "Stmt1418665415000a", "Effect": "Allow", "Action": [ "s3:GetBucketLocation", "s3:GetObject", "s3:ListAllMyBuckets", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::<BILLING_BUCKET_NAME>/*" ] }, { "Sid": "Stmt1418665415001a", "Effect": "Allow", "Action": [ "s3:GetBucketLocation", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::<BILLING_BUCKET_NAME>" ] } ] }
Any other ideas?
Regards DD
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Teevity/ice/issues/238#issuecomment-327468339, or mute the thread https://github.com/notifications/unsubscribe-auth/AACUpfitJnXkO3oq7lHOmE1BnCU2oW8Wks5sfpCHgaJpZM4L87Rm .
Dear all
Please. Help me to Automate the files from EC2 to S3 storage for this what I have to do.
We are sheduling the auto backups for every day, from EC2 to S3 storage.
On Sep 6, 2017 10:40 AM, "gdd1984" [email protected] wrote:
Hi Nicolas,
No I cannot.
aws s3api get-object --bucket <BUCKET-NAME> --key 931406743573-aws-billing-detailed-line-items-2015-08.csv.zip test.txt
An error occurred (AccessDenied) when calling the GetObject operation: Access Denied
I would agree that this is not really an application issue but a S3 bucket object issue. I am trying to figure out how other people get this working. Because I have tried this on 3 seperate account now and when I enable "Receive Billing Report" option via the AWS billing dashboard it asks me to enter in a sample policy which allows access to an account 386209384616. This seems to be some special AWS billing account which when uploading the objects(files) retains ownership of the object. So there after even my root AWS user cannot grant permissions on the object. Is there some trick to assigning permissions to the user who's credentials I'm using to access the bucket. Or do I have to edit the bucker policy itself. Note the user can list all the objects in the bucket. Just can't download.
Thanks again in advance to Nicolas or anyone who can help me with this.
Cheers DD
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Teevity/ice/issues/238#issuecomment-327376288, or mute the thread https://github.com/notifications/unsubscribe-auth/AXJeal8L_tqVAiLY_fPj0a3G5-VdS_8sks5sfiksgaJpZM4L87Rm .
