aws-cost-report icon indicating copy to clipboard operation
aws-cost-report copied to clipboard

Error while running the report

Open sathyajackson opened this issue 6 years ago • 20 comments

Hello,

Sorry for very late responds, I have hooked up some other project. Alright, here we go, I used the classic version of the run.py and I end up with below error

[ error ] Traceback (most recent call last): File "src/make_xlsx.py", line 630, in main(name) File "src/make_xlsx.py", line 612, in main gen_weekly_variations(workbook, header_format, val_format) File "src/make_xlsx.py", line 201, in gen_weekly_variations with open(IN_ABSOLUTE_COST_PER_MONTH) as f: FileNotFoundError: [Errno 2] No such file or directory: 'out/absolute.csv' Any suggestions would be of great help.

sathyajackson avatar Jan 07 '19 16:01 sathyajackson

any update would be of great help

sathyajackson avatar Jan 08 '19 11:01 sathyajackson

Hello,

Can I have more logs and the exact command you typed?

Thanks.

thibautcornolti avatar Jan 08 '19 16:01 thibautcornolti

here you go.

Command:

./run.py --billing default billing-aws-cost-report billing/XXX.csv --ec2 default --xlsx-name billing.

Here is the complete log.

 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  582M  100  582M    0     0  41.3M      0  0:00:14  0:00:14 --:--:-- 43.2M
Download billings for default...
Fetching ec2 data for all accounts in ap-south-1...
Fetching ec2 data for all accounts in eu-west-3...
Fetching ec2 data for all accounts in eu-north-1...
Fetching ec2 data for all accounts in eu-west-2...
Fetching ec2 data for all accounts in eu-west-1...
Fetching ec2 data for all accounts in ap-northeast-2...
Fetching ec2 data for all accounts in ap-northeast-1...
Fetching ec2 data for all accounts in sa-east-1...
Fetching ec2 data for all accounts in ca-central-1...
Fetching ec2 data for all accounts in ap-southeast-1...
Fetching ec2 data for all accounts in ap-southeast-2...
Fetching ec2 data for all accounts in eu-central-1...
Fetching ec2 data for all accounts in us-east-1...
Fetching ec2 data for all accounts in us-east-2...
Fetching ec2 data for all accounts in us-west-1...
Fetching ec2 data for all accounts in us-west-2...
[default - ap-northeast-1] Getting reserved instances...
[default - ap-northeast-2] Getting reserved instances...
[default - ap-south-1] Getting reserved instances...
[default - eu-north-1] Getting reserved instances...
[default - eu-west-1] Getting reserved instances...
[default - ap-southeast-1] Getting reserved instances...
[default - eu-west-2] Getting reserved instances...
[default - eu-west-3] Getting reserved instances...
[default - ca-central-1] Getting reserved instances...
[default - us-east-1] Getting reserved instances...
[default - sa-east-1] Getting reserved instances...
[default - us-west-1] Getting reserved instances...
[default - us-east-2] Getting reserved instances...
[default - us-west-2] Getting reserved instances...
[default - eu-central-1] Getting reserved instances...
[default - ap-southeast-2] Getting reserved instances...
[default - us-east-1] Getting on-demand instances...
[default - ca-central-1] Getting on-demand instances...
[default - us-east-2] Getting on-demand instances...
[global - ca-central-1] Getting offerings for all instances...
[global - us-east-2] Getting offerings for all instances...
[default - eu-west-1] Getting on-demand instances...
[global - ca-central-1] Matching on-demand instances with reserved instances...
[global - ca-central-1] Done!
[default - eu-west-2] Getting on-demand instances...
[global - us-east-2] Matching on-demand instances with reserved instances...
[global - us-east-2] Done!
[default - eu-west-3] Getting on-demand instances...
[global - eu-west-1] Getting offerings for all instances...
[default - us-west-1] Getting on-demand instances...
[default - eu-north-1] Getting on-demand instances...
[global - eu-west-2] Getting offerings for all instances...
[default - us-west-2] Getting on-demand instances...
[default - eu-central-1] Getting on-demand instances...
[global - eu-west-1] Matching on-demand instances with reserved instances...
[global - eu-west-1] Done!
[global - eu-west-2] Matching on-demand instances with reserved instances...
[global - eu-west-2] Done!
[default - ap-northeast-1] Getting on-demand instances...
[global - eu-west-3] Getting offerings for all instances...
[global - us-west-1] Getting offerings for all instances...
[global - us-west-2] Getting offerings for all instances...
[global - eu-north-1] Getting offerings for all instances...
[global - eu-west-3] Matching on-demand instances with reserved instances...
[global - eu-west-3] Done!
[global - eu-central-1] Getting offerings for all instances...
[global - us-east-1] Getting offerings for all instances...
[default - us-east-1] Getting offerings for instance 1/8...
[global - us-west-1] Matching on-demand instances with reserved instances...
[global - us-west-1] Done!
[default - sa-east-1] Getting on-demand instances...
[default - ap-south-1] Getting on-demand instances...
[global - us-west-2] Matching on-demand instances with reserved instances...
[global - us-west-2] Done!
[global - eu-north-1] Matching on-demand instances with reserved instances...
[global - eu-north-1] Done!
[global - eu-central-1] Matching on-demand instances with reserved instances...
[global - eu-central-1] Done!
[default - ap-northeast-2] Getting on-demand instances...
[global - ap-northeast-1] Getting offerings for all instances...
[global - ap-northeast-1] Matching on-demand instances with reserved instances...
[global - ap-northeast-1] Done!
[default - ap-southeast-1] Getting on-demand instances...
[default - ap-southeast-2] Getting on-demand instances...
[global - sa-east-1] Getting offerings for all instances...
[global - ap-south-1] Getting offerings for all instances...
[global - sa-east-1] Matching on-demand instances with reserved instances...
[global - sa-east-1] Done!
[global - ap-northeast-2] Getting offerings for all instances...
[global - ap-south-1] Matching on-demand instances with reserved instances...
[global - ap-south-1] Done!
[global - ap-northeast-2] Matching on-demand instances with reserved instances...
[global - ap-northeast-2] Done!
[global - ap-southeast-2] Getting offerings for all instances...
[global - ap-southeast-1] Getting offerings for all instances...
[global - ap-southeast-2] Matching on-demand instances with reserved instances...
[global - ap-southeast-2] Done!
[global - ap-southeast-1] Matching on-demand instances with reserved instances...
[global - ap-southeast-1] Done!
Fetched ec2 data for all accounts in ap-south-1
Fetched ec2 data for all accounts in eu-west-3
Fetched ec2 data for all accounts in eu-north-1
Fetched ec2 data for all accounts in eu-west-2
Fetched ec2 data for all accounts in eu-west-1
Fetched ec2 data for all accounts in ap-northeast-2
Fetched ec2 data for all accounts in ap-northeast-1
Fetched ec2 data for all accounts in sa-east-1
Fetched ec2 data for all accounts in ca-central-1
Fetched ec2 data for all accounts in ap-southeast-1
Fetched ec2 data for all accounts in ap-southeast-2
Fetched ec2 data for all accounts in eu-central-1
[default - us-east-1] Getting offerings for instance 2/8...
[default - us-east-1] Getting offerings for instance 3/8...
[default - us-east-1] Getting offerings for instance 4/8...
[default - us-east-1] Getting offerings for instance 5/8...
[default - us-east-1] Getting offerings for instance 6/8...
[default - us-east-1] Getting offerings for instance 7/8...
[default - us-east-1] Getting offerings for instance 8/8...
[global - us-east-1] Matching on-demand instances with reserved instances...
[global - us-east-1] Done!
Fetched ec2 data for all accounts in us-east-1
Fetched ec2 data for all accounts in us-east-2
Fetched ec2 data for all accounts in us-west-1
Fetched ec2 data for all accounts in us-west-2
Fetching ec2 metadata for default in ap-south-1...
Fetching ec2 metadata for default in eu-west-3...
Fetching ec2 metadata for default in eu-north-1...
Fetching ec2 metadata for default in eu-west-2...
Fetching ec2 metadata for default in eu-west-1...
Fetching ec2 metadata for default in ap-northeast-2...
Fetching ec2 metadata for default in ap-northeast-1...
Fetching ec2 metadata for default in sa-east-1...
Fetching ec2 metadata for default in ca-central-1...
Fetching ec2 metadata for default in ap-southeast-1...
Fetching ec2 metadata for default in ap-southeast-2...
Fetching ec2 metadata for default in eu-central-1...
Fetching ec2 metadata for default in us-east-1...
Fetching ec2 metadata for default in us-east-2...
Fetching ec2 metadata for default in us-west-1...
Fetching ec2 metadata for default in us-west-2...
[574925386644 - ap-southeast-1] Getting instances metadata...
[574925386644 - sa-east-1] Getting instances metadata...
[574925386644 - eu-west-2] Getting instances metadata...
[574925386644 - us-east-1] Getting instances metadata...
[574925386644 - eu-west-1] Getting instances metadata...
[574925386644 - ap-southeast-2] Getting instances metadata...
[574925386644 - us-east-2] Getting instances metadata...
[574925386644 - ap-south-1] Getting instances metadata...
[574925386644 - eu-central-1] Getting instances metadata...
[574925386644 - eu-west-3] Getting instances metadata...
[574925386644 - ap-northeast-1] Getting instances metadata...
[574925386644 - us-west-1] Getting instances metadata...
[574925386644 - us-east-2] Done!
[574925386644 - eu-north-1] Getting instances metadata...
[574925386644 - ca-central-1] Getting instances metadata...
[574925386644 - us-west-2] Getting instances metadata...
[574925386644 - us-east-1] Done!
[574925386644 - ca-central-1] Done!
[574925386644 - eu-west-1] Done!
[574925386644 - eu-west-2] Done!
[574925386644 - ap-northeast-2] Getting instances metadata...
[574925386644 - us-west-1] Done!
[574925386644 - eu-central-1] Done!
[574925386644 - eu-west-3] Done!
[574925386644 - us-west-2] Done!
[574925386644 - sa-east-1] Done!
[574925386644 - eu-north-1] Done!
[574925386644 - ap-northeast-1] Done!
[574925386644 - ap-southeast-2] Done!
[574925386644 - ap-southeast-1] Done!
[574925386644 - ap-northeast-2] Done!
[574925386644 - ap-south-1] Done!
Fetched ec2 metadata for default in ap-south-1
Fetched ec2 metadata for default in eu-west-3
Fetched ec2 metadata for default in eu-north-1
Fetched ec2 metadata for default in eu-west-2
Fetched ec2 metadata for default in eu-west-1
Fetched ec2 metadata for default in ap-northeast-2
Fetched ec2 metadata for default in ap-northeast-1
Fetched ec2 metadata for default in sa-east-1
Fetched ec2 metadata for default in ca-central-1
Fetched ec2 metadata for default in ap-southeast-1
Fetched ec2 metadata for default in ap-southeast-2
Fetched ec2 metadata for default in eu-central-1
Fetched ec2 metadata for default in us-east-1
Fetched ec2 metadata for default in us-east-2
Fetched ec2 metadata for default in us-west-1
Fetched ec2 metadata for default in us-west-2
Processing billing data (1/5 - billing diff)...
Traceback (most recent call last):
  File "src/get_bill_diff.py", line 65, in <module>
    first_month = preserved_months[0]
IndexError: list index out of range
Processing billing data (2/5 - instance history)...
Processing billing data (3/5 - ec2 last month)...
Processing billing data (4/5 - ebs last month)...
Processing billing data (5/5 - s3 cost)...
Generating xlsx file...
Traceback (most recent call last):
  File "src/make_xlsx.py", line 630, in <module>
    main(name)
  File "src/make_xlsx.py", line 612, in main
    gen_weekly_variations(workbook, header_format, val_format)
  File "src/make_xlsx.py", line 201, in gen_weekly_variations
    with open(IN_ABSOLUTE_COST_PER_MONTH) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'out/absolute.csv'

sathyajackson avatar Jan 08 '19 16:01 sathyajackson

The bill file I am using is more than 30 G and more than 3 million records.

sathyajackson avatar Jan 08 '19 16:01 sathyajackson

I guess your bucket is billing-aws-cost-report, is it a bucket with a report enabled? You can check that here https://console.aws.amazon.com/billing/home?#/reports

You should see something as below: image

thibautcornolti avatar Jan 08 '19 16:01 thibautcornolti

Well no, what i did is i created a bucket in one of testing account and uploaded the bill file in to the bucket.

I am doing a POC to showcase on what type of reports we can get using this tool.

All our accounts are merged with parent account for billing and i don't have access to that acciunt, i can down load the bill data and can upload the same to another s3 bucket for testing.

On Tue, 8 Jan 2019, 10:24 pm Thibaut Cornolti <[email protected] wrote:

I guess your bucket is billing-aws-cost-report, is it a bucket with a report enabled? You can check that here https://console.aws.amazon.com/billing/home?#/reports

[image: image] https://user-images.githubusercontent.com/4880378/50846000-71516200-136e-11e9-8a23-8a07580394ca.png

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/trackit/aws-cost-report/issues/37#issuecomment-452371537, or mute the thread https://github.com/notifications/unsubscribe-auth/Afb7tCWL0EHGBlF3N37jZaW_t-pgSVTYks5vBM1NgaJpZM4ZzyBD .

sathyajackson avatar Jan 08 '19 17:01 sathyajackson

Got it. So you should have a report enabled on your master account. A report is more than a single .csv, you have to download the entire bucket (sync it with the cli for example). Are you allowed to do that?

To clarify everything, you can get two types of report with AWS :

  • Old report (https://console.aws.amazon.com/billing/home?#/preferences) image

  • New report (https://console.aws.amazon.com/billing/home?#/reports) image

Old report will be removed by AWS. Our tool only works with the new type of report. You have to be sure you're using it.

thibautcornolti avatar Jan 08 '19 17:01 thibautcornolti

I understand about the old report and new report on AWS.

Just to clarify, can't i just run this script against one whole months bill and get the required informations?

Let's say i have a full details billing available for December in the zip formate (Just one file)can't i use that as a source file on any of my s3 bucket and run the script against it to get all my finds for the month of decemeber? ( including the sizing recommendation )

On Tue, 8 Jan 2019, 11:02 pm Thibaut Cornolti <[email protected] wrote:

Get it. So you should have a report enabled on your master account. A report is more than a single .csv, you have to download the entire bucket (sync it with the cli for example). Are you allowed to do that?

To clarify everything, you can get two types of report with AWS :

Old report (https://console.aws.amazon.com/billing/home?#/preferences) [image: image] https://user-images.githubusercontent.com/4880378/50848072-8250a200-1373-11e9-85bb-eb9cbaca4c9c.png

New report (https://console.aws.amazon.com/billing/home?#/reports) [image: image] https://user-images.githubusercontent.com/4880378/50848090-8f6d9100-1373-11e9-8d4d-a48273b3bc04.png

Old report will be removed by AWS. Our tool only works with the new type of report. You have to be sure you're using it.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/trackit/aws-cost-report/issues/37#issuecomment-452384350, or mute the thread https://github.com/notifications/unsubscribe-auth/Afb7tAeYsMgHTnAJ1hU_siJyu2U_Qlqxks5vBNYTgaJpZM4ZzyBD .

sathyajackson avatar Jan 08 '19 18:01 sathyajackson

Sorry you can't, your file is probably in the old format (typically one .csv.zip generated per month), we do not support that report. However, you can use our tool without billing, just remove the --billing argument. You just won't see any cost.

thibautcornolti avatar Jan 08 '19 18:01 thibautcornolti

Ok will try and let you know...

On Wed, 9 Jan 2019, 12:18 am Thibaut Cornolti <[email protected] wrote:

Sorry you can't, your file is probably in the old format (typically one .csv.zip generated per month), we do not support that report. However, you can use our tool without billing, just remove the --billing argument. You'll just see no cost.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/trackit/aws-cost-report/issues/37#issuecomment-452408884, or mute the thread https://github.com/notifications/unsubscribe-auth/Afb7tDKfj6SU_wLnimgUv-vjqdl1Mp2Eks5vBOfogaJpZM4ZzyBD .

sathyajackson avatar Jan 08 '19 19:01 sathyajackson

Ok! I look forward to your answer.

thibautcornolti avatar Jan 08 '19 20:01 thibautcornolti

I used the following command to execute without --billing and end up with the same error message.

 ./run.py   --ec2 default --xlsx-name bill

Logs

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  585M  100  585M    0     0  39.9M      0  0:00:14  0:00:14 --:--:-- 41.3M
Fetching ec2 data for all accounts in ap-south-1...
Fetching ec2 data for all accounts in eu-west-3...
Fetching ec2 data for all accounts in eu-north-1...
Fetching ec2 data for all accounts in eu-west-2...
Fetching ec2 data for all accounts in eu-west-1...
Fetching ec2 data for all accounts in ap-northeast-2...
Fetching ec2 data for all accounts in ap-northeast-1...
Fetching ec2 data for all accounts in sa-east-1...
Fetching ec2 data for all accounts in ca-central-1...
Fetching ec2 data for all accounts in ap-southeast-1...
Fetching ec2 data for all accounts in ap-southeast-2...
Fetching ec2 data for all accounts in eu-central-1...
Fetching ec2 data for all accounts in us-east-1...
Fetching ec2 data for all accounts in us-east-2...
Fetching ec2 data for all accounts in us-west-1...
Fetching ec2 data for all accounts in us-west-2...
[default - eu-north-1] Getting reserved instances...
[default - eu-west-3] Getting reserved instances...
[default - eu-west-1] Getting reserved instances...
[default - ap-south-1] Getting reserved instances...
[default - eu-west-2] Getting reserved instances...
[default - ap-northeast-1] Getting reserved instances...
[default - ap-northeast-2] Getting reserved instances...
[default - eu-central-1] Getting reserved instances...
[default - ap-southeast-1] Getting reserved instances...
[default - ap-southeast-2] Getting reserved instances...
[default - us-west-2] Getting reserved instances...
[default - us-east-2] Getting reserved instances...
[default - sa-east-1] Getting reserved instances...
[default - ca-central-1] Getting reserved instances...
[default - us-east-1] Getting reserved instances...
[default - us-west-1] Getting reserved instances...
[default - us-east-2] Getting on-demand instances...
[default - ca-central-1] Getting on-demand instances...
[default - us-east-1] Getting on-demand instances...
[global - us-east-2] Getting offerings for all instances...
[default - eu-west-3] Getting on-demand instances...
[global - us-east-2] Matching on-demand instances with reserved instances...
[global - us-east-2] Done!
[default - eu-west-2] Getting on-demand instances...
[global - ca-central-1] Getting offerings for all instances...
[global - ca-central-1] Matching on-demand instances with reserved instances...
[global - ca-central-1] Done!
[default - eu-west-1] Getting on-demand instances...
[default - us-west-2] Getting on-demand instances...
[default - eu-central-1] Getting on-demand instances...
[default - eu-north-1] Getting on-demand instances...
[default - us-west-1] Getting on-demand instances...
[global - eu-west-3] Getting offerings for all instances...
[global - eu-west-1] Getting offerings for all instances...
[default - ap-northeast-1] Getting on-demand instances...
[global - eu-west-3] Matching on-demand instances with reserved instances...
[global - eu-west-3] Done!
[global - eu-west-2] Getting offerings for all instances...
[global - eu-west-1] Matching on-demand instances with reserved instances...
[global - eu-west-1] Done!
[global - us-west-2] Getting offerings for all instances...
[default - ap-south-1] Getting on-demand instances...
[global - eu-north-1] Getting offerings for all instances...
[default - sa-east-1] Getting on-demand instances...
[global - eu-central-1] Getting offerings for all instances...
[global - eu-west-2] Matching on-demand instances with reserved instances...
[global - eu-west-2] Done!
[global - us-west-2] Matching on-demand instances with reserved instances...
[global - us-west-2] Done!
[default - ap-northeast-2] Getting on-demand instances...
[default - ap-southeast-2] Getting on-demand instances...
[global - eu-central-1] Matching on-demand instances with reserved instances...
[global - eu-central-1] Done!
[global - us-east-1] Getting offerings for all instances...
[default - us-east-1] Getting offerings for instance 1/8...
[global - us-west-1] Getting offerings for all instances...
[global - eu-north-1] Matching on-demand instances with reserved instances...
[global - eu-north-1] Done!
[global - sa-east-1] Getting offerings for all instances...
[global - us-west-1] Matching on-demand instances with reserved instances...
[global - us-west-1] Done!
[default - ap-southeast-1] Getting on-demand instances...
[global - ap-northeast-1] Getting offerings for all instances...
[global - ap-northeast-1] Matching on-demand instances with reserved instances...
[global - ap-northeast-1] Done!
[global - sa-east-1] Matching on-demand instances with reserved instances...
[global - sa-east-1] Done!
[global - ap-south-1] Getting offerings for all instances...
[global - ap-south-1] Matching on-demand instances with reserved instances...
[global - ap-south-1] Done!
[global - ap-northeast-2] Getting offerings for all instances...
[global - ap-southeast-2] Getting offerings for all instances...
[global - ap-northeast-2] Matching on-demand instances with reserved instances...
[global - ap-northeast-2] Done!
[global - ap-southeast-2] Matching on-demand instances with reserved instances...
[global - ap-southeast-2] Done!
[global - ap-southeast-1] Getting offerings for all instances...
[global - ap-southeast-1] Matching on-demand instances with reserved instances...
[global - ap-southeast-1] Done!
Fetched ec2 data for all accounts in ap-south-1
Fetched ec2 data for all accounts in eu-west-3
Fetched ec2 data for all accounts in eu-north-1
Fetched ec2 data for all accounts in eu-west-2
Fetched ec2 data for all accounts in eu-west-1
Fetched ec2 data for all accounts in ap-northeast-2
Fetched ec2 data for all accounts in ap-northeast-1
Fetched ec2 data for all accounts in sa-east-1
Fetched ec2 data for all accounts in ca-central-1
Fetched ec2 data for all accounts in ap-southeast-1
Fetched ec2 data for all accounts in ap-southeast-2
Fetched ec2 data for all accounts in eu-central-1
[default - us-east-1] Getting offerings for instance 2/8...
[default - us-east-1] Getting offerings for instance 3/8...
[default - us-east-1] Getting offerings for instance 4/8...
[default - us-east-1] Getting offerings for instance 5/8...
[default - us-east-1] Getting offerings for instance 6/8...
[default - us-east-1] Getting offerings for instance 7/8...
[default - us-east-1] Getting offerings for instance 8/8...
[global - us-east-1] Matching on-demand instances with reserved instances...
[global - us-east-1] Done!
Fetched ec2 data for all accounts in us-east-1
Fetched ec2 data for all accounts in us-east-2
Fetched ec2 data for all accounts in us-west-1
Fetched ec2 data for all accounts in us-west-2
Fetching ec2 metadata for default in ap-south-1...
Fetching ec2 metadata for default in eu-west-3...
Fetching ec2 metadata for default in eu-north-1...
Fetching ec2 metadata for default in eu-west-2...
Fetching ec2 metadata for default in eu-west-1...
Fetching ec2 metadata for default in ap-northeast-2...
Fetching ec2 metadata for default in ap-northeast-1...
Fetching ec2 metadata for default in sa-east-1...
Fetching ec2 metadata for default in ca-central-1...
Fetching ec2 metadata for default in ap-southeast-1...
Fetching ec2 metadata for default in ap-southeast-2...
Fetching ec2 metadata for default in eu-central-1...
Fetching ec2 metadata for default in us-east-1...
Fetching ec2 metadata for default in us-east-2...
Fetching ec2 metadata for default in us-west-1...
Fetching ec2 metadata for default in us-west-2...
[574925386644 - ap-south-1] Getting instances metadata...
[574925386644 - eu-west-1] Getting instances metadata...
[574925386644 - eu-north-1] Getting instances metadata...
[574925386644 - eu-west-3] Getting instances metadata...
[574925386644 - eu-west-2] Getting instances metadata...
[574925386644 - ap-northeast-1] Getting instances metadata...
[574925386644 - ca-central-1] Getting instances metadata...
[574925386644 - sa-east-1] Getting instances metadata...
[574925386644 - ap-southeast-1] Getting instances metadata...
[574925386644 - eu-central-1] Getting instances metadata...
[574925386644 - ap-southeast-2] Getting instances metadata...
[574925386644 - us-west-1] Getting instances metadata...
[574925386644 - us-east-1] Getting instances metadata...
[574925386644 - eu-west-1] Done!
[574925386644 - us-east-2] Getting instances metadata...
[574925386644 - us-west-2] Getting instances metadata...
[574925386644 - ca-central-1] Done!
[574925386644 - us-east-2] Done!
[574925386644 - eu-west-3] Done!
[574925386644 - eu-west-2] Done!
[574925386644 - eu-north-1] Done!
[574925386644 - us-west-1] Done!
[574925386644 - us-east-1] Done!
[574925386644 - eu-central-1] Done!
[574925386644 - ap-south-1] Done!
[574925386644 - us-west-2] Done!
[574925386644 - ap-northeast-2] Getting instances metadata...
Fetched ec2 metadata for default in ap-south-1
Fetched ec2 metadata for default in eu-west-3
Fetched ec2 metadata for default in eu-north-1
Fetched ec2 metadata for default in eu-west-2
Fetched ec2 metadata for default in eu-west-1
[574925386644 - ap-northeast-1] Done!
[574925386644 - sa-east-1] Done!
[574925386644 - ap-southeast-2] Done!
[574925386644 - ap-southeast-1] Done!
[574925386644 - ap-northeast-2] Done!
Fetched ec2 metadata for default in ap-northeast-2
Fetched ec2 metadata for default in ap-northeast-1
Fetched ec2 metadata for default in sa-east-1
Fetched ec2 metadata for default in ca-central-1
Fetched ec2 metadata for default in ap-southeast-1
Fetched ec2 metadata for default in ap-southeast-2
Fetched ec2 metadata for default in eu-central-1
Fetched ec2 metadata for default in us-east-1
Fetched ec2 metadata for default in us-east-2
Fetched ec2 metadata for default in us-west-1
Fetched ec2 metadata for default in us-west-2
Processing billing data (1/5 - billing diff)...
Traceback (most recent call last):
  File "src/get_bill_diff.py", line 65, in <module>
    first_month = preserved_months[0]
IndexError: list index out of range
Processing billing data (2/5 - instance history)...
Processing billing data (3/5 - ec2 last month)...
Processing billing data (4/5 - ebs last month)...
Processing billing data (5/5 - s3 cost)...
Generating xlsx file...
Traceback (most recent call last):
  File "src/make_xlsx.py", line 630, in <module>
    main(name)
  File "src/make_xlsx.py", line 612, in main
    gen_weekly_variations(workbook, header_format, val_format)
  File "src/make_xlsx.py", line 201, in gen_weekly_variations
    with open(IN_ABSOLUTE_COST_PER_MONTH) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'out/absolute.csv'

Correct me if i am worng in using the .run.py command.

sathyajackson avatar Jan 09 '19 11:01 sathyajackson

I just talked with the team: Finally, we can't run the tool without billing, really sorry for that.

I can help you to setup a report bucket though.

thibautcornolti avatar Jan 09 '19 14:01 thibautcornolti

Hi Thanks for your response, one more quick question, the tool actually use the CPU as the base point for the recommendation, I would like to know if we can have both memory and CPU?

I know that cloud watch will target only of the CPU and not memory ( other than writing custom metrics), but yeh, just thought of check if we can have this included on the script to somehow get the memory information of the machine and based on that providing the recommendation.

sathyajackson avatar Jan 09 '19 15:01 sathyajackson

No you can't use memory to influence recommendations, as you said, it's not supported by CloudWatch and you would need to install a custom software on every single instance.

thibautcornolti avatar Jan 09 '19 15:01 thibautcornolti

alright thanks 👍

sathyajackson avatar Jan 09 '19 15:01 sathyajackson

Thanks to you! :+1:

You can close this issue as soon as you have no more question.

thibautcornolti avatar Jan 09 '19 16:01 thibautcornolti

I get similar error when running the script. Can you please tell me what to enter "--billing profile_name". When I ran the script with --billing default i get the same error message.

Generating xlsx file... Traceback (most recent call last): File "src/make_xlsx.py", line 630, in main(name) File "src/make_xlsx.py", line 612, in main gen_weekly_variations(workbook, header_format, val_format) File "src/make_xlsx.py", line 201, in gen_weekly_variations with open(IN_ABSOLUTE_COST_PER_MONTH) as f: FileNotFoundError: [Errno 2] No such file or directory: 'out/absolute.csv'

vmpowercli avatar Feb 27 '19 17:02 vmpowercli

Hello,

$> ./run.py --billing profile_name billing-bucket-name prefix
  • profile_name is default or another profile available in ~/.aws/credentials.
  • billing_bucket_name is you billing bucket. It has to be configured here as following: image
  • prefix is prefix put in report config, you can get it by clicking on a report here: image Prefix here would be bills, billing being the report name.

Thanks.

thibautcornolti avatar Feb 28 '19 08:02 thibautcornolti

I can see the same error while running the reports, can you please suggest where it's going wrong. Command I am running-

./run.py --billing EBS esg-ebs-costing ebs-nv --ec2 EBS --xlsx-name EBS

MyError

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "src/get_ec2_data.py", line 73, in compute_instance_costs = json.load(f) File "/usr/lib64/python3.6/json/init.py", line 299, in load parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) File "/usr/lib64/python3.6/json/init.py", line 354, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "src/get_ec2_data.py", line 73, in compute_instance_costs = json.load(f) File "/usr/lib64/python3.6/json/init.py", line 299, in load parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) File "/usr/lib64/python3.6/json/init.py", line 354, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "src/get_ec2_data.py", line 73, in compute_instance_costs = json.load(f) File "/usr/lib64/python3.6/json/init.py", line 299, in load parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) File "/usr/lib64/python3.6/json/init.py", line 354, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "src/get_ec2_data.py", line 73, in compute_instance_costs = json.load(f) File "/usr/lib64/python3.6/json/init.py", line 299, in load parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) File "/usr/lib64/python3.6/json/init.py", line 354, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "src/get_ec2_data.py", line 73, in compute_instance_costs = json.load(f) File "/usr/lib64/python3.6/json/init.py", line 299, in load parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) File "/usr/lib64/python3.6/json/init.py", line 354, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "src/get_ec2_data.py", line 73, in compute_instance_costs = json.load(f) File "/usr/lib64/python3.6/json/init.py", line 299, in load parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) File "/usr/lib64/python3.6/json/init.py", line 354, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "src/get_ec2_data.py", line 73, in compute_instance_costs = json.load(f) File "/usr/lib64/python3.6/json/init.py", line 299, in load parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) File "/usr/lib64/python3.6/json/init.py", line 354, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "src/get_ec2_data.py", line 73, in compute_instance_costs = json.load(f) File "/usr/lib64/python3.6/json/init.py", line 299, in load parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) File "/usr/lib64/python3.6/json/init.py", line 354, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "src/get_ec2_data.py", line 73, in compute_instance_costs = json.load(f) File "/usr/lib64/python3.6/json/init.py", line 299, in load parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) File "/usr/lib64/python3.6/json/init.py", line 354, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) File "/usr/lib64/python3.6/json/init.py", line 354, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Fetched ec2 data for all accounts in eu-north-1 Fetched ec2 data for all accounts in ap-south-1 Fetched ec2 data for all accounts in eu-west-3 Fetched ec2 data for all accounts in eu-west-2 Fetched ec2 data for all accounts in eu-west-1 Fetched ec2 data for all accounts in ap-northeast-2 Fetched ec2 data for all accounts in ap-northeast-1 Fetched ec2 data for all accounts in sa-east-1 Fetched ec2 data for all accounts in ca-central-1 Fetched ec2 data for all accounts in ap-southeast-1 Fetched ec2 data for all accounts in ap-southeast-2 Fetched ec2 data for all accounts in eu-central-1 Fetched ec2 data for all accounts in us-east-1 Fetched ec2 data for all accounts in us-east-2 Fetched ec2 data for all accounts in us-west-1 Fetched ec2 data for all accounts in us-west-2 Fetching ec2 metadata for EBS in eu-north-1... Fetching ec2 metadata for EBS in ap-south-1... Fetching ec2 metadata for EBS in eu-west-3... Fetching ec2 metadata for EBS in eu-west-2... Fetching ec2 metadata for EBS in eu-west-1... Fetching ec2 metadata for EBS in ap-northeast-2... Fetching ec2 metadata for EBS in ap-northeast-1... Fetching ec2 metadata for EBS in sa-east-1... Fetching ec2 metadata for EBS in ca-central-1... Fetching ec2 metadata for EBS in ap-southeast-1... Fetching ec2 metadata for EBS in ap-southeast-2... Fetching ec2 metadata for EBS in eu-central-1... Fetching ec2 metadata for EBS in us-east-1... Fetching ec2 metadata for EBS in us-east-2... Fetching ec2 metadata for EBS in us-west-1... Fetching ec2 metadata for EBS in us-west-2... [954326812311 - eu-north-1] Getting instances metadata... [954326812311 - ap-south-1] Getting instances metadata... [954326812311 - eu-west-3] Getting instances metadata... [954326812311 - eu-north-1] Done! [954326812311 - eu-west-2] Getting instances metadata... [954326812311 - eu-west-1] Getting instances metadata... [954326812311 - eu-west-3] Done! [954326812311 - ap-south-1] Done! [954326812311 - ap-northeast-1] Getting instances metadata... [954326812311 - eu-west-2] Done! [954326812311 - eu-west-1] Done! Fetched ec2 metadata for EBS in eu-north-1 [954326812311 - ap-northeast-2] Getting instances metadata... [954326812311 - sa-east-1] Getting instances metadata... [954326812311 - ca-central-1] Getting instances metadata... Fetched ec2 metadata for EBS in ap-south-1 Fetched ec2 metadata for EBS in eu-west-3 [954326812311 - ap-southeast-1] Getting instances metadata... Fetched ec2 metadata for EBS in eu-west-2 Fetched ec2 metadata for EBS in eu-west-1 [954326812311 - ap-southeast-2] Getting instances metadata... [954326812311 - ca-central-1] Done! [954326812311 - ap-northeast-1] Done! [954326812311 - eu-central-1] Getting instances metadata... [954326812311 - sa-east-1] Done! [954326812311 - us-east-1] Getting instances metadata... [954326812311 - ap-northeast-2] Done! [954326812311 - us-east-2] Getting instances metadata... [954326812311 - us-west-1] Getting instances metadata... [954326812311 - us-west-2] Getting instances metadata... Fetched ec2 metadata for EBS in ap-northeast-2 Fetched ec2 metadata for EBS in ap-northeast-1 Fetched ec2 metadata for EBS in sa-east-1 Fetched ec2 metadata for EBS in ca-central-1 [954326812311 - us-east-2] Done! [954326812311 - eu-central-1] Done! [954326812311 - us-east-1] Done! [954326812311 - ap-southeast-2] Done! [954326812311 - us-west-1] Done! [954326812311 - ap-southeast-1] Done! [954326812311 - us-west-2] Done! Fetched ec2 metadata for EBS in ap-southeast-1 Fetched ec2 metadata for EBS in ap-southeast-2 Fetched ec2 metadata for EBS in eu-central-1 Fetched ec2 metadata for EBS in us-east-1 Fetched ec2 metadata for EBS in us-east-2 Fetched ec2 metadata for EBS in us-west-1 Fetched ec2 metadata for EBS in us-west-2 Processing billing data (1/5 - billing diff)... Traceback (most recent call last): File "src/get_bill_diff.py", line 65, in first_month = preserved_months[0] IndexError: list index out of range Processing billing data (2/5 - instance history)... Processing billing data (3/5 - ec2 last month)... Processing billing data (4/5 - ebs last month)... Processing billing data (5/5 - s3 cost)... Generating xlsx file... Traceback (most recent call last): File "src/make_xlsx.py", line 630, in main(name) File "src/make_xlsx.py", line 612, in main gen_weekly_variations(workbook, header_format, val_format) File "src/make_xlsx.py", line 201, in gen_weekly_variations with open(IN_ABSOLUTE_COST_PER_MONTH) as f: FileNotFoundError: [Errno 2] No such file or directory: 'out/absolute.csv'

AashishChughHome avatar Aug 07 '20 19:08 AashishChughHome