By default, refunds are ignored

[ansoni]

Hello Guys, Our company has received some refunds from AWS in the past and these values are not reflected in the ice graphs/data. I dug in awhile back and found that negative numbers aren't properly handled(ignored I believe). I will be addressing this issue, but was curious if anyone had any insight to this behavior.

Thanks!

[vfilanovsky]

Anthony - could you describe what exactly you see in the billing file?

[ansoni]

Here are two examples.

"45","111111111111","2222222222","LineItem","5400000000000000001","Amazon Elastic Compute Cloud","2400001",,,"EU-BoxUsage:m1.medium","RunInstances",,"N","COMPANY NAME CREDITS MAR21-JUN21",,,,,"-0.844161",,"-0.844161",,,,,,,,,,

"46","1111111111","22222222222","LineItem","54000000000000001","Amazon Elastic Compute Cloud","170000002",,,"EU-HeavyUsage:m2.4xlarge","RunInstances",,"Y","BusDev:0011111111",,,,,"-196.838912",,"-196.838912",,,,,,,,,,

[vfilanovsky]

I didn't find any lines with negative cost in our current billing file. From your examples, I see that those lines has no date range - is that right? How would you like to see these amounts reflected? As as single transaction or spread across the whole month?

[ansoni]

An RI Cancelation didn't have any dates associated with it. This makes sense to distribute across the entire month.

"12345678","111111111","22222222222","LineItem","Amazon Elastic Compute Cloud","0",,,,,,"Y","RI Cancellation:username:00111111111- TT:0011111111",,,,,"-4660.000000",,"-4660.000000"

The rest appear to have dates:

Here is a Sampling from a few months of our largest account(account identifiers removed):

"12345678","111111111","22222222222","LineItem","Amazon Elastic Compute Cloud","200000",,,"BoxUsage:r3.xlarge","RunInstances",,"N","SERVICE-COMPANYNAME CREDITS","2014-11-19 00:00:00","2014-11-30 23:59:59",,,"-25.709510",,"-25.709510"

"12345678","111111111","22222222222","LineItem","Amazon Elastic Compute Cloud","2345678","345678","500005","EBS:VolumeIOUsage","EBS:IO-Write","","N","$0.05 per 1 million I/O requests - US East (Northern Virginia)","2014-11-21 15:00:00","2014-11-21 16:00:00","-15665467.00000000","0.0000000500","-0.78327335","0.0000000500","-0.78327335"

"12345678","111111111","22222222222","LineItem","Amazon Elastic Compute Cloud","900001","35424642","200006","DataTransfer-Regional-Bytes","PublicIP-Out","","N","$0.010 per GB - regional data transfer - in/out/between EC2 AZs or using IPs or ELB","2014-10-04 12:00:00","2014-10-04 13:00:00","-3.24792093","0.0100000000","-0.03247921","0.0100000000","-0.03247921"

"123445678","111111111","22222222222","LineItem","Amazon Elastic Compute Cloud","2345678","345678","45678","USW2-LoadBalancerUsage","LoadBalancing","","N","$0.025 per LoadBalancer-hour (or partial hour)","2014-09-25 16:00:00","2014-09-25 17:00:00","-1.00000000","0.0250000000","-0.02500000","0.0250000000","-0.02500000"

My grep is ',"-' to find the negative line items.

[ckelner]

Hello, we've also seen this over at TWC and had planned to look into. Glad someone else has started the footwork :)! :+1: We had started a thread over on ice users/google group regarding this as well: https://groups.google.com/forum/#!topic/iceusers/XooSUnL56JU

I spoke with AWS about it (to get some details on what a refund looks like) because our file is pretty huge (70+ accounts, several million rows in the csv, several Gb in size). Anyways here's the AWS knowledge if it is helpful in any way:

A credit will appear in the detailed billing file with these attributes:

- The RateId column will be "0".
- The BlendedCost column will have a negative value.
- The negative values will appear at the end of the file just before the total.

I've attached a screenshot from my own account which shows a $1.00 credit against AWS Support.

Refunds will not appear in the DBR, this information only applies to credits.

And here's the provided screenshot:

Please let us know if we can be of assistance.

[ansoni]

So, it appears that this line in the BasicLineItemProcessor is causing these to be ignored:

  if (StringUtils.isEmpty(items[accountIdIndex]) ||
        StringUtils.isEmpty(items[productIndex]) ||
        StringUtils.isEmpty(items[usageTypeIndex]) ||
        StringUtils.isEmpty(items[operationIndex]) ||
        StringUtils.isEmpty(items[usageQuantityIndex]) ||
        StringUtils.isEmpty(items[costIndex]))
        return Result.ignore;

It appears that refunds are missing require fields such as usage, operation or quantity.

@vfilanovsky : can you confirm that you haven't started any work on this. I just don't want to duplicate your efforts.

[vfilanovsky]

@ansoni - no, I haven't. We didn't have a goal of reconciling with DBR. Ice is rather a usage tracker. Credits/refunds affect the totals only, not the cost or quantity of any real usage operation. Perhaps credits can be tracked under a separate operation, much like UpfrontAmortizedHeavy, which has cost component, but no usage.