containers-roadmap
containers-roadmap copied to clipboard
aws
[ECS] [request]: EC2 cost allocation by tag
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Tell us about your request Now that we have ECS tags for tasks and services from https://github.com/aws/containers-roadmap/issues/15, it would be nice if Cost Explorer was able to let us slice and dice our EC2 costs based on these tags.
Which service(s) is this request for? ECS
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? The majority of our EC2 spend is on ECS clusters. We're running around 30 applications on each cluster, so it would be really nice to be able to allocate that spend back to our teams easily within cost explorer.
We could probably figure out how to do some manual work each month to take our usage allocation by either vcpu or RAM and then subdivide our EC2 bill by this, but that feels like a lot of work that would be pretty easy to get wrong.
It seems like AWS should have all of the information it needs to do these calculations for us (maybe with some hint from us to determine if the cluster capacity is CPU or memory constrained):
- hourly spend on EC2
- vcpu and RAM allocation per task over time
Are you currently working around this issue? We don't currently allocate EC2 costs on an application or team basis. If needed for an estimate we'll take a proxy like RDS spend and allocate EC2 with the same ratio as a best effort.
Here's an old article covering how to do this manually (before tagging and usage were available): https://aws.amazon.com/blogs/compute/measuring-service-chargeback-in-amazon-ecs/
You can see how it might be a bit overwhelming to try to do manually.
What if some instances are CPU constrained and some are memory constrained? What if some instances are not fully utilized or are not running tasks at all - how would you like those costs allocated?
You'd almost certainly need to have those go to a designated cost center. Companies can then special case handling those.
The "uneven constraints" problem is trickier. If it helps, this doesn't need to be to-the-penny accurate, but rather at the 80%+ level for most use cases.
Having an “unused” dollar cost like Corey mentioned would be ideal (similar to the catch all “no tag key” category when grouping spend by tag). It would be a really interesting “efficiency” metric for your cluster, and might make it easier to figure out if you could save some money by moving to fargate, or figure out better auto scaling strategies.
I’m not sure if instances could figure out themselves if they were memory or cpu constrained (this feels awfully similar to some of the discussions on the autoscaling discussion btw).
In our case (right now at least), our cluster is memory constrained 100% across the board.
The other option I guess might be to calculate two cost metrics: one for memory and one for CPU, and let the user choose which one they’re interested in using?
Any clue, is this feature request accepted and work has started?
I really need this for a long time. An update will be helpful.
For now, we are having a report from the billing S3 bucket for vCPU hours per service. Though it is difficult for the management to understand.
There's a new repo with some automation for this: https://github.com/aws-samples/ecs-chargeback-cloudformation
Quick update: we're working on this feature with AWS Cost Insights to enable customers to view compute costs (aggregated using tags) for ECS Tasks running on EC2. As we work on this, we wanted to solicit your input on a couple of key topics:
- How would you want cost for the EC2 instance to be allocated across vcpu and memory resources?
- How would you want unutilized resources on the host to show up/ be accounted for?
Thank you, we really appreciate your feedback.
@vibhav-ag does it also make sense for item 1. to include GPU resources, if the instance has any tasks with GPUs required? This way for GPU intensive jobs we can tell it to allocate based on % of GPU resources used.
@vibhav-ag +1 I need this feature so much For your survey, I would like to suggest the following:
- I prefer to vote based on vCPU-time, and
- an unallocated portion should be considered.
@vibhav-ag great to see work has started on this :rocket: ! Any news?
Regarding your survey; Ideally both vCPU and RAM would be taken into account and unused should be considered. If you present unused separately, we could use the "used" ratios to distribute the costs. Or ideally, AWS would provide the unused part split as well.
AWS's example of split cost allocation data (based on EC2) seems sound [1]
[1] https://docs.aws.amazon.com/cur/latest/userguide/example-split-cost-allocation-data.html
We have released the feature that Vibhav mentioned in April. The official name is Split Cost Allocation Data. You can find more details in the following links: What's new post Blog Technical Details
If you'd like to see this in action, we recently also did a Container from the Couch session here. You can refer to this GitHub page here to see how you can set up the prerequisites and the demo.
Note: currently this release doesn't address the case that CPU or memory is constrained. If there is an interest, please let us know. We also welcome feedback to this release.
@weijuans Thanks! Wouldn't you consider this "[ECS] [request]: EC2 cost allocation by tag" Done then? It's still Open and in the "Coming Soon" column. (P.S. For proper feedback of the current feature, I'd first need to gain some real life experience with it.)