Support Azure Through Exports

[tkennes]

Hi there,

During our last discussion, it was mentioned that Azure does not properly support querying resource Usage functionality, as is common within AWS and GCP. I've spent some time figuring out the current status and the options, mostly buried knee-deep in Azure docs... It's a bit messy to get everything together, but here is the result:

• At the moment of writing, the "Usage Details - List" API is used. See the implementation here.
• First of all, the rate limit is pretty tight here. I don't know how much rows are coming through per call though, seems to be 1000 nextPage calls here. And that's at an Enrollment-level (doesn't get higher than that).
• Secondly, the Usage Details - List API is also deprecated (here again, on the top and here for moving away.
• Hence, seems like it might be better to move away.

So, what is Azure moving towards to:

• For small implementations, Azure recommends this, which I haven't tested myself yet. But there seem to be implementations here
• For bigger implementations, from 2GB month-to-month onwards so around 2 million records, Azure recommends using the Exports option, which basically exports it as a csv to blob. See also here.
• So, to me it seems pretty difficult to go around, you could still set up some sort of ingestion and keep MongoDB in the backend for caching. At the same time, from a security perspective, it's not all that bad. You can set up the exports through a different account and let the identity running CCF just have access to the storage account where the data is stored.
• As a sidenote, PowerBI integration seems to be pretty neat.

Let me know what you think! Also happy to jump in a call in order to unpack it a bit, and perhaps discuss implementation details, can't wait to have this working for bigger implementations! 😄

Tom

[4upz]

Hi @tkennes this is awesome! Thanks for gathering all of this information and sharing with us. Apologies for the time that it took to get back to you.

While we've heard of the Usage Details API approaching deprecation and the recommendation to migrate, I think this is a really solid argument to prioritize doing so. We especially were unaware of the exact rate limits associated with the pagination calls, which puts a lot of the roadblocks we've been facing into perspective.

I think supporting both Exports and the Cost Details API are the way to go. Perhaps we can dive into the best way to implement both approaches, while also ensuring that both pay-as-you-go and enterprise users are supported. We'll especially need to investigate a way to support the Blob exports that are compatible with both MongoDB and non-MongoDB users.

If you're still down, let's hop on a call do a mini "story grooming" session to unpack this a bit more and iron out some steps to implement. Would also love to chat more about the Power BI integration, as we've been seeing interest there as well. I'll shoot you an email to discuss availability.

[mountainaireman]

Wanted to share some thoughts about the various methods of getting data from Azure, and hiccups I've run into.

The type of account you have with Azure matters a lot. I just wanted to add a couple notes here for context.

Regarding the Cost Details API (For small implementations, Azure recommends this,), Azure docs state:

The Cost Details API doesn't support management groups for either EA or MCA customers.

There is this about the Cost Details report option:

The Cost Details report is only available for customers with an Enterprise Agreement or Microsoft Customer Agreement. If you have an MSDN, pay-as-you-go, or Visual Studio subscription, you can migrate to Exports or continue using the Consumption Usage Details API.

For exports, there is a list of what is supported and what is not here.

The hiccup my company has run into can be seen in this issue. If the project intends to move away from the deprecated API, it might make sense to close that enhancement. Filing that issue is what led me to learn about how different types of accounts have access in different ways to the APIs.

I hope this is helpful. I think it would be responsible to point out in CCF documentation, where feasible, the foibles of Azure accounts and what is supported and what is not.

[4upz]

@mountainaireman Thanks for sharing your thoughts and insights on this! I think you make a very good point, and is something that we've been realizing about the varying levels of support for Azure Billing accounts when it comes to the usage APIs ever since the issue that you mentioned was created.

We'll take this into account, as well as the very good point of providing additional documentation around this, as we move forward with planning our next steps for implementing these improvements. I think we are very aligned on the position to at least provide the Azure package with more flexibility on handling different types of billing accounts. So enhancements like different filtering methods or exports for different billing agreements would be taken into account when migrating to the new API and Exports options.