Azure
Unreasonably high storage transaction activity (GetBlobProperties) causing increased cost of running Functions
I am running into the same issue as described in #2231, where my functions are generating some ridiculous and unexplained storage activity causing the cost of running functions to increase - more details below. It's worth noting that there were quite a few other similar issues raised in this repo (#2138, #1307, #832), so the problem is not necessarily new.
I also made a comment in one #2231 but it didn't get any traction for a week so I decided to create a new issue, hoping that someone will have a look at this.
To set the context, I have a single .NET 7 Linux Azure Function v4 running on a consumption plan with a queue trigger. It's an open source project so the function code can be found here. As others who experienced this issue, I essentially have noticed that my bills were growing unexpectedly, which lead me to review where the cost is coming from. For some completely unknown reason to me I found out that there is close to ~700k GetBlobProperties operations happening in the storage over the span of a day.
To my huge disappointment, there doesn't seem to be a proper solution to fix this issue and neither there is a documentation to understand why the operations number is so high and how to control/reduce it. I browsed around and found a few "hacks" that allegedly helped others reduce numbers of operations and brought the cost of the storage down. Namely I have done the following:
- Downgraded v2 storage to legacy v1 as per recommendation by @nzthiago (comment)
- Deleted the
AzureWebJobsDashboardfrom the configuration (SO comment) - Ensured that I don't have this line
.AddJsonFile("local.settings.json", optional: true, reloadOnChange: true)in the config code (SO comment) - Switched from Zip Deploy to Zip Deploy to Run With Package (docs) as per this comment
Unfortunately none of the above helped. The storage transaction numbers have not dropped one bit. I'm getting really annoyed and helpless. It is frustrating, to say the least, that there is this "hidden" cost of running functions that is not documented anywhere, which misleads developers into thinking that they can run something inexpensively in Azure. It also amazes me that this has not been properly addressed for such a long time - the first comments that I found relating to this issue were made around 2018. That said, I still hope that someone will be able to help me out.
Hi Mikolaj - we do have it documented in several areas that a storage account is required and used by Azure Functions. For example:
- Pricing page has a note "A storage account is created by default with each Functions app. The storage account is not included in the free grant. Standard storage rates and networking rates charged separately as applicable."
- This page regarding storage considerations
- Other related costs documented here that calls out storage account.
So I wouldn't necessarily say that this is hidden and not documented anywhere. But always looking at how we could improve on this. In our backlog for the long run we hope to remove the need for storage account overall for Functions.
As you mentioned on the other thread a support ticket isn't an option for you but please do share that info as per these instructions. The steps you followed should have addressed this, so we would need to investigate your app specifically to see if there's any extra issue that your app is running into that's causing increasing storage transactions numbers... I really want to help figure out what is going on with your app as this is not expected, thank you for reporting this and for your patience!
hey @nzthiago, I didn't necessarily mean that the storage part of the costs of running Functions is hidden, as I it is clear from the docs you shared. The "hidden" cost that I was referring to is the undocumented part where the storage activity skyrockets if you don't configure the function in certain way - which also doesn't help in all cases (i.e. mine).
I would be very gratefully if you and/or someone from your team could have a look at what could potentially be wrong with my setup. The function that is causing issues is https://bgc-prod-func.azurewebsites.net - let me know if you need any other information for your investigation. Unsure if this is important for your investigations but the function is stopped atm, to avoid any extra charges caused by the storage activity. I'm happy to turn it on if/when needed. I'm also happy to get on a call to explain the setup/config and update things in Azure, for you to test/verify.
Hi Mikolaj - apologies, I might need some more folks from the team to help me investigate. In the meantime, if this is viable, I just followed you on twitter in case you can DM me your email address I can help set up a follow up investigation too. Let me know what you prefer, thank you.