bicep
bicep copied to clipboard
Azure
Increase MaxLiteralCharacterLimit for loadTextContent (specifically for non-JSON content)
Increase MaxLiteralCharacterLimit for loadTextContent https://github.com/Azure/bicep/blob/main/src/Bicep.Core/LanguageConstants.cs#L32
Some of logic app JSON code as well as Azure Workbooks cannot be used with this function due to them being larger than 131072 characters. A lot of these resources would be created via azure portal and then used within arm templates/bicep for deployment. It would be great to have the ability to load it's content from within a separate file.
Options to implement this:
- We could increase the limit in the runtime.
- We could put the content into a variable instead of a JSON expression string literal. (Would have to avoid clashes with user-defined variables, though.)
Is there any update on this investigation? I've found the same character limitation brings down an idea of using otherwise fully functioning API Management api publishments using bicep, because openapi files can be larger than the current 131072 character limitation allows.
I'd really like this feature too. I have a separate process that adds a lot of elements to an app gateway, which I then update intermittently via Bicep as part of my core infrastructure maintenance. I read in the appGw config as it stands and can output it to a file for Bicep to use so that the appGw template is made up of all the features I need without overwriting the separate process changes that were made. The file size limit really restricts this as my appGw grows in size though, so runtime changes would be brilliant to amend this higher... @alex-frankel is this on a roadmap for future implementation at all?
It's something we would like to do, but I doubt it will be dealt with in the next 3-6 months. It would be helpful if this issue gets more upvotes, so we can shift priorities accordingly.
@alex-frankel I have a monitor workbook with more than 131072 characters. I wonder can you somehow make the function to not account for tabs/whitespaces or that does not matter for the function?
You can try to deploy the SOC Process Framework Sentinel workbook from https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/Workbooks/SOCProcessFramework.json - I have found no way to do this using Bicep, because of this limitation. The same number appears when i try to add the content using json() with a string rather than through loadTextContent
It is a blocker for deploying open api spec files for an API. Those files can get larger if they have more operations :(
Any ETA on when this could be implemented ? im hitting this problem right now with a workbook im trying to use.
As someone mentioned above it's a blocker really. Bicep seems ok when you want to build something simple but if you got something advanced to do it might be better to stick to something else like terraform. Is there any information when we can expect solution for that issue?
We'll discuss it again in triage this week, but I'm moving this to the 1.0 milestone for now. We'll see if we can get it done sooner.
maximum size of 131072 characters. Please make sure you understand this is characters not bytes. This makes this feature unusable. Any work arounds?
I hit the same problem today with a swagger import file that was 213,293 bytes.
I found a workaround for large json files, by using loadJsonContent() then wrapping the result with string(..) as follows...
var swagger = string(loadJsonContent('api.swagger.json'))
@qualitas-software i see you highlight json, i assume this doesn't work for loadTextContent ? 😄
@verschaevesiebe yeah the original issue is about loadTextContent(). I haven't tried it, but could you put the encoded file into json, use a version of the method above and extract it? Eg- { "encoded": "<base64 encoded file>" } then load it var fileContent = string(loadJsonContent('file.encoded.as.json').encoded) ? Good luck.
@qualitas-software nice workaround, would have come up with it myself but I need structure so I've translated all OpenAPI Specs from YAML to JSON and we're now using the loadJsonContent 😄
Goodluck to anyone else running into file size limitations 🤗
Turns out that loadJsonContent() does not have this same problem. It seems most, if not all, of the cases where someone would run into this limit is for JSON content. Given that, we are going to close this and recommend you use loadJsonContent() instead.
If we are missing any cases that do not involve JSON content, please let us know and we can revisit.
@alex-frankel may be for both commands will be best to document the maximum limits of size/characters? If not already, I have not checked.
If we are missing any cases that do not involve JSON content, please let us know and we can revisit.
I have several cases of xml files (edifact xsd schemas) which exceeds the text limit by far, so this is absolutely not for just json documents.
Why are there even different size limits for json and text content?
It would be great to have some transparency on this. Is there some sort of limitation that would prevent this limit to be raised globally?
Without this transparency we might be guessing the reasons for this not being implemented which isn't exactly helpful both for Microsoft nor open-source community.
Why are there even different size limits for json and text content?
@miqm knows better than me, but my guess is we can handle JSON more efficiently because it's the same format as the template.
Thank you for sharing the examples @johan-burman. Which resource types are you using those XML schemas for? I will re-open the issue and make it more specific to non-JSON content.
@JayDoubleu -- hopefully you have seen that we always try to be as transparent as possible. The reason we closed this is because we thought it only needed to cover JSON content. At this point, we will need to take another look and see if we need to prioritize this sooner.
The limit for loadTextContent is a limit for ARM template expression as noted here
As Alex pointed, loadJsonContent leverages ARM's JSON structure to "omit" the bicep compilation and put file directly into Intermediate Language - which is ARM JSON (load is not direct though - any potential ARM Template expression escaping is done to avoid loaded file being processed by ARM runtime).
LoadJsonContent does not enforce any limit - the only limit is 4MB - which is a limit for ARM template. So even if you have a 4MB JSON file your template most probably will fail due to other compiled resources "taking space" in the intermediate ARM file.
AFAIK the expression limit is in ARM Runtime to not process too long expressions. However @anthony-c-martin or @majastrz can give more light on this.
Perhaps a solution would be to give ability to provide some storage account details which could be used as intermediate artifact storage during deployment?
Thanks for re-opening the ticket!
Thank you for sharing the examples @johan-burman. Which resource types are you using those XML schemas for? I will re-open the issue and make it more specific to non-JSON content.
The case I'm looking at right now is adding a large edi flatfile schema (xsd) to an integration account. Before this I came across the same limit with an Xslt, but was able to make it fit by removing whitespace.
If this limit needs to stay as-is, maybe there's some other way to achieve the same result, which we are overlooking?