azapi_resource_list and response_export_values support for more complex JSON

[DevopsMercenary]

I'm trying to get the name of all the file shares in a storage account. I run azapi_resource_list and returns the following JSON

  "value":
    [
      {
        "id": "/subscriptions/0000000-0000-0000-0000-000000000000/resourceGroups/maintenance-configuration/providers/Microsoft.Storage/storageAccounts/winfunapp/fileServices/default/shares/maintenance-function-app-bfc8",
        "name": "maintenance-function-app-bfc8",
        "type": "Microsoft.Storage/storageAccounts/fileServices/shares",
        "etag": "0x8DC8E72C1214326",
        "properties":
          {
            "leaseStatus": "unlocked",
            "leaseState": "available",
            "accessTier": "TransactionOptimized",
            "lastModifiedTime": "2024-06-17T02:11:09.0000000Z",
            "shareQuota": 102400,
            "enabledProtocols": "SMB"
          }
      }
    ]
}

Great, awesome. Now I see this response_export_values option in azapi_resource_list and I would like to just get the name to return...

And for the life of me I have no idea as to how I can do it. I've tried a bunch of options but doesn't seem to be possible for instance...

  response_export_values = ["value"]
  response_export_values = ["value.name"]
  response_export_values = ["value[].name"]
  response_export_values = ["value[0].name"]

[stemaMSFT]

Thanks for the issue @DevopsMercenary! This one is a lack of clarity that we may want to update in docs. Value is already implicitly included, so you’d want to do [“name”] I believe. Let me know if that doesn’t work. @ms-henglu for FYI

[ms-henglu]

Hi @DevopsMercenary ,

Thank you for taking time to report this issue!

The output of azapi_resource_list data source could be in HCL format if specifying enable_hcl_output_for_data_source = true in the provider block. The default output is JSON format, but it's also possible to use jsondecode function to parse it to an HCL object.

After that, you could use the splat expression to get the name.

Here's an example:

provider "azapi" {
  enable_hcl_output_for_data_source = true
}

data "azapi_resource_list" "test" {
  type = "Microsoft.Automation/automationAccounts@2023-11-01"
  parent_id = "/subscriptions/{subscription id}"
  response_export_values = ["*"]
}

output "o1" {
  // value = ["foo", "bar"]
  value = data.azapi_resource_list.test.output.value[*].name
}

More details about the splat expression: https://developer.hashicorp.com/terraform/language/expressions/splat

[smokedlinq]

Is the HCL output required for this to work? jmsepath on the json would be handy here...

[smokedlinq]

@ms-henglu I did try doing this with something like this but it's not filtering the output ...

terraform {
  required_providers {
    azapi = {
      source  = "Azure/azapi"
      version = "~> 1.13"
    }
  }
}

provider "azapi" {
  enable_hcl_output_for_data_source = true
}

variable "scope" {
  type = string
}

data "azapi_resource_list" "role_definitions" {
  parent_id              = var.scope
  type                   = "Microsoft.Authorization/roleDefinitions@2022-04-01"
  response_export_values = ["id", "properties.roleName"]
}

output "value" {
  value = data.azapi_resource_list.role_definitions.output
}

[smokedlinq]

Best I can tell from the filtering, it doesn't support properties that are arrays to filter the child values as well ...

[ms-henglu]

Hi @smokedlinq ,

Yes, we'll consider to support JMSEPath in the next major release.

Currently, we need to make the output as an HCL object first(either using the enable_hcl_output_for_data_source or the jsondecode function). After that, you could use Terraform functions to filter the result, for example, using the for expression to filter the result: https://developer.hashicorp.com/terraform/language/expressions/for

[smokedlinq]

Yes but it still stores the full response in the state file. This causes excessively large state files and we actually hit a limit because of this. Our use case is each subscription needs its role definitions list for lookups so we load that. That response alone is 700kb. Multiplied across 150 subscriptions that single lookup is 100+mb. Combine this with other use cases such as the list of subscriptions or role assignments and suddenly 1+gb state files are a thing...

[smokedlinq]

FYI the hcl output is awesome but the dynamic schema is stored per item of a list in these kind of responses causing the large state data demands.

[ms-henglu]

Thanks for the details! I've created this issue to track the JSMEPath support: https://github.com/Azure/terraform-provider-azapi/issues/535

[DevopsMercenary]

Is the HCL output required for this to work? jmsepath on the json would be handy here...

Totally Agree @smokedlinq . For now I did convert JSON to HCL and a bunch of local vars to carefully parse out what I wanted. JMSEPath would be awesome, to (1) only get what I want directly, and (2) to reduce the size of the state file.

Looking forward to https://github.com/Azure/terraform-provider-azapi/issues/535

[cwe1ss]

I've been trying to filter the output of a azapi_resource_list in v2 and have been struggling quite a lot until I found a working solution:

data "azapi_resource_list" "role_definitions" {
  type      = "Microsoft.Authorization/roleDefinitions@2022-05-01-preview"
  parent_id = "/subscriptions/${var.subscription_id}"

  response_export_values = {
    # Returns null
    "name"     = "name"
    "roleName" = "properties.roleName"

    # Works but returns two separate arrays instead of objects with two properties
    "name2" = "value[].name"
    "role2" = "value[].properties.roleName"

    # Correctly returns a list of objects with two properties each
    "value" = "value[].{name: name, roleName: properties.roleName}"
  }
}

So in order to properly filter the results one has to combine all desired properties into a single field using "value" = "value[].{name: name, roleName: properties.roleName}". This syntax seems quite unintuitive to me and quickly becomes hard to read if there are multiple properties involved.

Figuring this out was even harder, because the existing documentation for this on the azapi_resource_list-page seems to be wrong, as it indicates that you can just use e.g. "role_name" = "properties.roleName".

Would it make sense, that we can actually use the syntax from the documentation and AzApi would automatically convert this into value[].{name: name, roleName: properties.roleName} if the response contains a value-array?

response_export_values = {
  "name" = "name"
  "roleName" = "properties.roleName"
}

[ms-henglu]

Thanks @cwe1ss for adding this great example!

Figuring this out was even harder, because the existing documentation for this on the azapi_resource_list-page seems to be wrong, as it indicates that you can just use e.g. "role_name" = "properties.roleName".

I'll open a PR to update the document to use a valid example.

Would it make sense, that we can actually use the syntax from the documentation and AzApi would automatically convert this into value[].{name: name, roleName: properties.roleName} if the response contains a value-array?

Thanks for the suggestion, but to align with implementation in other azapi resources/data sources, I think it's better to keep using the value of map is a JMESPath query string to filter the response.

[ms-henglu]

https://github.com/Azure/terraform-provider-azapi/pull/651/files

[ms-henglu]

Hi all,

This feature has been released, more details please see: https://registry.terraform.io/providers/Azure/azapi/latest/docs/guides/2.0-upgrade-guide#jmespath-query-support

I'll close this issue as it's completed, feel free to reopen it if there's any questions.