In azure.azure_firewall table seeing few keys are missing for json object when passing it directly using transform.FromField

[kaidaguerre]

Slack conversation thread https://turbothq.slack.com/archives/C01AC8JQNHH/p1610017247038000

		{
			Name:        "ip_configurations",
			Description: "A collection of IP configuration of the Azure Firewall resource",
			Type:        proto.ColumnType_JSON,
			Transform:   transform.FromField("AzureFirewall.AzureFirewallPropertiesFormat.IPConfigurations"),
		},

Test column using transform

			{
				Name:        "test_column",
				Description: "A list of AKAs (also-known-as) that uniquely identify this resource",
				Type:        proto.ColumnType_JSON,
				Transform:   transform.From(testFunction),
			},

Here in field test_column we are getting all the properties but not in the field ip_configurations missing keys for ip_configurations

    "PrivateIPAddress": "10.0.0.4",
    "ProvisioningState": "Succeeded",

steampipe query --output json "select name, test_column, ip_configurations from azure.azure_firewall"

[
 {
  "ip_configurations": [
   {
    "id": "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbot_rg/providers/Microsoft.Network/azureFirewalls/test_firewall/azureFirewallIpConfigurations/test_pi",
    "name": "test_pi",
    "properties": {
     "publicIPAddress": {
      "id": "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbot_rg/providers/Microsoft.Network/publicIPAddresses/test_pi"
     },
     "subnet": {
      "id": "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbot_rg/providers/Microsoft.Network/virtualNetworks/testvn/subnets/AzureFirewallSubnet"
     }
    }
   }
  ],
  "name": "test_firewall",
  "test_column": [
   {
    "PrivateIPAddress": "10.0.0.4",
    "ProvisioningState": "Succeeded",
    "PublicIPAddress": {
     "id": "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbot_rg/providers/Microsoft.Network/publicIPAddresses/test_pi"
    },
    "Subnet": {
     "id": "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbot_rg/providers/Microsoft.Network/virtualNetworks/testvn/subnets/AzureFirewallSubnet"
    }
   }
  ]
 },
]

Test transform function

func testFunction(ctx context.Context, d *transform.TransformData) (interface{}, error) {
	data := d.HydrateItem.(firewallInfo)

	var output []map[string]interface{}
	for _, firewall := range *data.AzureFirewall.AzureFirewallPropertiesFormat.IPConfigurations {
		output = append(output, map[string]interface{}{
			"PrivateIPAddress":  firewall.AzureFirewallIPConfigurationPropertiesFormat.PrivateIPAddress,
			"Subnet":            firewall.AzureFirewallIPConfigurationPropertiesFormat.Subnet,
			"PublicIPAddress":   firewall.AzureFirewallIPConfigurationPropertiesFormat.PublicIPAddress,
			"ProvisioningState": firewall.AzureFirewallIPConfigurationPropertiesFormat.ProvisioningState,
		})
	}
	return output, nil
}

Struct for IPConfigurations

type AzureFirewallIPConfigurationPropertiesFormat struct {
	// PrivateIPAddress - READ-ONLY; The Firewall Internal Load Balancer IP to be used as the next hop in User Defined Routes.
	PrivateIPAddress *string `json:"privateIPAddress,omitempty"`
	// Subnet - Reference to the subnet resource. This resource must be named 'AzureFirewallSubnet' or 'AzureFirewallManagementSubnet'.
	Subnet *SubResource `json:"subnet,omitempty"`
	// PublicIPAddress - Reference to the PublicIP resource. This field is a mandatory input if subnet is not null.
	PublicIPAddress *SubResource `json:"publicIPAddress,omitempty"`
	// ProvisioningState - READ-ONLY; The provisioning state of the Azure firewall IP configuration resource. Possible values include: 'Succeeded', 'Updating', 'Deleting', 'Failed'
	ProvisioningState ProvisioningState `json:"provisioningState,omitempty"`
}

Branch name azure-firewall-test on https://github.com/turbotio/steampipe-plugin-azure repo

[tyagiparth]

This seems to be a bug with Azure SDK where we have a custom marshaler for the AzureFirewallIPConfigurationPropertiesFormat struct which does not marshal the PrivateIPAddress and ProvisioningState properties. The function is:

// MarshalJSON is the custom marshaler for AzureFirewallIPConfigurationPropertiesFormat.
func (aficpf AzureFirewallIPConfigurationPropertiesFormat) MarshalJSON() ([]byte, error) {
	objectMap := make(map[string]interface{})
	if aficpf.Subnet != nil {
		objectMap["subnet"] = aficpf.Subnet
	}
	if aficpf.PublicIPAddress != nil {
		objectMap["publicIPAddress"] = aficpf.PublicIPAddress
	}
	return json.Marshal(objectMap)
}

The workaround for this would be something we are already doing with a transform function.