Module output that uses a for loop with map/union is not rendered correctly as input to another module

[JustinGrote]

Bicep version bicep v0.29.47

Describe the bug A bicep output that uses a for loop along with map/union

output tableInfo array = [for (table, i) in tableNames: {
  name: auxTables[i].name
  columns: map(
    union(auxTables[i].properties.schema.columns, tables[i].properties.schema.standardColumns),
    c => {
      name: c.name
      type: c.type
    }
  )
}]

Looks just fine as output to the module:

However when now passing that output as input to another module:

The columns are now missing

What gives?

To Reproduce See above

Additional context Add any other context about the problem here.

[JustinGrote]

As a hack workaround, applying string to the columns property and json on the other side ends up preserving it. Very strange.

[anthony-c-martin]

@JustinGrote would you be able to share the full output value for tableInfo from your screenshot? I wasn't able to reproduce this myself with the code you shared. Feel free to reach me at [email protected] if you're not comfortable sharing this publicly.

It would also be interesting to see if we can rule out map/union as related - I would expect the same issue to appear with the following:

// './tableInfo.json' is a valid json file, with the contents copied+pasted from `tableInfo` output in your screenshot
output tableInfo array = loadJsonContent('./tableInfo.json')

One hypothesis is that the JSON happens to contain something which the 2nd Bicep file thinks is an expression that needs to be evaluated, rather than seeing it as a constant 'value'.

[JustinGrote]

Sure, here it is, I'm trying to create auxiliary log equivalents from existing Sentinel tables. If you remove the string/jsonifying on the column property in each template, it doesn't work and columns is blank. Note I've added some filters but those aren't creating the issue.

Also I am using an api version for the workspaces/tables that doesn't exist in bicep yet, because you need that to create Auxiliary tables.

main.bicep

param workspaceName string
param logRetentionDays int = 90
param tableNames array = [
  'ASimNetworkSessionLogs'
  'ASimAuthenticationEventLogs'
  'ASimWebSessionLogs'
  'ASimAuditEventLogs'
  'ASimDhcpEventLogs'
  'ASimDnsActivityLogs'
  'ASimFileEventLogs'
  'Syslog'
  'CommonSecurityLog'
]
@allowed(['Basic', 'Auxiliary'])
param plan string = 'Auxiliary'

param reservedColumnNames array = [
  '_ResourceId'
  'id'
  '_ResourceId'
  '_SubscriptionId'
  'TenantId'
  'Type'
  'UniqueId'
  'Title'
  'MG'
]

resource workspace 'Microsoft.OperationalInsights/workspaces@2023-09-01' existing = {
  name: workspaceName
}
resource tables 'Microsoft.OperationalInsights/workspaces/tables@2022-10-01' existing = [for tableName in tableNames: {
  parent: workspace
  name: tableName
}]

resource auxTables 'Microsoft.OperationalInsights/workspaces/tables@2023-03-01-preview' = [for (tableName, i) in tableNames: {
  parent: workspace
  name: '${tables[i].name}_Aux_CL'

  properties: {
    schema: union(
      tables[i].properties.schema,
      {
        name: '${tables[i].properties.schema.name}_Aux_CL'
        displayName: null
      }
    )
    plan: plan
    totalRetentionInDays: logRetentionDays
  }
}]

// Has to be done as a separate module because we want to toObject a for loop
module dcr 'asimDCR.bicep' = {
  name: '${deployment().name}-dcr'
  params: {
    dcrName: '${deployment().name}-dcr'
    location: workspace.location
    workspaceName: workspaceName
    tableInfo: [for (table, i) in tableNames: { //Ideally would be a var but can't because of dependent reference
      name: auxTables[i].name
      columns: string(
        filter(
          filter(
            map(
              union(auxTables[i].properties.schema.columns, tables[i].properties.schema.standardColumns),
              c => {
                name: c.name
                type: c.type
              }
            ),
            c => !contains(reservedColumnNames, c.name)
          ),
        c => c.type != 'Guid')
      )
    }]
  }
}

asimDCR.bicep

param dcrName string
param location string = resourceGroup().location
param workspaceName string
param tableInfo array

var workspaceReference = 'workspace'

resource workspace 'Microsoft.OperationalInsights/workspaces@2023-09-01' existing = {
  name: workspaceName
}

resource dcr 'Microsoft.Insights/dataCollectionRules@2023-03-11' = {
  name: dcrName
  location: location
  kind: 'Direct'
  properties: {
    description: 'Log Analytics Ingestion Point for Auxiliary Tables'
    destinations: {
      logAnalytics: [
        {
          name: workspaceReference
          workspaceResourceId: workspace.id
        }
      ]
    }
    streamDeclarations: toObject(
      tableInfo,
      k=>'Custom-${k.name}',
      v=>{columns: map(
        json(v.columns),
        c => {
          name: c.name
          // Guid is not a valid stream type even though it's a valid table type
          type: replace(c.type, 'guid', 'string')
        }
      )}
    )
    dataFlows: [for table in tableInfo: {
      streams: ['Custom-${table.name}']
      destinations: [workspaceReference]
      outputStream: 'Custom-${table.name}'
    }]
  }
}

output dcrRuleId string = dcr.properties.immutableId
output dcrLogIngestionEndpoint string = dcr.properties.endpoints.logsIngestion
output streamNames array = objectKeys(dcr.properties.streamDeclarations)

[anthony-c-martin]

@JustinGrote, thanks! Would you be willing to share the actual resolved value for the tableInfo parameter in the working scenario?

[JustinGrote]

Sure, I reduced it to one table for simplicity.

{
  "name": "ASimNetworkSessionLogs_Aux_CL",
  "columns": "[{\"name\":\"TimeGenerated\",\"type\":\"datetime\"},{\"name\":\"EventCount\",\"type\":\"int\"},{\"name\":\"EventSchemaVersion\",\"type\":\"string\"},{\"name\":\"DvcAction\",\"type\":\"string\"},{\"name\":\"EventMessage\",\"type\":\"string\"},{\"name\":\"EventSeverity\",\"type\":\"string\"},{\"name\":\"EventStartTime\",\"type\":\"datetime\"},{\"name\":\"EventEndTime\",\"type\":\"datetime\"},{\"name\":\"DvcMacAddr\",\"type\":\"string\"},{\"name\":\"Dvc\",\"type\":\"string\"},{\"name\":\"DvcZone\",\"type\":\"string\"},{\"name\":\"EventProductVersion\",\"type\":\"string\"},{\"name\":\"DvcOriginalAction\",\"type\":\"string\"},{\"name\":\"DvcInterface\",\"type\":\"string\"},{\"name\":\"DvcSubscriptionId\",\"type\":\"string\"},{\"name\":\"EventOriginalSeverity\",\"type\":\"string\"},{\"name\":\"EventOriginalSubType\",\"type\":\"string\"},{\"name\":\"NetworkApplicationProtocol\",\"type\":\"string\"},{\"name\":\"NetworkProtocolVersion\",\"type\":\"string\"},{\"name\":\"NetworkDirection\",\"type\":\"string\"},{\"name\":\"NetworkIcmpCode\",\"type\":\"int\"},{\"name\":\"NetworkIcmpType\",\"type\":\"string\"},{\"name\":\"NetworkConnectionHistory\",\"type\":\"string\"},{\"name\":\"DstBytes\",\"type\":\"long\"},{\"name\":\"SrcBytes\",\"type\":\"long\"},{\"name\":\"NetworkBytes\",\"type\":\"long\"},{\"name\":\"DstPackets\",\"type\":\"long\"},{\"name\":\"SrcPackets\",\"type\":\"long\"},{\"name\":\"NetworkPackets\",\"type\":\"long\"},{\"name\":\"NetworkSessionId\",\"type\":\"string\"},{\"name\":\"DstZone\",\"type\":\"string\"},{\"name\":\"DstInterfaceName\",\"type\":\"string\"},{\"name\":\"DstInterfaceGuid\",\"type\":\"string\"},{\"name\":\"DstMacAddr\",\"type\":\"string\"},{\"name\":\"DstVlanId\",\"type\":\"string\"},{\"name\":\"DstSubscriptionId\",\"type\":\"string\"},{\"name\":\"DstGeoCountry\",\"type\":\"string\"},{\"name\":\"DstGeoRegion\",\"type\":\"string\"},{\"name\":\"DstGeoCity\",\"type\":\"string\"},{\"name\":\"DstGeoLatitude\",\"type\":\"real\"},{\"name\":\"DstGeoLongitude\",\"type\":\"real\"},{\"name\":\"DstUserId\",\"type\":\"string\"},{\"name\":\"DstUserIdType\",\"type\":\"string\"},{\"name\":\"DstUsername\",\"type\":\"string\"},{\"name\":\"DstUsernameType\",\"type\":\"string\"},{\"name\":\"DstUserType\",\"type\":\"string\"},{\"name\":\"DstOriginalUserType\",\"type\":\"string\"},{\"name\":\"DstAppName\",\"type\":\"string\"},{\"name\":\"DstAppId\",\"type\":\"string\"},{\"name\":\"DstAppType\",\"type\":\"string\"},{\"name\":\"SrcZone\",\"type\":\"string\"},{\"name\":\"SrcInterfaceName\",\"type\":\"string\"},{\"name\":\"SrcInterfaceGuid\",\"type\":\"string\"},{\"name\":\"SrcMacAddr\",\"type\":\"string\"},{\"name\":\"SrcVlanId\",\"type\":\"string\"},{\"name\":\"SrcSubscriptionId\",\"type\":\"string\"},{\"name\":\"SrcGeoCountry\",\"type\":\"string\"},{\"name\":\"SrcGeoRegion\",\"type\":\"string\"},{\"name\":\"SrcGeoCity\",\"type\":\"string\"},{\"name\":\"SrcGeoLatitude\",\"type\":\"real\"},{\"name\":\"SrcGeoLongitude\",\"type\":\"real\"},{\"name\":\"SrcAppName\",\"type\":\"string\"},{\"name\":\"SrcAppId\",\"type\":\"string\"},{\"name\":\"SrcAppType\",\"type\":\"string\"},{\"name\":\"DstNatIpAddr\",\"type\":\"string\"},{\"name\":\"DstNatPortNumber\",\"type\":\"int\"},{\"name\":\"SrcNatIpAddr\",\"type\":\"string\"},{\"name\":\"SrcNatPortNumber\",\"type\":\"int\"},{\"name\":\"DvcInboundInterface\",\"type\":\"string\"},{\"name\":\"DvcOutboundInterface\",\"type\":\"string\"},{\"name\":\"NetworkRuleName\",\"type\":\"string\"},{\"name\":\"NetworkRuleNumber\",\"type\":\"int\"},{\"name\":\"ThreatId\",\"type\":\"string\"},{\"name\":\"ThreatName\",\"type\":\"string\"},{\"name\":\"ThreatCategory\",\"type\":\"string\"},{\"name\":\"ThreatRiskLevel\",\"type\":\"int\"},{\"name\":\"ThreatOriginalRiskLevel\",\"type\":\"string\"},{\"name\":\"EventType\",\"type\":\"string\"},{\"name\":\"EventSubType\",\"type\":\"string\"},{\"name\":\"EventResult\",\"type\":\"string\"},{\"name\":\"EventResultDetails\",\"type\":\"string\"},{\"name\":\"EventOriginalType\",\"type\":\"string\"},{\"name\":\"EventProduct\",\"type\":\"string\"},{\"name\":\"EventVendor\",\"type\":\"string\"},{\"name\":\"DvcIpAddr\",\"type\":\"string\"},{\"name\":\"DvcHostname\",\"type\":\"string\"},{\"name\":\"DvcDomain\",\"type\":\"string\"},{\"name\":\"DvcDomainType\",\"type\":\"string\"},{\"name\":\"DvcOs\",\"type\":\"string\"},{\"name\":\"DvcOsVersion\",\"type\":\"string\"},{\"name\":\"AdditionalFields\",\"type\":\"dynamic\"},{\"name\":\"SrcIpAddr\",\"type\":\"string\"},{\"name\":\"SrcPortNumber\",\"type\":\"int\"},{\"name\":\"DstIpAddr\",\"type\":\"string\"},{\"name\":\"NetworkProtocol\",\"type\":\"string\"},{\"name\":\"EventOriginalUid\",\"type\":\"string\"},{\"name\":\"EventReportUrl\",\"type\":\"string\"},{\"name\":\"DvcFQDN\",\"type\":\"string\"},{\"name\":\"DvcId\",\"type\":\"string\"},{\"name\":\"DvcIdType\",\"type\":\"string\"},{\"name\":\"SrcHostname\",\"type\":\"string\"},{\"name\":\"SrcDomain\",\"type\":\"string\"},{\"name\":\"SrcDomainType\",\"type\":\"string\"},{\"name\":\"SrcFQDN\",\"type\":\"string\"},{\"name\":\"SrcDvcId\",\"type\":\"string\"},{\"name\":\"SrcDvcIdType\",\"type\":\"string\"},{\"name\":\"ThreatIpAddr\",\"type\":\"string\"},{\"name\":\"SrcDeviceType\",\"type\":\"string\"},{\"name\":\"SrcUserId\",\"type\":\"string\"},{\"name\":\"SrcUserIdType\",\"type\":\"string\"},{\"name\":\"SrcUsername\",\"type\":\"string\"},{\"name\":\"SrcUsernameType\",\"type\":\"string\"},{\"name\":\"SrcUserType\",\"type\":\"string\"},{\"name\":\"SrcOriginalUserType\",\"type\":\"string\"},{\"name\":\"DstPortNumber\",\"type\":\"int\"},{\"name\":\"DstHostname\",\"type\":\"string\"},{\"name\":\"DstDomain\",\"type\":\"string\"},{\"name\":\"DstDomainType\",\"type\":\"string\"},{\"name\":\"DstFQDN\",\"type\":\"string\"},{\"name\":\"DstDvcId\",\"type\":\"string\"},{\"name\":\"DstDvcIdType\",\"type\":\"string\"},{\"name\":\"DstDeviceType\",\"type\":\"string\"},{\"name\":\"TcpFlagsAck\",\"type\":\"boolean\"},{\"name\":\"TcpFlagsFin\",\"type\":\"boolean\"},{\"name\":\"TcpFlagsPsh\",\"type\":\"boolean\"},{\"name\":\"TcpFlagsRst\",\"type\":\"boolean\"},{\"name\":\"TcpFlagsUrg\",\"type\":\"boolean\"},{\"name\":\"TcpFlagsSyn\",\"type\":\"boolean\"},{\"name\":\"ThreatField\",\"type\":\"string\"},{\"name\":\"ThreatIsActive\",\"type\":\"boolean\"},{\"name\":\"ThreatConfidence\",\"type\":\"int\"},{\"name\":\"NetworkDuration\",\"type\":\"int\"},{\"name\":\"DvcDescription\",\"type\":\"string\"},{\"name\":\"EventOriginalResultDetails\",\"type\":\"string\"},{\"name\":\"ThreatFirstReportedTime\",\"type\":\"datetime\"},{\"name\":\"ThreatLastReportedTime\",\"type\":\"datetime\"},{\"name\":\"ThreatOriginalConfidence\",\"type\":\"string\"},{\"name\":\"DstDescription\",\"type\":\"string\"},{\"name\":\"SrcDescription\",\"type\":\"string\"},{\"name\":\"SourceSystem\",\"type\":\"string\"}]"
}

[anthony-c-martin]

@JustinGrote I think I'll need the whole thing, sorry! I wasn't able to reproduce with just the table you shared.

Again, feel free to email if you're not comfortable sharing here.

[JustinGrote]

@anthony-c-martin I can't reproduce now either but I did change something: here I am supplying to the child module directly, whereas in my initial attempt I had a "main" module that would take the output of the tables and then feed it into the input of the DCR module. So good to know this at least works and I will try to get a reproducible example where you have a "main" module and two "table" and "dcr" modules, and are passing the tableinfo output from one to the other.

[stephaniezyen]

We will look into this more deeply when we get a reproducible file

[microsoft-github-policy-service[bot]]

Hi @JustinGrote, this issue has been marked as stale because it was labeled as requiring author feedback but has not had any activity for 7 days. It will be closed if no further activity occurs within 3 days of this comment. Thanks for contributing to bicep! :smile: :mechanical_arm: