Microsoft Exchange Connector - Message Tracking is broken

[FormindMPO]

Describe the bug

Error in the [Option 6] Message Tracking of Exchange Servers from the Microsoft Exchange Logs and Events Connector. The manual DCR deployement can't be done.

To Reproduce

1. Install Message Tracking of Exchange Servers Connector
2. Create a DCE
3. Try to create the DCR
4. Get the ESI-MessageTrackingLogs.json DCR-based tempalte
5. See error (1)
6. Transform date-time to TimeGenerated fields
7. See error (2)

Expected behavior

Successful creation

Screenshots

Error (1) :

Error (2) :

Desktop (please complete the following information):

• OS: Windows
• Browser: Firefox
• Version 122

Additional context Add any other context about the problem here.

[github-actions[bot]]

Thank you for submitting an Issue to the Azure Sentinel GitHub repo! You should expect an initial response to your Issue from the team within 5 business days. Note that this response may be delayed during holiday periods. For urgent, production-affecting issues please raise a support ticket via the Azure Portal.

[v-muuppugund]

Hi @FormindMPO , Thanks for flagging this issue, we will investigate this issue and get back to you with some updates by 06/02/2024. Thanks!

[v-muuppugund]

Hi @FormindMPO , Still need some more time for replicating the issue with detailed analysis ,will update you

[v-muuppugund]

Hi @FormindMPO ,I am able to replicate the issue and working on further analysis for fix,please find below screen shot for reference

[v-muuppugund]

Hi @FormindMPO ,Working on data changes,once done,will test and update you

[samet-ibis]

Hi @v-muuppugund, does it also block data ingestion from Exchange servers with ARM template? I faced the same issue with manual deployment, then I tried with ARM template, but it's not ingesting any data from Exchange.

[v-muuppugund]

Hi @t0neex ,this is different issue,Could you please share more details,what issue you are facing,where and when it happened? with screen shots.

[samet-ibis]

Hi @v-muuppugund, thanks for your response, actually it seems same issue.

I tried to create MessageTrackingLog as like issue reporter FormindMPO and I faced same issue. Then i tried with Option 1 - Azure Resource Manager ARM Template option and template deployed succesfully but its not ingesting any data to my table.

If the ESI-MessageTrackingLogs.json file is also being applied in the ARM template, then the template should be changed with the new JSON file you will fix.

[v-muuppugund]

Hi @v-muuppugund, thanks for your response, actually it seems same issue.

I tried to create MessageTrackingLog as like issue reporter FormindMPO and I faced same issue. Then i tried with Option 1 - Azure Resource Manager ARM Template option and template deployed succesfully but its not ingesting any data to my table.

If the ESI-MessageTrackingLogs.json file is also being applied in the ARM template, then the template should be changed with the new JSON file you will fix.

Sure @t0neex ,Please let me know convenient time slots for teams meeting for option 1 issue to this email id i.e. [email protected]

[v-muuppugund]

Hi @t0neex ,As discussed on Monday the issue is due to permissions could you please check and let us know if you still facing any issues.

[v-muuppugund]

Hi @FormindMPO ,Working on data changes,once done,will test and update you

Hi @FormindMPO ,Could you please try option 1 as work around and still need some more time on further data changes

[v-muuppugund]

Hi @FormindMPO ,Gentle reminder ,Could you please try option 1 once and let me know if any issues and working on option 6 steps needs to be done,will update you

[FormindMPO]

Hello @v-muuppugund, sorry I worked around this issue by using custom DCR I raised this issue to fix implem and refresh documentation

[v-muuppugund]

Hello @v-muuppugund, sorry I worked around this issue by using custom DCR I raised this issue to fix implem and refresh documentation

Sure @FormindMPO ,Working on it, meanwhile suggested alternate option for workaround.

[thom2804]

Any status updates on this issue? Also running into the same issue, if someone has a fix or details about a workaround that would be appreciated.

[v-muuppugund]

Any status updates on this issue? Also running into the same issue, if someone has a fix or details about a workaround that would be appreciated.

Hi @thom2804 , Working on it ,will update you, As a work around could you please check option 6

[thom2804]

Any status updates on this issue? Also running into the same issue, if someone has a fix or details about a workaround that would be appreciated.

Hi @thom2804 , Working on it ,will update you, As a work around could you please check option 6

Option 6 is the part of the steps where I am running into the issues, all the other options work fine, I am just unable to get the Transform action with the provided KQL query to work on the Custom Text log data source. I have tried both the manual and the automatic version.

Query which is giving issues: source| extend TimeGenerated = todatetime(['date-time'])| extend clientHostname = ['client-hostname'], clientIP = ['client-ip'], connectorId = ['connector-id'], customData = ['custom-data'], eventId = ['event-id'], internalMessageId = ['internal-message-id'], logId = ['log-id'], messageId = ['message-id'], messageInfo = ['message-info'], messageSubject = ['message-subject'], networkMessageId = ['network-message-id'], originalClientIp = ['original-client-ip'], originalServerIp = ['original-server-ip'], recipientAddress= ['recipient-address'], recipientCount= ['recipient-count'], recipientStatus= ['recipient-status'], relatedRecipientAddress= ['related-recipient-address'], returnPath= ['return-path'], senderAddress= ['sender-address'], senderHostname= ['server-hostname'], serverIp= ['server-ip'], sourceContext= ['source-context'], schemaVersion=['schema-version'], messageTrackingTenantId = ['tenant-id'], totalBytes = ['total-bytes'], transportTrafficType = ['transport-traffic-type']| project-away ['client-ip'], ['client-hostname'], ['connector-id'], ['custom-data'], ['date-time'], ['event-id'], ['internal-message-id'], ['log-id'], ['message-id'], ['message-info'], ['message-subject'], ['network-message-id'], ['original-client-ip'], ['original-server-ip'], ['recipient-address'], ['recipient-count'], ['recipient-status'], ['related-recipient-address'], ['return-path'], ['sender-address'], ['server-hostname'], ['server-ip'], ['source-context'], ['schema-version'], ['tenant-id'], ['total-bytes'], ['transport-traffic-type']

Error I am getting: Update Error - Error occurred while compiling query in query: SemanticError:0x00000006 at 2:36 : Undefined symbol: date-time

[v-muuppugund]

Any status updates on this issue? Also running into the same issue, if someone has a fix or details about a workaround that would be appreciated.

Hi @thom2804 , Working on it ,will update you, As a work around could you please check option 6

Option 6 is the part of the steps where I am running into the issues, all the other options work fine, I am just unable to get the Transform action with the provided KQL query to work on the Custom Text log data source. I have tried both the manual and the automatic version.

Query which is giving issues: source| extend TimeGenerated = todatetime(['date-time'])| extend clientHostname = ['client-hostname'], clientIP = ['client-ip'], connectorId = ['connector-id'], customData = ['custom-data'], eventId = ['event-id'], internalMessageId = ['internal-message-id'], logId = ['log-id'], messageId = ['message-id'], messageInfo = ['message-info'], messageSubject = ['message-subject'], networkMessageId = ['network-message-id'], originalClientIp = ['original-client-ip'], originalServerIp = ['original-server-ip'], recipientAddress= ['recipient-address'], recipientCount= ['recipient-count'], recipientStatus= ['recipient-status'], relatedRecipientAddress= ['related-recipient-address'], returnPath= ['return-path'], senderAddress= ['sender-address'], senderHostname= ['server-hostname'], serverIp= ['server-ip'], sourceContext= ['source-context'], schemaVersion=['schema-version'], messageTrackingTenantId = ['tenant-id'], totalBytes = ['total-bytes'], transportTrafficType = ['transport-traffic-type']| project-away ['client-ip'], ['client-hostname'], ['connector-id'], ['custom-data'], ['date-time'], ['event-id'], ['internal-message-id'], ['log-id'], ['message-id'], ['message-info'], ['message-subject'], ['network-message-id'], ['original-client-ip'], ['original-server-ip'], ['recipient-address'], ['recipient-count'], ['recipient-status'], ['related-recipient-address'], ['return-path'], ['sender-address'], ['server-hostname'], ['server-ip'], ['source-context'], ['schema-version'], ['tenant-id'], ['total-bytes'], ['transport-traffic-type']

Error I am getting: Update Error - Error occurred while compiling query in query: SemanticError:0x00000006 at 2:36 : Undefined symbol: date-time

Sorry option 6 fix i am working ,Could you please try other options,For option 6 ,i am working on it,will update you

[thom2804]

Any status updates on this issue? Also running into the same issue, if someone has a fix or details about a workaround that would be appreciated.

Hi @thom2804 , Working on it ,will update you, As a work around could you please check option 6

Option 6 is the part of the steps where I am running into the issues, all the other options work fine, I am just unable to get the Transform action with the provided KQL query to work on the Custom Text log data source. I have tried both the manual and the automatic version. Query which is giving issues: source| extend TimeGenerated = todatetime(['date-time'])| extend clientHostname = ['client-hostname'], clientIP = ['client-ip'], connectorId = ['connector-id'], customData = ['custom-data'], eventId = ['event-id'], internalMessageId = ['internal-message-id'], logId = ['log-id'], messageId = ['message-id'], messageInfo = ['message-info'], messageSubject = ['message-subject'], networkMessageId = ['network-message-id'], originalClientIp = ['original-client-ip'], originalServerIp = ['original-server-ip'], recipientAddress= ['recipient-address'], recipientCount= ['recipient-count'], recipientStatus= ['recipient-status'], relatedRecipientAddress= ['related-recipient-address'], returnPath= ['return-path'], senderAddress= ['sender-address'], senderHostname= ['server-hostname'], serverIp= ['server-ip'], sourceContext= ['source-context'], schemaVersion=['schema-version'], messageTrackingTenantId = ['tenant-id'], totalBytes = ['total-bytes'], transportTrafficType = ['transport-traffic-type']| project-away ['client-ip'], ['client-hostname'], ['connector-id'], ['custom-data'], ['date-time'], ['event-id'], ['internal-message-id'], ['log-id'], ['message-id'], ['message-info'], ['message-subject'], ['network-message-id'], ['original-client-ip'], ['original-server-ip'], ['recipient-address'], ['recipient-count'], ['recipient-status'], ['related-recipient-address'], ['return-path'], ['sender-address'], ['server-hostname'], ['server-ip'], ['source-context'], ['schema-version'], ['tenant-id'], ['total-bytes'], ['transport-traffic-type'] Error I am getting: Update Error - Error occurred while compiling query in query: SemanticError:0x00000006 at 2:36 : Undefined symbol: date-time

Sorry option 6 fix i am working ,Could you please try other options,For option 6 ,i am working on it,will update you

No problem, the oher options are already configured and work properly, just waiting on a fix for option 6.

[slivoski]

Also ran into this issue. Option 1 (ARM template) does deploy, but if you try to modify the DCR you get the same error as manually creating the DCR and transformation 'Undefined symbol: date-time'.

Logs do not ingest, likely due to the transform error.

[v-muuppugund]

Also ran into this issue. Option 1 (ARM template) does deploy, but if you try to modify the DCR you get the same error as manually creating the DCR and transformation 'Undefined symbol: date-time'.

Logs do not ingest, likely due to the transform error.

Checked on it,Will update you

[v-muuppugund]

Hi @FormindMPO / @slivoski / @thom2804 / @samet-ibis ,Apologies for delayed response,Could you please update the following query in transformation editor during custom dcr source | extend TimeGenerated = todatetime(['date-time']) | extend clientHostname = ['client-hostname'], clientIP = ['client-ip'], connectorId = ['connector-id'], customData = ['custom-data'], eventId = ['event-id'], internalMessageId = ['internal-message-id'], logId = ['log-id'], messageId = ['message-id'], messageInfo = ['message-info'], messageSubject = ['message-subject'], networkMessageId = ['network-message-id'], originalClientIp = ['original-client-ip'], originalServerIp = ['original-server-ip'], recipientAddress= ['recipient-address'], recipientCount= ['recipient-count'], recipientStatus= ['recipient-status'], relatedRecipientAddress= ['related-recipient-address'], returnPath= ['return-path'], senderAddress= ['sender-address'], senderHostname= ['server-hostname'], serverIp= ['server-ip'], sourceContext= ['source-context'], schemaVersion=['schema-version'], messageTrackingTenantId = ['tenant-id'], totalBytes = ['total-bytes'], transportTrafficType = ['transport-traffic-type'] | project-away ['client-ip'], ['client-hostname'], ['connector-id'], ['custom-data'], ['date-time'], ['event-id'], ['internal-message-id'], ['log-id'], ['message-id'], ['message-info'], ['message-subject'], ['network-message-id'], ['original-client-ip'], ['original-server-ip'], ['recipient-address'], ['recipient-count'], ['recipient-status'], ['related-recipient-address'], ['return-path'], ['sender-address'], ['server-hostname'], ['server-ip'], ['source-context'], ['schema-version'], ['tenant-id'], ['total-bytes'], ['transport-traffic-type']

Please find below screen shot after updated query successfully created ,please let me know if any issues

[v-muuppugund]

Hi @FormindMPO / @slivoski / @thom2804 / @samet-ibis PR has been raised for the Option 6,Please find below PR link (https://github.com/Azure/Azure-Sentinel/pull/10274) as per our standard operating procedures. If you still need support for this issue(https://github.com/Azure/Azure-Sentinel/issues/9862), feel free to re-open at any time. Thank you for your co-operation!

[v-muuppugund]

Hi @FormindMPO / @slivoski / @thom2804 / @samet-ibis, we have raised the PR with the enhancements. The changes will be reflected once the PR is merged. PR link - https://github.com/Azure/Azure-Sentinel/pull/10274

Thank you for your cooperation.