Function App returns 202 when incorrect function name configured as Event Grid webhook - should return error code instead

[rodolfograve]

Setup

Blob Storage trigger using Source=EventGrid (as per https://learn.microsoft.com/en-us/azure/azure-functions/functions-event-grid-blob-trigger)

This setup requires an Event Grid Subscription using an Endpoint Type = Webhook, and the URL must be in the following format: https://<FUNCTION_APP_NAME>.azurewebsites.net/runtime/webhooks/blobs?functionName=<TARGET_FUNCTION_NAME>&code=<BLOB_EXTENSION_KEY

Issue Summary

If the <TARGET_FUNCTION_NAME> is wrong, i.e. there is no function with that name, the Function App returns 202 to Event Grid, which causes Event Grid to consider the event delivered. However, since there is no function with that name, the Function App logs a DEBUG-level log and drops the event.

Consequences

We have irrecoverable and undetected lost events. Event Grid does not retry, and worse, does not apply any dead-lettering policies to the event.

We are also not able to see this error in the logs using the default configuration.

Via a Microsoft Support ticket (#2206230050002214) we have been told to set the logging level to DEBUG so that we can pick the "error" message from the Function App, but even though that would help us with spotting the issue as soon as possible, it still does not address the underlying issue and its consequences. Critically, we lose the ability to retry any lost events.

Investigative information

• Timestamp: September 29th, 2022, 18:00 (cannot see exact timestamp via the Event Grid metrics)
• Function App version: ~4
• Function App name: nemsdevwebappd0anonfn
• Function name(s) (as appropriate): N/A (any name that does not map to an existing function)
• Invocation ID: 9fb4acad-a569-4ecb-b76c-30685e6c22a7
• Region: UK South

Repro steps

1. Create an Event Grid Subscription using a Webhook.
2. Use the following format for the Webhook URL, making sure the Function App exists but not the function: https://<FUNCTION_APP_NAME>.azurewebsites.net/runtime/webhooks/blobs?functionName=<TARGET_FUNCTION_NAME>&code=<BLOB_EXTENSION_KEY
3. Trigger the Event Grid event so that Event Grid attempts to deliver the event to your non-existing function.

Expected behavior

The Function App should:

1. Return some error code (404 seems natural given that the target function was not found). This will allow Event Grid to realise the event was not delivered and trigger all its retry + dead-lettering mechanisms.
2. Log an ERROR-level message.

Actual behavior

1. Event Grid reports the event as delivered.
2. The event is not processed and it's lost.
3. There is no error log in the Function App.

There seems to be a DEBUG log entry, which is not visible using a Function App's recommended log configuration:

Known workarounds

None

Related information

• Programming language used: C#
• Triggers used: BlobTrigger(Source = BlobTriggerSource.EventGrid)

[rodolfograve]

Hello. Is this a valid channel to report this? After 8 days I haven't received even an acknowledgement.

[rodolfograve]

14 days and counting. Is there any other way to contact this team?

We are a paying customer and this level of support feels more appropriate for free software than a profitable company like Microsoft.

We already went through Microsoft's/Azure support channel and were sent to this GitHub repository as the only way to engage with the Product Team.

[rodolfograve]

I was hoping that after the .NET 7 GA someone would pick this ticket up?

[rodolfograve]

Echoooooooo

[fabiocav]

@rodolfograve apologies for the delayed response here.

For issues like this, if you're seeing production impact, and since you have access to support, using that channel would ensure the issue is handled in a timely manner.

Are you still experiencing this?

[rodolfograve]

Yes, unfortunately the situation hasn't changed. 7 months on and still experiencing the same behaviour.

At the time I raised a support request and spent weeks going back and forth with the support team. The outcome was "please, create a ticket in this GitHub project".

There is an obvious issue with the way problems are dealt with in Azure Functions, and I hope that by it being so obvious it can be fixed.

Or maybe I just have bad luck :-)