Azure-Functions icon indicating copy to clipboard operation
Azure-Functions copied to clipboard

"409 The specified container already exists."

Open tymtam2 opened this issue 3 years ago • 39 comments

In my Azure Function which doesn't have any storage account code I'm getting the following warning in Application Insights.

Error response [15fd74...ec3601] 409 The specified container already exists. (00.0s)
Server:Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0
x-ms-request-id:b72b1...f5-26564f000000
x-ms-client-request-id:15fd74...0ec3601
x-ms-version:2020-08-04
x-ms-error-code:ContainerAlreadyExists
Date:Mon, 21 Feb 2022 07:33:56 GMT
Content-Length:230
Content-Type:application/xml

I think it's reasonable to assume this is something to do with the storage account that the Azure Functions use to record their state etc.

SDK version azurefunctions: 4.1.3.17473

Is there a way to debug and/or resolve the issue?

tymtam2 avatar Feb 21 '22 07:02 tymtam2

We are experiencing the same issue on some of our v4 functions.

From what we investigated it appears that this warning (or similar one 404 The specified container does not exist.) are being spit by function sdk, but in the same time, by default, they are supressed and if you have default host.json in place, you shouldn't see them.

Our assumption is that we are seeing these warnings because we override default log level. So currently we are just filtering the category Azure.Core with Error severity and it fixes the issue.

Just for reference, our host.json looks like this:

{
  "version": "2.0",
  "logging": {
    "logLevel": {
      "default": "Warning",
      "Custom": "Information",
      "Function": "Information", // dependency telemetry, used to analyzing dependencies and their performance
      "Host.Results": "Information", // request telemetry, used for analyzing execution performance
      "Azure.Core":  "Error" // suppressing sdk blob warnings 
    }
  }
}

domasd avatar Feb 23 '22 09:02 domasd

I'm not sure if this is relevant but all my v4 function apps complain that "Storage is not configured correctly", but they all work.

image

tymtam2 avatar Feb 24 '22 07:02 tymtam2

Hi @pragnagopa , Could you please look into this issue

v-bbalaiagar avatar Feb 24 '22 09:02 v-bbalaiagar

Tagging @AnatoliB @gzuber for triage

pragnagopa avatar Feb 24 '22 20:02 pragnagopa

Just adding more info. The problem seem to be coming in batches of 4 issues:

Error response [06ac509e-...] 409 The specified container already exists. (00.1s)
Server:Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0
x-ms-request-id:e483b8d7-...
x-ms-client-request-id:06ac509e-...
x-ms-version:2020-08-04
x-ms-error-code:ContainerAlreadyExists
Date:Tue, 01 Mar 2022 07:11:44 GMT
Content-Length:230
Content-Type:application/xml

Error response [4c57b299-...] 404 The specified blob does not exist. (00.0s)
Server:Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0
x-ms-request-id:80603de1-...
x-ms-client-request-id:4c57b299-...
x-ms-version:2020-10-02
x-ms-error-code:BlobNotFound
Date:Tue, 01 Mar 2022 07:11:14 GMT
Content-Length:215
Content-Type:application/xml

Error response [9d09446f-...] 404 The specified blob does not exist. (00.0s)
Transfer-Encoding:chunked
Server:Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0
x-ms-request-id:80603da6-...
x-ms-client-request-id:9d09446f-...
x-ms-version:2020-10-02
x-ms-error-code:BlobNotFound
Date:Tue, 01 Mar 2022 07:11:14 GMT

Error response [b873e541-...] 409 The specified container already exists. (00.3s)
Server:Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0
x-ms-request-id:80603cc6-...
x-ms-client-request-id:b873e541-...
x-ms-version:2020-10-02
x-ms-error-code:ContainerAlreadyExists
Date:Tue, 01 Mar 2022 07:11:13 GMT
Content-Length:230
Content-Type:application/xml

tymtam2 avatar Mar 01 '22 07:03 tymtam2

I think Application Insights is just a messenger here. Something in the host process is trying to access a blob, and this attempt is captured, but I don't see any indication that this blob is accessed because of Application Insights.

Does this repro with the simplest "hello world" app with a single HTTP trigger?

AnatoliB avatar Mar 03 '22 01:03 AnatoliB

Category for these isssues is Azure.Core.1.

As I was setting up trying the out-of-the-box http trigger function I realised that this issue is only reported by the functions with [EventHubTrigger]. I can see now that the functions that only have a http trigger do not report these 4 issues.

tymtam2 avatar Mar 03 '22 02:03 tymtam2

@tymtam2 Do you know which version of the Event Hubs extension you are using?

AnatoliB avatar Mar 05 '22 01:03 AnatoliB

Here are all the referenced packages

<PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
    <AzureFunctionsVersion>v4</AzureFunctionsVersion>
    <Nullable>enable</Nullable>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Dapper" Version="2.0.123" />
    <PackageReference Include="Microsoft.ApplicationInsights" Version="2.20.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventHubs" Version="5.0.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.1.0" />
    <PackageReference Include="Microsoft.CodeAnalysis.NetAnalyzers" Version="6.0.0">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="4.1.0" />
    <PackageReference Include="StyleCop.Analyzers" Version="1.2.0-beta.406">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="System.Data.SqlClient" Version="4.8.3" />
  </ItemGroup>

I get the 4 issues when running locally (not just when deployed) and in the bin folder the version of Azure.Messaging.EventHubs.dll is 5.6.2 and Microsoft.Azure.WebJobs.Extensions.EventHubs.dll is 5.0.0.

As an experiment I added an explicit dependency on Azure.Messaging.EventHubs 5.7.0-beta.3 but after doing that the function reports on startup: Microsoft.Azure.WebJobs.Host: Error indexing method '(name_here)'. Microsoft.Azure.WebJobs.Host: Multiple properties named 'ContentType' found in type 'EventData'..

tymtam2 avatar Mar 07 '22 06:03 tymtam2

@alrod Track2 extension issue?

AnatoliB avatar Mar 07 '22 07:03 AnatoliB

@tymtam2, can you please try to use "Microsoft.Azure.WebJobs.Extensions.EventHubs" Version="4.3.1"? do you see the "409 The specified container already exists." in this case?

alrod avatar Mar 09 '22 00:03 alrod

I've tried "Microsoft.Azure.WebJobs.Extensions.EventHubs" Version="4.3.1" and I don't see the warnings.

tymtam2 avatar Mar 09 '22 09:03 tymtam2

This occurs with [ServiceBusTrigger] as well

GuiGimenez avatar Apr 13 '22 01:04 GuiGimenez

I had this bug

Multiple properties named 'ContentType' found in type 'EventData'.

Problem: I had Microsoft.Azure.WebJobs.Extensions.EventHubs 4.2 that comes with VS template when creating function app (iothub trigger). It does not work. I had to install Azure.Messaging.EventHubs.Processor. Then it works just fine. But if I update the Microsoft.Azure.WebJobs.Extensions.EventHubs to latest version then it complains about Microsoft.Azure.EventHubs which is deprecated and needed to be replaced with with Azure.Messaging.EventHubs. Finally you have produced the error.

Fix: I have not updated the Microsoft.Azure.WebJobs.Extensions.EventHubs 4.2 and installed Azure.Messaging.EventHubs.Processor. Now its behaving properly.

Concern: I am sure this is not a proper fix but I could not solve it any other way. Please share your solution if you find any. The Microsoft.Azure.EventHubs is deprecated. It seems to be Azure.Messaging.EventHubs is buggy.

mail4hafij avatar May 11 '22 14:05 mail4hafij

Any resolution on this. Using VS 2022 MAC RC2 on M1. Worked fine until Microsoft.Azure.Webjobs.Extensions.EventHubs updated to 5.1.0. Rolling back however does not resolve the issue. This is specifically an issue with the event hub trigger in Azure functions ~4 / .NET 6.

[2022-05-12T17:32:08.551Z] Microsoft.Azure.WebJobs.Host: Error indexing method '<FunctionName>'. Microsoft.Azure.WebJobs.Host: Multiple properties named 'ContentType' found in type 'EventData'. [2022-05-12T17:32:08.600Z] Error indexing method '<FunctionName>' [2022-05-12T17:32:08.600Z] Microsoft.Azure.WebJobs.Host: Error indexing method '<FunctionName>'. Microsoft.Azure.WebJobs.Host: Multiple properties named 'ContentType' found in type 'EventData'. [2022-05-12T17:32:08.601Z] Function '<FunctionName>' failed indexing and will be disabled.

When deployed to Azure the functions load and execute properly. This issue appears to be limited to the Visual Studio environment only.

jawinnerjr avatar May 12 '22 17:05 jawinnerjr

I'm also seeing this warning on startup with [TimerTrigger] on .NET 6 with dotnet-isolated.

[2022-05-25T10:17:21.413Z] Error response [26c59ccf-e824-4fb0-a586-9cd340504cba] 409 The specified container already exists. (00.0s)
[2022-05-25T10:17:21.417Z] Server:Azurite-Blob/3.14.1
[2022-05-25T10:17:21.419Z] x-ms-error-code:ContainerAlreadyExists
[2022-05-25T10:17:21.420Z] x-ms-request-id:277e9701-a041-4f85-802d-234f3d0c8caa
[2022-05-25T10:17:21.421Z] Date:Wed, 25 May 2022 10:17:21 GMT
[2022-05-25T10:17:21.423Z] Connection:keep-alive
[2022-05-25T10:17:21.424Z] Keep-Alive:REDACTED
[2022-05-25T10:17:21.426Z] Transfer-Encoding:chunked
[2022-05-25T10:17:21.427Z] Content-Type:application/xml

Seems to me like the Core is trying to do a CreateIfNotExists somewhere internally or something because TimerTrigger also uses blob storage under the hood, and that is shown in our logs.

The timers run fine by the way.

wkoeter avatar May 25 '22 10:05 wkoeter

No doubt a manifestation of this issue, which hasn't been fixed for 10 years 😖 Container.CreateIfNotExist throws exception if container exists

If anyone gets assigned to this (and I hope someone does), make sure to check if the container exists before trying to create it. CreateIfNotExist throws an exception if the container exists. If that sounds absurd, it's because it is.

michaeldaw avatar Jun 18 '22 04:06 michaeldaw

I'm getting this error after upgrading a project/host from .NET Core 3.1 / AF 3 to .NET 6 / AF 4. I'm not using any event* triggers, only http/timer/durableTask.

klenium avatar Jul 17 '22 11:07 klenium

My http triggers work fine but my event hub triggers are all failing with the errors described in this thread.

net6 func v4 Microsoft.Azure.Functions.Worker.Extensions.EventHubs 5.1.0

brennfoster avatar Jul 21 '22 15:07 brennfoster

I just realized that I was not awaiting the host.RunAsync() command in Program.cs and that's why this was failing for me.

brennfoster avatar Aug 17 '22 18:08 brennfoster

is there any update on this issue? I am facing the same issue while using net6 and function app v4. Thanks

xsurfer avatar Oct 25 '22 23:10 xsurfer

This occurs with [ServiceBusTrigger] after upgrading from .NET Core 3.1 / AF 3 to .NET 6 / AF 4.

SaveliyKolesnikov avatar Oct 27 '22 14:10 SaveliyKolesnikov