[Cosmos DB Trigger] Null _ts, _etag etc. properties from IReadOnlyList parameters.

[andersson09]

Hi,

It appears that I can't read _ts, _etag etc. properties from the change feed, even though I have them set in my entity like so:

[JsonConverter(typeof(UnixDateTimeConverter))]
[JsonProperty("_ts")]
public DateTime? LastModified {get;set;}

[JsonProperty("_etag")]
public string? Etag {get;set;}

I need this properties for processing my dead letter queue.

Thanks

Edit: This started working when I switch to the system text json custom serializer.

[Ved2806]

Hi @ealsur Could you please help with this?

[ealsur]

@andersson09 Which version of the extension are you using? Can you provide a sample code of the Trigger Function signature? Any repro that you can provide?

[ealsur]

You mention that they started working when you switched to System.Text.Json - This probably means the JsonProperty you were using were from System.Text.Json namespace?

[ealsur]

The trigger always receives and passes these properties, custom serialization won't add or remove them from what the trigger delivers, you can always use a JObject or JsonDocument (if you use System.Text.Json as the serializer) to verify the raw JSON.

[andersson09]

System.Text.Json uses JsonPropertyName so I don't think it was this. Yes the trigger passes the json as a string successfully into InputData of IFunctionsBindingsFeature. However, it wouldn't bind to my model. The extension version I was using is 3.0.9.

I'll try reproducing it again soon since I haven't been on this code for a while. Tbh I'm happy with System.Text.Json so may be unnecessary to leave this open, especially if Newtonsoft will be discontinued.

Fyi there is another separate issue though with System.Text.Json and GetItemLinqQueryable. The linq serializer options allows you to set CamelCase but Ignores any System.Text.Json attributes. Therefore I need to write code such as this which isn't great:

    [Newtonsoft.Json.JsonProperty("_ts")]
    [JsonConverter(typeof(UnixDateTimeConverter))]
    [JsonPropertyName("_ts")]
    public DateTime? LastModified { get; set; }

    [Newtonsoft.Json.JsonProperty("_etag")]
    [JsonPropertyName("_etag")]
    public string? Etag { get; set; }

I believe the issue is in here https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/Linq/CosmosLinqQuery.cs

[ealsur]

Extension 3.0.9 uses the V2 SDK and the only type you can bind the trigger is Document. It is always IReadOnlyList<Document> on the [CosmosDBTrigger] binding.

That would mean you were taking the Document and then converting by some means to your type

Extension 4.X is the one that allows you to use a custom type on the Trigger to bind to a type of your choice: IReadOnlyList<T>.

The code you linked has nothing to do with extension 3.0.9 because as mentioned, that version uses the old V2 SDK and the link you shared is from the V3 SDK. The 4.X extension uses the V3 SDK but the code you linked is from Linq query support, not used on the Trigger.

Could you please provide a repro using 3.0.9?

[ealsur]

@alrod - This is not a Track2 SDK issue, this is a report on the old SDK (V2)

[ealsur]

@andersson09 - Could you please share a full example of a [CosmosDBTrigger] using 3.0.9 where you are not getting the _ts?

[andersson09]

Nugets:

<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.CosmosDB" Version="3.0.9" />

Trigger:

[Function("DoSomething")]
public async Task DoSomething(
    [CosmosDBTrigger("%DbName%", "container", 
    ConnectionStringSetting = "CosmosDbConnectionString", 
    LeaseCollectionName = "leases",
    CreateLeaseCollectionIfNotExists = true)] IReadOnlyList<Entity> entities)
{
}

Payload: .../admin/functions/DoSomething

{
  "input": "[{\"test\":\"hello\"}]"
}

Works: Returns "hello"

public class Entity
{    
    public string? test { get; set; }
}

public class Entity
{    
    [System.Text.Json.Serialization.JsonPropertyName("test")]
    public string? Testing {get;set;}
}

Doesn't work: Returns null

public class Entity
{    
    [Newtonsoft.Json.JsonProperty("test")]
    public string? Testing {get;set;}
}

Hypothesis:

So it looks like the serializer is ignoring Newtonsoft JsonProperty attribute and nothing to do with the system fields (_ts, _etag). Or it's just a case that the extension's default serializer is system.text.json and the cosmos sdks's is newtonsoft.

Regarding

...but the code you linked is from Linq query support, not used on the Trigger.

I was just referring to a separate issue (nothing to do with this ticket) since I remembered it whilst I was typing.

Hope this helps.

[ealsur]

3.X Extension uses IReadOnlyList<Document> which is what is shown in all the docs and examples: https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-cosmosdb-v2-trigger?tabs=in-process%2Cfunctionsv2&pivots=programming-language-csharp#configuration

Using any other type is not supported nor an expected scenario.

4.X extension does support custom types with even custom serialization (you can use System.Text.Json if you want).

The repro code you are sharing does not work, there is a validation on the 3.X extension that fails if you use anything other that Document:

[2022-05-25T14:59:25.206Z] The 'Function1' function is in error: Microsoft.Azure.WebJobs.Host: Error indexing method 'Function1'. Microsoft.Azure.WebJobs.Host: Can't bind CosmosDBTrigger to type 'System.Collections.Generic.IReadOnlyList`1[OldFunctions.Function1+Entity]'.

[andersson09]

I'm using Azure functions v4 isolated, but with the 3.0.9 cosmos nuget. Does that change things?

<AzureFunctionsVersion>v4</AzureFunctionsVersion>

[ealsur]

The package is still the same, the check is in the source code: https://github.com/Azure/azure-webjobs-sdk-extensions/blob/cosmos/v3.x/src/WebJobs.Extensions.CosmosDB/Config/CosmosDBExtensionConfigProvider.cs#L74, the binding is only done to IReadOnlyList<Document>.

Does your Entity in your code inherit from Document, like public class Entity: Document ?

[andersson09]

Really strange. It works for me. No the entity in the example doesn't inherit from anything. I'm also bringing in the cosmos client. But I don't think that changes anything.

<PackageReference Include="Microsoft.Azure.Cosmos" Version="3.26.1" />

[ealsur]

That client is not related to the 3.0.9 extension, it is related to the 4.0.0 extension:

https://www.nuget.org/packages/Microsoft.Azure.WebJobs.Extensions.CosmosDB/3.0.9

https://www.nuget.org/packages/Microsoft.Azure.WebJobs.Extensions.CosmosDB/4.0.0-preview3

Are you really sure you are not mixing Extension versions in the same project?

[andersson09]

You're pointing to a different nuget. This is the one I am using

https://www.nuget.org/packages/Microsoft.Azure.Functions.Worker.Extensions.CosmosDB/3.0.9

Note Worker and not WebJobs

[ealsur]

The 2 Nugets I mention are the ones I authored, I'm not sure what the Worker version of the Nuget includes (the Dependencies section is empty) but looking at the Versions, it's a mirror with the other ones.

So assuming that, it should work the same way (3.X uses the V2 SDK and Document, 4.X uses V3 SDK and can use your custom <T>), but since I didn't author or work on those, I cannot confirm.

@Ved2806 - I cannot comment on the Worker packages as I don't know who authored them or what they contain. I tried to repro the scenario with the packages we author and the scenario is simply not possible (using Entity instead of Document with version 3.0.9). Whoever owns the Worker packages might be able to comment on what do they actually include and work.

[andersson09]

I think the isolated model uses worker packages. Maybe there has been a mix up with the cosmos extensions versions.

[Ved2806]

Hi @andersson09 Is it resolved for you?

[ghost]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.