Microsoft.Data.Services.Client v5.8.5 failes on ValidateNavigationPropertyDefined

[MartinKPal]

Describe the bug

Microsoft.Data.Services.Client v5.8.5 fails on parsing OData response when container app starts under high parallel load with error message "A property with name 'to_BuPaIdentification' on type '{our namesapce}.A_BusinessPartnerType' has kind 'None', but it is expected to be of kind 'Navigation'." Initial container app replica behaves normally, based on scaling rules new replicas are created and these have the issue. This is similar to https://learn.microsoft.com/en-us/answers/questions/922234/navigation-property-error-between-two-azure-sql-ta?source=docs

This issue is not happening with version 5.8.4.

Assemblies affected

Microsoft.Data.Services.Client version 5.8.5 This library is used in .NET 9 AspNET WebAPI project.

Steps to Reproduce

I don't have exact steps to reproduce, but high level steps are:

1. create a new ASP.NET Core Web API project using .NET 9
2. Add WCF reference to ODATA resource (in our case, SAP BusinessPartner endpoint, which is still on ODATA V2, https://api.sap.com/products/SAPS4HANACloud/apis/ODATA)
3. create REST endpoint that calls ODATA endpoint
4. deploy as container app with scaling (for example based on number of parallel http requests)
5. Loadscript

import http from 'k6/http';
import { check } from 'k6';
import { Rate } from 'k6/metrics';

export const errorRate = new Rate('errors');

// execute by running "k6 run script.js -d 5m -u 100"
export default function () {
    const params = {
        headers: {
            'Authorization': 'Bearer  ...',
        },
    };

    check(http.get('https://rest-endpoint-url', params), {
        'status is 200': (r) => r.status == 200,
    }) || errorRate.add(1);
}

6. execute load script - multiple replicas are running

7. observe errors like ---> Microsoft.Data.OData.ODataException: A property with name 'to_BuPaIdentification' on type '{our namespace}e.A_BusinessPartnerType' has kind 'None', but it is expected to be of kind 'Navigation'. at Microsoft.Data.OData.ReaderValidationUtils.ValidateNavigationPropertyDefined(String propertyName, IEdmEntityType owningEntityType, ODataMessageReaderSettings messageReaderSettings) at Microsoft.Data.OData.Atom.ODataAtomEntryAndFeedDeserializer.TryReadNavigationLinkInEntry(IODataAtomReaderEntryState entryState, String linkRelation, String linkHRef) at Microsoft.Data.OData.Atom.ODataAtomEntryAndFeedDeserializer.ReadAtomLinkElementInEntry(IODataAtomReaderEntryState entryState) at Microsoft.Data.OData.Atom.ODataAtomEntryAndFeedDeserializer.ReadAtomElementInEntry(IODataAtomReaderEntryState entryState) at Microsoft.Data.OData.Atom.ODataAtomEntryAndFeedDeserializer.ReadEntryContent(IODataAtomReaderEntryState entryState) at Microsoft.Data.OData.Atom.ODataAtomReader.ReadEntryStart() at Microsoft.Data.OData.Atom.ODataAtomReader.ReadAtFeedStartImplementation() at Microsoft.Data.OData.ODataReaderCore.ReadImplementation() at Microsoft.Data.OData.ODataReaderCore.ReadSynchronously() at Microsoft.Data.OData.ODataReaderCore.InterceptException[T](Func1 action) at Microsoft.Data.OData.ODataReaderCore.Read() at System.Data.Services.Client.Materialization.ODataReaderWrapper.Read() at System.Data.Services.Client.Materialization.FeedAndEntryMaterializerAdapter.TryRead() --- End of inner exception stack trace --- at System.Data.Services.Client.Materialization.FeedAndEntryMaterializerAdapter.TryRead() at System.Data.Services.Client.Materialization.FeedAndEntryMaterializerAdapter.TryStartReadFeedOrEntry() at System.Data.Services.Client.Materialization.FeedAndEntryMaterializerAdapter.TryReadEntry(MaterializerEntry& entry) at System.Data.Services.Client.Materialization.FeedAndEntryMaterializerAdapter.<LazyReadEntries>d__0.MoveNext() at System.Data.Services.Client.Materialization.FeedAndEntryMaterializerAdapter.Read() at System.Data.Services.Client.Materialization.ODataReaderEntityMaterializer.ReadNextFeedOrEntry() at System.Data.Services.Client.Materialization.ODataEntityMaterializer.ReadImplementation() at System.Data.Services.Client.Materialization.ODataMaterializer.Read() at System.Data.Services.Client.MaterializeAtom.MoveNextInternal() at System.Data.Services.Client.MaterializeAtom.MoveNext() at System.Linq.Enumerable.CastIterator[TResult](IEnumerable source)+MoveNext() at System.Linq.Enumerable.TryGetFirstNonIterator[TSource](IEnumerable1 source, Boolean& found) at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable1 source) at {our namespace}.ODataCore.Commons.Extensions.DataServiceQueryableExtensions.FirstOrDefaultAsync[TSource](IQueryable1 source, CancellationToken cancellationToken) at {our namespace}.Microservice.BPM.SapOData.Services.SapBusinessPartnerService.GetBusinessPartnerByNumberAsync(String sapNumber, SapNumberType sapNumberType, Boolean includeArchived, CancellationToken cancellationToken)

8. call https://rest-endpoint-url manually few times and observe that some of the responses return 200 and some 500, depending on which container app replica serve them.

NOTE: when container app replicas were manually scaled to maximum (in our case 5 replicas) upfront, then this issue was not observed. Seems like the problem is at startup together with high parallel load.

Expected behaviour

Parsing of ODATA response from Microsoft.Data.Services.Client library always works the same way, also when called with high parallel load.

Actual behaviour

Parsing of ODATA response, is different when Web API is created and asked to serve a high parallel load of requests.

Additional details

Raw ODATA response have some sensitive data, but it can be provided to Microsoft support if needed.

[habbes]

@MartinKPal This looks like a race condition, perhaps a code path in the client that does not have proper thread safety, that could explain why it's sporadic and appears only on high parallel load. One possible causes of race conditions is using a singleton DataServiceContext. If used in a service, it may be beneficial to create a DataServiceContext per request as scoped service.

But it could also be a bug in the client code that hides a race condition. We've had cases in the 7.x+ client where we observed a similar pattern and found a bug in some locking code. This may or may not be related to that: https://github.com/OData/odata.net/issues/2532, https://github.com/OData/odata.net/pull/2533

Unfortunately, the 5.x client is not under active support no new changes are planned. Are you able to migrate to the 8.x version or v4 API?

[uffelauesen]

@MartinKPal @habbes I think this is the same issue I reported back in 2022 here: #2452 /uffe

[MartinKPal]

@habbes , DataServiceContext is created per request as scoped service. We don't use singletons as a wrapping class or anything else, what should hold an instance of DataServiceContext. Parallel load with ~50 requests per second is quite common for this microservice, but the issue happens, only in described special use case.

SAP OData endpoint doesn't seem to be migrated to v4 protocol anytime soon, so we need to use Microsoft.Data.Services.Client v5.8.x.