Read/Write endpoints unreachable

[Makciek]

Describe the bug After pulling the newest image my integration tests started breaking with some very strange errors. So I tried to create the simplest repro but I found out that even just the basic scenario copied from docs(https://learn.microsoft.com/en-us/azure/cosmos-db/how-to-develop-emulator?tabs=docker-linux%2Ccsharp&pivots=api-nosql#import-the-emulators-tlsssl-certificate) doesn't work.

To Reproduce Follow https://learn.microsoft.com/en-us/azure/cosmos-db/how-to-develop-emulator?tabs=docker-linux%2Ccsharp&pivots=api-nosql#import-the-emulators-tlsssl-certificate

When running:

using CosmosClient cosmosClient = new(
    accountEndpoint: "https://localhost:8081/",
    authKeyOrResourceToken: "C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw=="
);
Database database = await cosmosClient.CreateDatabaseIfNotExistsAsync(
    id: "cosmicworks",
    throughput: 400
);

Application(.net 8, Microsoft.Azure.Cosmos:3.40.0) hangs, the output windows print the following:

... DocDBTrace Information: 0 : Current WriteEndpoints = (https://172.17.0.5:8081/) ReadEndpoints = (https://172.17.0.5:8081/) DocDBTrace Information: 0 : GlobalEndpointManager: StartLocationBackgroundRefreshWithTimer() refreshing locations DocDBTrace Information: 0 : Creating RNTBD TransportClient with options Rntbd.TransportClient.Options OpenTimeout: 00:00:05 RequestTimeout: 00:00:06 TimerPoolResolution: 00:00:01 MaxChannels: 65535 PartitionCount: 1 MaxRequestsPerChannel: 30 ReceiveHangDetectionTime: 00:01:05 SendHangDetectionTime: 00:00:10 IdleTimeout: -00:00:01 UserAgent: cosmos-netstandard-sdk/3.40.0|1|X64|Microsoft Windows 10.0.22631|.NET 8.0.5|N| Suffix: CertificateHostNameOverride: LocalRegionTimeout: 00:00:05 EnableChannelMultiplexing: False MaxConcurrentOpeningConnectionCount: 5 Use_RecyclableMemoryStream: False Use_CustomDnsResolution: False IsDistributedTracingEnabled: False DocDBTrace Information: 0 : TimerPool Created with minSupportedTimerDelayInSeconds = 1 DocDBTrace Information: 0 : Creating RNTBD TransportClient with options Rntbd.TransportClient.Options OpenTimeout: 00:00:05 RequestTimeout: 00:00:06 TimerPoolResolution: 00:00:01 MaxChannels: 65535 PartitionCount: 1 MaxRequestsPerChannel: 1 ReceiveHangDetectionTime: 00:01:05 SendHangDetectionTime: 00:00:10 IdleTimeout: -00:00:01 UserAgent: cosmos-netstandard-sdk/3.40.0|1|X64|Microsoft Windows 10.0.22631|.NET 8.0.5|N| Suffix: CertificateHostNameOverride: LocalRegionTimeout: 00:00:05 EnableChannelMultiplexing: False MaxConcurrentOpeningConnectionCount: 5 Use_RecyclableMemoryStream: False Use_CustomDnsResolution: False IsDistributedTracingEnabled: False DocDBTrace Information: 0 : TimerPool Created with minSupportedTimerDelayInSeconds = 1 DocDBTrace Information: 0 : Resolving Master service address, forceMasterRefresh: False, currentMaster: The thread '.NET TP Worker' (2488) has exited with code 0 (0x0). DocDBTrace Warning: 0 : ClientRetryPolicy: Gateway HttpRequestException Endpoint not reachable. Failed Location: https://172.17.0.5:8081/; ResourceAddress: dbs/cosmicworks DocDBTrace Information: 0 : GlobalEndpointManager: Marking endpoint https://172.17.0.5:8081/ unavailable for read DocDBTrace Information: 0 : Current WriteEndpoints = (https://172.17.0.5:8081/) ReadEndpoints = (https://172.17.0.5:8081/) DocDBTrace Information: 0 : Endpoint https://172.17.0.5:8081/ unavailable for Read added/updated to unavailableEndpoints with timestamp 05/22/2024 14:53:57 DocDBTrace Information: 0 : GlobalEndpointManager: Marking endpoint https://172.17.0.5:8081/ unavailable for Write DocDBTrace Information: 0 : Current WriteEndpoints = (https://172.17.0.5:8081/) ReadEndpoints = (https://172.17.0.5:8081/) DocDBTrace Information: 0 : Endpoint https://172.17.0.5:8081/ unavailable for Write added/updated to unavailableEndpoints with timestamp 05/22/2024 14:53:57 DocDBTrace Information: 0 : Current WriteEndpoints = (https://172.17.0.5:8081/) ReadEndpoints = (https://172.17.0.5:8081/) DocDBTrace Information: 0 : Resolving Master service address, forceMasterRefresh: False, currentMaster: DocDBTrace Warning: 0 : ClientRetryPolicy: Gateway HttpRequestException Endpoint not reachable. Failed Location: https://172.17.0.5:8081/; ResourceAddress: dbs/cosmicworks ... and so on...

Expected behavior Doesn't hang

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Windows 11

**Docker Images Used: **

• Linux

**Arguments && Environment variables to start Docker: docker run --publish 8081:8081 --publish 10250-10255:10250-10255 --interactive --tty mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest

Docker Environment

• Docker compose
• Standalone
• Others (Please specify)   Additional context I also tried using HttpClientHandler.DangerousAcceptAnyServerCertificateValidator without installing the certificate but then I always got the SSL error

[Makciek]

regarding the original issue - it looks like the newest version returns the malformed response when there are no records as a result of the query - sorry for not creating a separate issue but I can't create a simple repro for it because of above

[Makciek]

the same code works when running on the non-docker emulator

[Makciek]

looks like using --env AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE=127.0.0.1 helps with the issue of hangning

[Makciek]

and here's the simplest repro for my 'original' issue:

[Makciek]

using

var query = container
    .GetItemLinqQueryable<Model>
    (
        linqSerializerOptions: new CosmosLinqSerializerOptions()
        {
            PropertyNamingPolicy = CosmosPropertyNamingPolicy.CamelCase
        },
        requestOptions: new QueryRequestOptions()
        {
            PartitionKey = new PartitionKey(Guid.NewGuid().ToString())
        }
    )
    .Where(x => true);

instead fixes the issue

[Makciek]

@sajeetharan hey, my workaround is just a workaround - the issue is still there the new docker emulator behaves differently from the actual cosmos db/windows emulator

[PhilippRoessner]

this is still happening here, any updates?