Unable to Connect to the New Linux Based Emulator from a Docker Container (cosmosEmulatorVnextPreview)

[tnguyen2206]

I'm giving the new Linux-based emulator a try and running into an issue when trying to connect to it from a ASP.Net Core 8 app running in a Docker container. The request just times out after a while. I'm able to connect to the emulator from a console app just fine. I ran into the same issue when using the Windows local installed emulator but was able to get past the same issue by passing the /AllowNetworkAccess flag when running the emulator. I don't see any Docker command in this list https://learn.microsoft.com/en-us/azure/cosmos-db/emulator-linux#docker-commands that will allow me to set the AllowNetworkAccess flag. Is allowing network access not supported?

[alaye-ms]

You're right that we don't have the AllowNetworkAccess flag as part of the linux based emulator at this time. However, there might be something you can do with docker to allow for network access.

If you run your containers on the host network using the --network host flag, then they should be able to communicate with each other on localhost. See here for more details: https://docs.docker.com/engine/network/tutorials/host/

Alternatively, if you could try to point your client container to the emulator container using it's docker-assigned IP address (if they are both on the default bridge network). You can get that ip by running docker inspect <container_id>.

[tnguyen2206]

I tried the workarounds but no luck. When running the emulator container using the --network host flag, I could not access the explorer in the browser. When trying to connect to the emulator container using its docker-assigned IP, the request sent from my client app just timed out after a while.

[alaye-ms]

Would you be able to give me more details about how you're running this? Like how are you deploying these containers, what platform are you using, and what's your host OS? Maybe if you could post some output of the commands it could help to debug as well.

Here's how I tested things specifically. Running Docker in WSL Ubuntu on Windows.

docker run --rm --network host mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview For this workaround I had to turn on the feature on Docker Desktop following this instruction: "To enable this feature, navigate to the Resources tab in Settings, and then under Network select Enable host networking."

And then also allow networking to WSL from host windows with: New-NetFirewallRule -DisplayName "WSL" -Direction Inbound -InterfaceAlias "vEthernet (WSL (Hyper-V firewall))" -Action Allow

---

For the bridge network test, I ran the container like this: docker run --rm mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview and then ran in a new terminal docker inspect happy_elion and got the ip address 172.17.0.2

From there I checked the connection with docker run -it --rm busybox inside which I ran wget 172.17.0.2:8081 -O - which successfully connected to the endpoint.

[breaker05]

Having the same issue here. When I run the emulator standalone and my Api project standlone as well I can reach the CosmosDb and explorer, however when the two are in the same compose as below the connection just times out. The k-api has the correct connection string too.

k-api can not resolve the k-cosmo

name: k

services:
  
  k-cosmosdb:
    container_name: k-cosmosdb
    image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview
    command: --enable-telemetry true
    ports:
      - "8081:8081"
      - "1234:1234"

  k-api:
    container_name: k-api
    build:
      dockerfile: Dockerfile
      args:   
       - DbConfiguration__EndpointUrl=http://k-cosmosdb:8081
    ports:
      - "5180:8080"
    depends_on:
      - k-cosmosdb

[alaye-ms]

I used a modified version of your compose file with just a busybox:

name: k

services:

  k-cosmosdb:
    container_name: k-cosmosdb
    image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview
    command: --enable-telemetry false
    ports:
      - "8081:8081"
      - "1234:1234"

  k-api:
    container_name: k-api
    image: busybox
    command: tail -F anything
    environment:
       - EndpointUrl=http://k-cosmosdb:8081
    depends_on:
      - k-cosmosdb

and then connected to the k-api: docker exec -it k-api /bin/sh and ran wget $EndpointUri -O - And was able to get a response.

Would you be able to provide me some more details about your app? Like what cosmosdb API does it use, I know .NET and java don't yet support http connection.

[breaker05]

@alaye-ms This works for me too this way, however that is simply checking if it can reach the container by doing a wget and attempting to connect to the actual emulator.

I am using .NET 8 and Entity Framework. When it attempts to establish a connection that's when it throws the connection refused timeout. This is the case as well when using the --protocol https command flag.

[tnguyen2206]

Here is more information: I'm developing on Windows using Docker Desktop in Linux mode. My app is a .Net 8 minimal api application dockerized as a Linux container. In my local dev, I use skaffold to deploy my app to my local docker kubernetes cluster.

This is the command that I use to run the emulator: docker run --mount type=bind,source=/run/desktop/mnt/host/c/cosmodb-emulator-data,target=/usr/cosmo/data --detach --publish 8081:8081 --publish 1234:1234 mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview --data-path /usr/cosmo/data --protocol http

This is my .Net 8 console app code that I use to test the emulator outside container environment. This code works perfectly fine. `CosmosClientOptions clientOptions = new() { SerializerOptions = new CosmosSerializationOptions { PropertyNamingPolicy = CosmosPropertyNamingPolicy.CamelCase }, }; clientOptions.ConnectionMode = ConnectionMode.Gateway;

var cosmosClient = new CosmosClient(connectionString: "AccountEndpoint=http://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==", clientOptions: clientOptions);

var container = cosmosClient.GetDatabase("db-name").GetContainer("dummy");

await container.UpsertItemAsync(new DummyEntity { Id = Guid.NewGuid().ToString(), Name = "Dummy", Description = "This is a dummy entity", Category = "Dummy-1" });`

However, when the containerized app uses the same code to connect to the emulator, the request will just time out with "ExceptionType":"System.Net.Http.HttpRequestException","ExceptionMessage":"Connection refused (127.0.0.1:8081)" error. For my containerized app, I change the connection string to "AccountEndpoint=http://host.docker.internal:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw=="

My team currently uses the Windows local installed one. When I set that up for my team, I ran into the same connection issue when I ran the emulator normally from the Start menu. I had to spend a few days to find a workaround which was to run this command when running the emulator CosmosDB.Emulator.exe /AllowNetworkAccess /KeyFile=cosmosdbauthkey. I guess the key here is the AllowNetworkAccess flag.

With this new emulator, why does my console app connect just fine but the containerized app cannot?

[ulfandersson]

I had similar issues, i was able to curl the the 8081 endpoint, but when the SDK (in my case the Node SDK) was trying to connect it tried to connect to localhost and failed.

What fixed it for me was setting the (undocumented) env variable GATEWAY_PUBLIC_ENDPOINT to the hostname of the docker container inside my compose file:

cosmos:
    image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview
    environment:
      - PROTOCOL=http
      - GATEWAY_PUBLIC_ENDPOINT=cosmosdb
    hostname: cosmosdb

[alaye-ms]

I see, that makes sense. Thank you @ulfandersson for figuring this out! We will add more documentation around this variable to the official docs.

It will need to be the same as the $HOSTNAME variable inside the container to connect from inside the docker bridge network, and the default 127.0.0.1 to connect from the host machine when forwarding the port.

[xgerman]

We will be investigating to find. amore clean way to implement this so you don't need to set the GATEWAY_PUBLIC_IP

[tnguyen2206]

Does anyone have a solution for connecting to the emulator from a Docker container using the .Net SDK without using Docker compose?

[xgerman]

Wonder if you checked out the example in https://github.com/Azure/azure-cosmos-db-emulator-docker/issues/126

@christopheranderson should we make a second ticket to improve our docs and/or provide examples.

[tnguyen2206]

Wonder if you checked out the example in #126

@christopheranderson should we make a second ticket to improve our docs and/or provide examples.

I looked at the example in #126. The code is similar wo what I have. The issue that I still have is my console app can connect to the emulator just fine but my dockerized app cannot connect to the emulator even though both apps use the same .Net SDK and the same code.

[rmh54]

I have the same issue. I can connect from the host, but cannot connect when running the same code with an amended endpoint using the service name from the compose file. I've tried the undocumented env variable suggested by @ulfandersson but still can't make it work. I'm using .net 8.

I get the error

System.AggregateException: One or more errors occurred. (Connection refused (127.0.0.1:8081)) ---> System.Net.Http.HttpRequestException: Connection refused (127.0.0.1:8081) ---> System.Net.Sockets.SocketException (111): Connection refused

From the container failing to connect, I can get a response from curl https://linux-cosmosemulator-preview:8081 --insecure

[alaye-ms]

Would you be able to give me a few more details about your situation? What value did you give to GATEWAY_PUBLIC_ENDPOINT compared to the docker-compose service name and the endpoint given to the application? Also, are you using http or https? This is the docker-compose file I ended up testing with to get things to work

name: k

services:
  
  k-cosmosdb:
    container_name: k-cosmosdb
    image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview
    command: --enable-telemetry false --protocol http
    ports:
      - "8081:8081"
      - "1234:1234"
    environment:
      - GATEWAY_PUBLIC_ENDPOINT=k-cosmosdb

  k-api:
    container_name: k-api
    image: test-cosmos-dotnet
    environment:
       - ENDPOINT=http://k-cosmosdb:8081
    depends_on:
      - k-cosmosdb

where k-api is a dotnet application that just adds values to the db in a loop. I can provide that code as well if it would be useful

[rmh54]

Bit of an odd situation where I can make this work on my main desktop, but exactly the same code and set up doesn't work on my laptop. I'm using .net so using https and ConnectionMode.Gateway;

cosmosdb:
    container_name: cosmosdb
    image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview
    ports:
      - 8081:8081
      - 1234:1234
    command: ["--protocol", "https"]
    volumes:
      - ./linux_cosmosdbemulator/data:/data
    environment:
      GATEWAY_PUBLIC_ENDPOINT: cosmosdb

the container I'm using to write to cosmos is

auditwriter:
    container_name: auditwriter
    build:
      dockerfile: ./audit_writer/Dockerfile
      target: development
      secrets:
        - gh_lib_username
        - gh_lib_password
    volumes:
      - ./audit_writer:/audit_writer
    ports:
      - 5032:15672
    environment:
       ASPNETCORE_ENVIRONMENT: Development
    stdin_open: true
    tty: true
    depends_on:
      - rabbitmq

I've been testing this using the following f# script executed from the auditwriter container;

...
//set up connection
let endPointUri = "https://cosmosdb:8081"
let primaryKey = "C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw=="
let primaryConnectionString = $"AccountEndpoint={endPointUri}/;AccountKey={primaryKey}"
let options = CosmosClientOptions(
    ConnectionMode = ConnectionMode.Gateway,
    HttpClientFactory = fun () ->
        new HttpClient(new HttpClientHandler(
            ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator
        ))
)
...
//execute
let createAuditRecordDirect (item : Audit) =
    task {
        let! db = client.CreateDatabaseIfNotExistsAsync(dbId)
        let! containerResp =
            db.Database.CreateContainerIfNotExistsAsync(id = "auditlogs", partitionKeyPath = "/compositeKey", throughput = 400)
        let container = containerResp.Container
        let! createdItem = container.CreateItemAsync<Audit>(item = item, partitionKey = new PartitionKey(item.compositeKey))
        return createdItem
    }
    |> Async.AwaitTask
    |> Async.RunSynchronously

On the working desktop, https://localhost:8081 shows the correct databaseAccountEndpoint

{
    "_self": "",
    "id": "cosmosdev",
    "_rid": "cosmosdev",
    "media": "//media/",
    "addresses": "//addresses/",
    "_dbs": "//dbs/",
    "enableMultipleWriteLocations": false,
    "writableLocations": [
        {
            "name": "Primary",
            "databaseAccountEndpoint": "https://cosmosdb:8081/"
        }
    ],
    "readableLocations": [
        {
            "name": "Primary",
            "databaseAccountEndpoint": "https://cosmosdb:8081/"
        }
    ],
    ...
}

whereas on non-working laptop the read out is the same, but the endpoint is "databaseAccountEndpoint": "https://127.0.0.1:8081/"

Executing the F# test script hangs until eventually I get the Connection Refused exception mentioned previously.

?

[alaye-ms]

That's odd. If both setups are running the same code, could it be that they are using different versions of the azure-cosmos-emulator docker image?

[rmh54]

Ok, got this working. The problem is the volume in the cosmosdb service. This builds and connects when that is not used. Enabling it afterwards then allows the auditwriter to connect and the data is successfully persisted.

[boy-pabu]

Not sure if this is still needed, but setting GATEWAY_PUBLIC_ENDPOINT to "*" allowed for the same emulator to be used from both host and another container.

[xgerman]

@christopheranderson wonde rif we cna add this to docs.

[xgerman]

We will set the GATEWAY_PUBLIC_ENDPOINT per default to '*'

[despian]

--env GATEWAY_PUBLIC_ENDPOINT="*" seems required when using the mount option and using data created by another container instance.

I'm using the latest image. I suppose this default has not been added yet.

[xgerman]

We learned we can't deafult the value to this becsuae it breaks other things. @abhirockzz will document.