Unable to connect to cosmos db emulator when using linux containers with container network

[ajaygoyal]

I have a .net 6 app that needs to connect to cosmos db emulator.

I have below setup:

test-app: env_file: - ../../Environment/local.env build: context: ./ dockerfile: Services/Dockerfile ports: - "80:80" entrypoint: > /bin/sh -c " update-ca-certificates dotnet MyApp.dll" depends_on: - cosmosdb_emulator networks: - container_network

cosmosdb_emulator: image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator ports: - "8081:8081" environment: AZURE_COSMOS_EMULATOR_PARTITION_COUNT: 10 ACCEPT_EULA: "Y" networks: - container_network

Downloaded the cert on host using: curl -k https://host.docker.internal:8081/_explorer/emulator.pem > /c/certs/emulatorcert.crt

But application fails to connect to the cosmos with below error:

2023-10-22 20:06:51 ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. 2023-10-22 20:06:51 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure: RemoteCertificateNameMismatch

Disabling SSL is not something that we want to do.

[kbegiedza]

Did you install downloaded cert inside your test-app container?

[abkolant-MSFT]

Could you check the example YAML file and this folder in the .NET examples. This example shows how to use Cosmos DB Emulator with containers and networks. The example, also shows how to download and install certs into your test app, that connects to the Emulator.

[JoeAmedeo]

Is there a way to make this work without using HttpClientHandler.DangerousAcceptAnyServerCertificateValidator as a cert validation callback? I am trying to set up effectively the same application as the example, but with the Azure Cosmos DB Migration Tool where I can't make this configuration, and removing that from the example app causes the same RemoteCertificateNameMismatch error.

I am not super confident on my understanding of SSL certificates, but is it possible to use the /GenCert={host name for the service} arg to make this work? I have been following this example from another issue and tried overriding the ./startup.sh file to use said argument, but at that point the emulator will not start.

[Davilink]

This really seem like an oversight. I can't use cosmos emulator in devcontainer because of this simple issue... one thing could be to add the hostname as an alternate name in the certificate.

[Davilink]

I got it working without using the HttpClientHandler.DangerousAcceptAnyServerCertificateValidator.

I inspected the certificate and found out that it does contains alternate subject name, but there a twist, you have to declare a hostname for your service.

docker-compose.yml (good)

version: '3'
services:
  cosmos:
    image:  mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest
    hostname: cosmos # <-- required
    restart: unless-stopped
    ports:
      - 8081:8081
    networks:
      game:
        aliases:
          - "cosmos.domain" # <-- required

If you don't specifiy a hostname, docker will generate a random one example:

docker-compose.yml (bad)

version: '3'
services:
  cosmos:
    image:  mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest
    # hostname: cosmos
    restart: unless-stopped
    ports:
      - 8081:8081
    networks:
      game:
        aliases:
          - "cosmos.domain" # <-- required

Note: I tried using the cosmosdbemulatormtls.localhost but it doesn't work (problably because of the .localhost at the end).

The hostname doesn't have to be cosmos, i just choose that name, but if you change it don't forget to update the aliase use in the docker compose file.

docker-compose.yml (alternate config good)

version: '3'
services:
  cosmos:
    image:  mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest
    hostname: othername
    restart: unless-stopped
    ports:
      - 8081:8081
    networks:
      game:
        aliases:
          - "othername.domain"

Once the cosmos emulator container is started, it take a minute or so to be able to retrieve the certificate. I'm currently setting up a devcontainer using cosmos emulator and added this in my postCreateCommand.sh script:

#!/bin/bash
echo "Installing cosmos db emulator ssl certificate"

mkdir /usr/share/ca-certificates/cosmos

for i in {1..60};
do
  curl -fsk https://cosmos.domain:8081/_explorer/emulator.pem > /usr/share/ca-certificates/cosmos/emulator.crt
  if [ $? -eq 0 ];  then
    echo "Cosmos emulator ready"
    break
  else
    echo "Not ready yet..."
    sleep 10
  fi
done

chmod o+r /usr/share/ca-certificates/cosmos/emulator.crt
echo "cosmos/emulator.crt" >> /etc/ca-certificates.conf
update-ca-certificates

after this, you will need to use the cosmos.domain:8081 (update the port number if your setup use an another port)

appsetings.Development.json

{
  // ...
  "ConnectionStrings": {
    "CosmosConnection": "AccountEndpoint=https://cosmos.domain:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==",
  },
  // ...
}

The account key is the default one used by the cosmos emulator, so you can copy-paste the connection string for you use.

[cdytoby]

Additional information to the post above:

curl is default not installed. So for automation, you need to install curl.

This is the example snippet to the Dockerfile of the asp.net project

FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS base
WORKDIR /app
RUN apt-get update && apt-get install -y curl
EXPOSE 8080

Credit: https://stackoverflow.com/questions/73678713/how-to-install-curl-from-dockerfile

PS: after add this RUN line, you need to delete relevant containers and re-pull and build.

But..... I still get SSL error:

The SSL connection could not be established, see inner exception.

InnerException: The remote certificate is invalid according to the validation procedure: RemoteCertificateNameMismatch, RemoteCertificateChainErrors

[hughesjs]

I'm getting the same as @cdytoby... Any ideas beyond just disabling SSL validation for now?

[hughesjs]

Okay, so I worked out my problem.

The cosmos.domain listed as an alias above isn't just "any generic domain" it's literally ".domain".

Seems to me you can summarise that long post into:

• Set the hostname in your compose (not/as well as the container_name).
• Add an alias to the network for <hostname>.domain

This will then ensure that you can resolve the emulator with a hostname that is present in the certificate's alternate names

You can probably get all of this from the long answer above, but I missed it so figured I'd share what I learned a bit more concisely.

[hughesjs]

So following on from my last post, I can go one better.

I don't think you even need the alias, just set the hostname to cosmos.domain and refer to it through that everywhere else. I'd even recommend leaving out the container_name so you can't resolve it any other way unless you really like having nicely named containers.

  cosmos-db-emulator:
    image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest
    hostname: cosmos.domain # <--- This .domain is the only thing that matters really
    environment:
      AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE: true
      AZURE_COSMOS_EMULATOR_PARTITION_COUNT: 1
    networks:
      - internal

[seesharprun]

@hughesjs, @cdytoby, and @davilink:

I'm just letting you know that we are going to update the Azure Cosmos DB emulator documentation to include steps on using the emulator in Docker Compose for devcontainer/Codespaces scenarios and using the emulator in GitHub Actions for service container scenarios.

In the interim, here's what I've been able to do.

GitHub Codespaces

For GitHub Codespaces, I'm using a devcontainer configuration and the hostname:

{
    "dockerComposeFile": "docker-compose.yml",
    "service": "dev",
    "workspaceFolder": "/workspace",
    "containerEnv": {
        "COSMOSDB__CONNECTIONSTRING": "AccountEndpoint=https://cosmos.domain:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw=="
    },
    "postStartCommand": "bash ./.devcontainer/postStartCommand.sh",
    "remoteUser": "root"
}

services:
  dev:
    image: mcr.microsoft.com/dotnet/sdk:8.0
    command: sleep infinity
    volumes:
      - ..:/workspace
    depends_on:
      - cosmosdb
  cosmosdb:
    image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator
    hostname: cosmos.domain
    ports:
      - 8081

echo "Installing emulator certificate"

curl --insecure https://cosmos.domain:8081/_explorer/emulator.pem > ~/emulatorcert.crt

cp ~/emulatorcert.crt /usr/local/share/ca-certificates/

update-ca-certificates

GitHub Actions

For GitHub Actions, I'm using a workflow similar to this:

on:
  pull_request:
jobs:
  test:
    runs-on: ubuntu-latest
    services:
      cosmosdb:
        image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator
        ports:
          - 8081:8081
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Setup .NET
        uses: actions/setup-dotnet@v4
        with:
          dotnet-version: 8.x
      - name: Install SSL certificate
        run: |
          curl --insecure https://localhost:8081/_explorer/emulator.pem > ~/emulatorcert.crt
          sudo cp ~/emulatorcert.crt /usr/local/share/ca-certificates/
          sudo update-ca-certificates
      - name: Run .NET test
        run: dotnet test --logger "console"
        env:
          COSMOSDB__CONNECTIONSTRING: "AccountEndpoint=https://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw=="
        working-directory: ./validate/test

Next steps

I'm working on finding a way to use docker health checks to ping the emulator and determine if it's ready so we don't have to do it in Bash.

[hughesjs]

That's great to hear, thanks! Let me know if there's anything I can do to help

[seesharprun]

I'm working on a pull request for an existing code sample repository and am trying to implement these: https://github.com/Azure-Samples/cosmos-db-nosql-query-samples/pull/35

Once it's ready, I will follow up on this thread

[seesharprun]

Quick update. I had a breakthrough. Here's the updated YML files that will be included in the new documenation:

GitHub Action workflow

on:
  pull_request:
jobs:
  validate:
    runs-on: ubuntu-latest
    services:
      cosmosdb:
        image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest
        ports:
          - 8081:8081
          - 10250-10255:10250-10255
        env:
          AZURE_COSMOS_EMULATOR_PARTITION_COUNT: 5
          AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE: true
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Install .NET 8
        uses: actions/setup-dotnet@v4
        with:
          dotnet-version: 8.x
      - name: Install SSL certificate
        run: |
          retry_count=0
          max_retry_count=10
          until sudo curl --insecure --silent --fail --show-error "https://localhost:8081/_explorer/emulator.pem" --output "/usr/local/share/ca-certificates/cosmos-db-emulator.crt"; do
            if [ $retry_count -eq $max_retry_count ]; then
              echo "Failed to download certificate after $retry_count attempts."
              exit 1
            fi
            echo "Failed to download certificate. Retrying in 5 seconds..."
            sleep 5
            retry_count=$((retry_count+1))
          done
          sudo update-ca-certificates
      - name: Run .NET test
        run: dotnet test --logger "console" --logger "trx;verbosity=detailed"
        env:
          COSMOSDB__CONNECTIONSTRING: "AccountEndpoint=https://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw=="
        working-directory: ./validate/test
      - name: Upload TRX test results
        uses: actions/upload-artifact@v4
        with:
          name: test-results
          path: validate/test/TestResults/**/*.trx

Devcontainer postStartCommand shell script

#!/bin/bash

echo "Installing Azure Cosmos DB emulator certificate"

for _ in {1..10};
do  
  if curl --insecure --silent --fail --show-error https://cosmosdb.domain:8081/_explorer/emulator.pem > /usr/local/share/ca-certificates/cosmos-db-emulator.crt;
  then
    echo "Azure Cosmos DB emulator certificate downloaded..."
    break
  else
    echo "Failed to download Azure Cosmos DB emulator certificate. Retrying in 5 seconds..."
    sleep 5
  fi
done

update-ca-certificates

I'll post another update on this thread once the new documentation is live. Thanks for your patience!

[hughesjs]

Is there a reason the cert has to be regenerated on every container start? Being able to just bake the cert into our containers at build time would make this whole process a lot more straightforward

[seesharprun]

I don't know the answer to that question. It's been my experience that the certificate can be different between starts of the container.

@sajeetharan, do you know more about how the certificates work in the Docker (Linux) container image?

[seesharprun]

I also am wondering aloud if there's a healthcheck built into the container image.

GitHub cites Redis and PostgreSQL container images as examples of container images with good health checks you can use to make sure the container is ready. Here's some examples in service containers within GitHub Actions (using docker run --health-cmd):

---

Redis has the redis-cli tool and you can run the ping command to see if its ready.

jobs:
  run-redis:
    services:
      rediscache:
        image: redis
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

https://docs.github.com/actions/using-containerized-services/creating-redis-service-containers

---

PostgreSQL has the pg_isready tool.

jobs:
  run-postgres:
    services:
      postgresdb:
        image: postgres
        env:
          POSTGRES_PASSWORD: "[email protected]"
        options: >-
          --health-cmd "pg_isready"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

https://docs.github.com/actions/using-containerized-services/creating-postgresql-service-containers

---

You can even do it with MongoDB.

jobs:
  run-mongodb:
    services:
      mongodb:
        image: mongo
        env:
          MONGO_INITDB_ROOT_USERNAME: "mongo"
          MONGO_INITDB_ROOT_PASSWORD: "[email protected]"
        options: >-
          --health-cmd "mongosh --username 'mongo' --password '[email protected]' --eval 'db.runCommand(""ping"").ok' --authenticationDatabase 'admin' --quiet"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

If a healthcheck script or tool doesn't already exist, it may be worth building. In the interim we can probably build custom Docker container images and write our own health check script.

[hughesjs]

I also am wondering aloud if there's a healthcheck built into the container image.

GitHub cites Redis and PostgreSQL container images as examples of container images with good health checks you can use to make sure the container is ready. Here's some examples in service containers within GitHub Actions (using docker run --health-cmd):

---

Redis has the redis-cli tool and you can run the ping command to see if its ready.

jobs:

  run-redis:

    services:

      rediscache:

        image: redis

        options: >-

          --health-cmd "redis-cli ping"

          --health-interval 10s

          --health-timeout 5s

          --health-retries 5

https://docs.github.com/actions/using-containerized-services/creating-redis-service-containers

---

PostgreSQL has the pg_isready tool.

jobs:

  run-postgres:

    services:

      postgresdb:

        image: postgres

        env:

          POSTGRES_PASSWORD: "[email protected]"

        options: >-

          --health-cmd "pg_isready"

          --health-interval 10s

          --health-timeout 5s

          --health-retries 5

https://docs.github.com/actions/using-containerized-services/creating-postgresql-service-containers

---

You can even do it with MongoDB.

jobs:

  run-mongodb:

    services:

      mongodb:

        image: mongo

        env:

          MONGO_INITDB_ROOT_USERNAME: "mongo"

          MONGO_INITDB_ROOT_PASSWORD: "[email protected]"

        options: >-

          --health-cmd "mongosh --username 'mongo' --password '[email protected]' --eval 'db.runCommand(""ping"").ok' --authenticationDatabase 'admin' --quiet"

          --health-interval 10s

          --health-timeout 5s

          --health-retries 5

If a healthcheck script or tool doesn't already exist, it may be worth building. In the interim we can probably build custom Docker container images and write our own health check script.

Health check wise, I had to add curl to the image and then just curl the cert to detect when it's online. It's not the most elegant option though

[jjICP]

I have done everything here, I actually wrote the below script but I still get the error:

Unhandled exception. System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: UntrustedRoot

curl gives me:

$ curl https://cosmos.domain:8081/_explorer/emulator.pem
curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

It doesn't matter whether I use cosmos.domain or localhost. Any ideas? My script for WSL:

#!/bin/bash

create_folder() {
  dir=~/devcosmos

  # Check if the directory exists, and create it if it doesn't
  if [ ! -d "$dir" ]; then
    echo "Directory $dir does not exist. Creating..."
    mkdir "$dir"
  else
    echo "Directory $dir already exists."
  fi
}

create_compose_file() {
  file_path=~/devcosmos/docker-compose.yml

  # Write the content to the file using a here document
cat <<EOL > "$file_path"
  services:
    cosmos:
      image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest
      hostname: cosmos.domain
      restart: unless-stopped
      ports:
        - $1:8081
EOL
}

get_cosmosDBEmu_ssl() {
  url="https://localhost:$1/_explorer/emulator.pem"
  file_path=~/devcosmos/emulatorcert.crt

  if [ -f "$file_path" ]; then
    rm "$file_path"
  fi

  echo "Downloading Cosmos DB Emulator SSL file (can take a few minutes while the container spins up)..>

  until [ -f "$file_path" ]; do
    if curl -s --insecure "$url" -o "$file_path"; then
      echo "File downloaded successfully."
    else
      # Optionally, you can delete the file if it was partially downloaded
      [ -f "$file_path" ] && rm "$file_path"
    fi

    # Add a short sleep to avoid hammering the server
    sleep 5
  done

  echo "File downloaded successfully."
}

install_ssl() {
  echo "Installing Cosmos DB Emulator SSL cert..."
  sudo cp ~/emulatorcert.crt /usr/local/share/ca-certificates/
  sudo update-ca-certificates
  echo "SSL cert installed successfully!"
}

port="8081"

cd ~
create_folder
cd ~/devcosmos
create_compose_file "$port"
docker-compose up -d
get_cosmosDBEmu_ssl "$port"
install_ssl