azure-cosmos-db-emulator-docker icon indicating copy to clipboard operation
azure-cosmos-db-emulator-docker copied to clipboard

Receiving PGCosmosError on the latest vnext-EN20251118 docker image

Open Nicky1992 opened this issue 2 weeks ago • 4 comments

Describe the bug Integrations tests from our CI/CD started to fail once we switch to the latest image (e.g.: azure-cosmos-emulator:vnext-EN20251118). In our tests, after creating the docker container to ensure that the service is up & running we are making a GET call to https://localhost:8081/dbs and if this call returns 200 then we move forward with running the actual tests. This GET call started to fail by returning 403 and upon inspecting the return body we noticed this error: PGCosmosError(2, "Authorization header missing", <disabled>). Once we revert back to azure-cosmos-emulator:vnext-EN20251022, things get back to normal.

To Reproduce Steps to reproduce the behavior:

  1. Update to azure-cosmos-emulator:vnext-EN20251118
  2. Issue a HTTP GET call to https://localhost:8081/dbs (e.g.: curl --insecure https://localhost:8081/dbs)
  3. Call fails with 403, PGCosmosError(2, "Authorization header missing", <disabled>)

Expected behavior The call should return 200 like the previous version. E.g.:

curl --insecure https://localhost:8081/dbs
{"_rid": "", "_count": 0, "Databases": []}

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]
  • SDK version
  • SDK language

Additional context Add any other context about the problem here.

Nicky1992 avatar Nov 21 '25 11:11 Nicky1992

The authorization header is a required header in the main Azure CosmosDB REST API (https://learn.microsoft.com/en-us/rest/api/cosmos-db/common-cosmosdb-rest-request-headers). In the latest emulator release we added a check to make sure this header has a valid value. You can add this header to any curl with a header that has "Basic" at the beginning: curl --insecure -H "Authorization: Basic $(echo -n 'username:password' | base64)" https://localhost:8081/dbs This is meant to ensure compatibility for customers who want to migrate their data from the emulator to the main Azure service while having their REST calls still be valid.

Ritvik-Jayaswal avatar Nov 21 '25 17:11 Ritvik-Jayaswal

Why wasn’t this check clearly captured in the release notes or added in the documentation?

Secondly, can we add anything for username:password? cc: @Ritvik-Jayaswal

Nicky1992 avatar Nov 21 '25 18:11 Nicky1992

I will confirm to make sure this behavior is documented properly. For the string section username:password anything can be added as long as it can be converted to base64.

Ritvik-Jayaswal avatar Nov 21 '25 19:11 Ritvik-Jayaswal

@Ritvik-Jayaswal , @xgerman

I've managed to pass over the issue related to the missing mandatory Authorization header.

But now, I'm hitting an error while issuing HTTP PATCH operation on a record which was previously created. It worked previously on EN20251022.

On EN20251022 I'm making the following call which succeeds:

curl --http1.1 --insecure  --request PATCH --url https://localhost:59707/dbs/ups_FB3C3EC6621DC3480A495EB2@AdobeOrg_sg/colls/_xdm.context.profile/docs/test-adl-1 \
  -H "Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=" \
  -H "x-ms-version: 2018-09-17" \
  -H "x-ms-documentdb-partitionkey: [\"ADL-KY-1\"]" \
  -H "Accept: application/json" \
  -H "x-ms-date: Fri, 21 Nov 2025 21:56:17 GMT" \
  -H "Content-Type: application/json_patch+json" \
  --data '{"operations":[{"op":"set","path":"/ttl","value":30}]}' -vvv

*   Trying [::1]:59707...
* Connected to localhost (::1) port 59707
* ALPN: curl offers http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: C=US; ST=WA; O=SqlPostgresHostConsole; CN=localhost
*  start date: Nov  4 17:04:47 2025 GMT
*  expire date: Nov  4 17:04:47 2026 GMT
*  issuer: C=US; ST=WA; O=SqlPostgresHostConsole; CN=localhost
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.1
> PATCH /dbs/ups_FB3C3EC6621DC3480A495EB2@AdobeOrg_sg/colls/_xdm.context.profile/docs/test-adl-1 HTTP/1.1
> Host: localhost:59707
> User-Agent: curl/8.4.0
> Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=
> x-ms-version: 2018-09-17
> x-ms-documentdb-partitionkey: ["ADL-KY-1"]
> Accept: application/json
> x-ms-date: Fri, 21 Nov 2025 21:56:17 GMT
> Content-Type: application/json_patch+json
> Content-Length: 54
>
< HTTP/1.1 200 OK
< Date: Fri, 21 Nov 2025 22:13:36 GMT
< Transfer-Encoding: chunked
< Content-Type: application/json
< Server: PGSQL
< x-ms-gatewayversion: 2.0.0-EN20251022
< x-ms-activity-id: c627dc98-b4b9-4dc9-acdc-0d8a79b63b10
< x-ms-documentdb-partitionkeyrangeid: 0
< x-ms-session-token: 0:-1#1
< x-ms-interopandsdk-time-ms: 33
< etag: 3b79bd23-fd9d-44ca-a52a-c0b6f67c9384
<
* Connection #0 to host localhost left intact
{"ek": "66b4aa990af6ac2aee3d6aab", "et": {"tp": {"_id": "20240617-133124_test_adl", "eventType": "adl_integration_test"}}, "id": "test-adl-1", "ky": "ADL-KY-1", "rt": "timeseries", "ts": "1718631084000", "_ts": 1763763216, "tag": "someTag", "ttl": 30, "_rid": "AQAAAAEAAAAAAAAZqoeXitAAAAE=", "_etag": "3b79bd23-fd9d-44ca-a52a-c0b6f67c9384", "_self": "dbs/ups_FB3C3EC6621DC3480A495EB2@AdobeOrg_sg/colls/_xdm.context.profile/docs/test-adl-1/", "kyhash": "fb563ad3f9d0151291bef543517df099"

However, when performing the same call against EN20251118, I'm getting 400 error:

*   Trying [::1]:60255...
* Connected to localhost (::1) port 60255
* ALPN: curl offers http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: C=US; ST=WA; O=SqlPostgresHostConsole; CN=localhost
*  start date: Nov 19 01:41:46 2025 GMT
*  expire date: Nov 19 01:41:46 2026 GMT
*  issuer: C=US; ST=WA; O=SqlPostgresHostConsole; CN=localhost
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.1
> PATCH /dbs/ups_FB3C3EC6621DC3480A495EB2@AdobeOrg_sg/colls/_xdm.context.profile/docs/test-adl-1 HTTP/1.1
> Host: localhost:60255
> User-Agent: curl/8.4.0
> Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=
> x-ms-version: 2018-09-17
> x-ms-documentdb-partitionkey: ["ADL-KY-1"]
> Accept: application/json
> x-ms-date: Fri, 21 Nov 2025 21:56:17 GMT
> Content-Type: application/json_patch+json
> Content-Length: 54
>
< HTTP/1.1 400 Bad Request
< content-length: 149
< access-control-max-age: 3600
< content-type: application/json
< access-control-allow-methods: GET, POST, PUT, PATCH, DELETE, OPTIONS
< access-control-allow-headers: *
< access-control-allow-origin: *
< date: Fri, 21 Nov 2025 22:21:41 GMT
<
* Connection #0 to host localhost left intact
{"error":"Invalid path or method","method":"PATCH","path":"/dbs/ups_FB3C3EC6621DC3480A495EB2@AdobeOrg_sg/colls/_xdm.context.profile/docs/test-adl-1"}

Can someone please tell me what changed since the last release when it worked without any problem?

Nicky1992 avatar Nov 21 '25 22:11 Nicky1992

@Nicky1992 We removed the requirement that the authorization header is required and @Ritvik-Jayaswal also addressed the issue with PATCH in EN20251124. We are hooing this resolves your issues.

xgerman avatar Nov 25 '25 00:11 xgerman

Thanks @xgerman and team! @Nicky1992 please check https://github.com/Azure/azure-cosmos-db-emulator-docker/releases/tag/EN20251124 for details

abhirockzz avatar Nov 25 '25 02:11 abhirockzz

Thanks everyone for looking into this! I'll pull the latest and see if the issue is still reproducing.

Nicky1992 avatar Nov 25 '25 17:11 Nicky1992

I can confirm that the PATCH operation is working fine now.

Nicky1992 avatar Nov 26 '25 13:11 Nicky1992

The main issue reported about PATH operation seems to be solved with the latest release of the image.

However, I noticed something while testing: when trying to DELETE an item using raw HTTP calls it seems that I’m receiving back from the emulator in the header associated with the http version : nullHTTP/1.1 instead of HTTP/1.1. This is obviously breaking our HTTP client that we’re using because nullHTTP/1.1 is malformed and not parsable.

Somehow this issue is related to the connection reuse mechanism because if I shut it down by sending in the header Connection: close​ then the emulator is sending back this time a well formed version. Unfortunately, I wasn’t able to reproduce it with curls because each curl command seems to be using a sole connection and each in its own, ran serially seems to be fine. I think the only way to reproduce it is to fire the DELETE by using a HTTP client where connection reuse is enabled. In our test we used http4s

Thanks!

Nicky1992 avatar Nov 26 '25 14:11 Nicky1992

We will have a fix for the null issue in the next release.

xgerman avatar Nov 26 '25 16:11 xgerman