[Bug]: Cannot connect to CosmosDB Container if Direct mode is used

[fabriciorby]

Module

Azure

Testcontainers version

1.17.2

Using the latest Testcontainers version?

Yes

Docker version

❯ docker version
Client:
 Cloud integration: v1.0.20
 Version:           20.10.10
 API version:       1.41
 Go version:        go1.16.9
 Git commit:        b485636
 Built:             Mon Oct 25 07:43:15 2021
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.10
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.9
  Git commit:       e2f740d
  Built:            Mon Oct 25 07:41:30 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.11
  GitCommit:        5b46e404f6b9f661a205e28d59c982d3634148f8
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

What happened?

Hello,

I want to run the testcontainer using direct connection instead of gateway connection for Cosmos DB. https://www.testcontainers.org/modules/azure/

After following this tutorial I was able to run on gateway mode, but when I try to run on direct I get Fail to reach global gateway [https://localhost:54950]

I've exposed the missing ports 10251, 10252, 10253, 10254 but the error still goes on.

When I use the image directly on Docker I am able to run the tests without any issues. https://docs.microsoft.com/en-us/azure/cosmos-db/linux-emulator?tabs=sql-api%2Cssl-netstd21

Relevant log output

25-06-2022 14:16:17.549 [main] INFO  ?.m.com/cosmosdb/linux/azure-cosmos-emulator:latest].tryStart - Container mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest is starting: 35b1dfb27ae5fbd9011249ffd8cd7bb602d470387fb4d7142504e4d09f8f245c
25-06-2022 14:16:36.009 [main] INFO  ?.m.com/cosmosdb/linux/azure-cosmos-emulator:latest].tryStart - Container mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest started in PT18.625343S
25-06-2022 14:16:36.012 [main] INFO  c.a.c.i.ImplementationBridgeHelpers.setDirectConnectionConfigAccessor - Setting DirectConnectionConfigAccessor...
25-06-2022 14:17:25.608 [main] INFO  c.a.c.i.ImplementationBridgeHelpers.setCosmosClientBuilderAccessor - Setting CosmosClientBuilderAccessor...
25-06-2022 14:17:26.557 [main] INFO  c.a.c.i.RxDocumentClientImpl.<init> - Initializing DocumentClient [1] with serviceEndpoint [https://localhost:54950], connectionPolicy [ConnectionPolicy{httpNetworkRequestTimeout=PT1M, tcpNetworkRequestTimeout=PT5S, connectionMode=DIRECT, maxConnectionPoolSize=1000, idleHttpConnectionTimeout=PT1M, idleTcpConnectionTimeout=PT0S, userAgentSuffix='', throttlingRetryOptions=RetryOptions{maxRetryAttemptsOnThrottledRequests=9, maxRetryWaitTime=PT30S}, endpointDiscoveryEnabled=false, preferredRegions=null, multipleWriteRegionsEnabled=true, proxyType=null, inetSocketProxyAddress=null, readRequestsFallbackEnabled=true, connectTimeout=PT5S, idleTcpEndpointTimeout=PT1H, maxConnectionsPerEndpoint=130, maxRequestsPerConnection=30, tcpConnectionEndpointRediscoveryEnabled=true}], consistencyLevel [null], directModeProtocol [Tcp]
25-06-2022 14:17:26.956 [main] DEBUG c.a.c.i.GlobalEndpointManager.startRefreshLocationTimerAsync - registering a refresh in [300000] ms
25-06-2022 14:17:26.982 [cosmos-parallel-1] DEBUG c.a.c.i.GlobalEndpointManager.lambda$startRefreshLocationTimerAsync$11 - startRefreshLocationTimerAsync() - Invoking refresh, I was registered on [2022-06-25T14:17:26.956854]
25-06-2022 14:17:26.983 [cosmos-parallel-1] INFO  c.a.c.i.RxDocumentClientImpl.getDatabaseAccountFromEndpoint - Getting database account endpoint from https://localhost:54950
25-06-2022 14:17:27.061 [cosmos-parallel-1] INFO  c.a.c.i.ImplementationBridgeHelpers.setCosmosDiagnosticsAccessor - Setting CosmosDiagnosticsAccessor...
25-06-2022 14:18:27.380 [reactor-http-kqueue-1] WARN  r.n.http.client.HttpClientConnect.warn - [e00b6593-1, L:/127.0.0.1:55053 - R:localhost/127.0.0.1:54950] The connection observed an error
io.netty.handler.timeout.ReadTimeoutException: null
25-06-2022 14:18:27.383 [reactor-http-kqueue-1] ERROR c.a.c.i.RxGatewayStoreModel.lambda$toDocumentServiceResponse$3 - Network failure
io.netty.handler.timeout.ReadTimeoutException: null
25-06-2022 14:18:27.385 [reactor-http-kqueue-1] INFO  c.a.c.i.ImplementationBridgeHelpers.setCosmosExceptionAccessor - Setting CosmosExceptionAccessor...
25-06-2022 14:18:27.398 [reactor-http-kqueue-1] WARN  c.a.c.i.RxDocumentClientImpl.lambda$getDatabaseAccountFromEndpoint$147 - Failed to retrieve database account information. io.netty.handler.timeout.ReadTimeoutException
25-06-2022 14:18:27.459 [reactor-http-kqueue-1] ERROR c.a.c.i.GlobalEndpointManager.lambda$getDatabaseAccountFromAnyLocationsAsync$2 - Fail to reach global gateway [https://localhost:54950], [{"innerErrorMessage":null,"cosmosDiagnostics":{"userAgent":"azsdk-java-cosmos/4.31.0 MacOSX/12.4 JRE/11.0.12","activityId":null,"requestLatencyInMs":60332,"requestStartTimeUTC":"2022-06-25T12:17:27.064109Z","requestEndTimeUTC":"2022-06-25T12:18:27.396643Z","responseStatisticsList":[],"supplementalResponseStatisticsList":[],"addressResolutionStatistics":{},"regionsContacted":[],"retryContext":{"statusAndSubStatusCodes":null,"retryCount":0,"retryLatency":0},"metadataDiagnosticsContext":{"metadataDiagnosticList":null},"serializationDiagnosticsContext":{"serializationDiagnosticsList":null},"gatewayStatistics":{"sessionToken":null,"operationType":"Read","resourceType":"DatabaseAccount","statusCode":408,"subStatusCode":10002,"requestCharge":0.0,"requestTimeline":[{"eventName":"connectionCreated","startTimeUTC":"2022-06-25T12:17:27.068561Z","durationInMicroSec":271607},{"eventName":"connectionConfigured","startTimeUTC":"2022-06-25T12:17:27.340168Z","durationInMicroSec":20503},{"eventName":"requestSent","startTimeUTC":"2022-06-25T12:17:27.360671Z","durationInMicroSec":9624},{"eventName":"transitTime","startTimeUTC":"2022-06-25T12:17:27.370295Z","durationInMicroSec":60016179},{"eventName":"received","startTimeUTC":null,"durationInMicroSec":0}],"partitionKeyRangeId":null,"exceptionMessage":null,"exceptionResponseHeaders":"{x-ms-substatus=10002}"},"systemInformation":{"usedMemory":"95911 KB","availableMemory":"4098393 KB","systemCpuLoad":"(2022-06-25T12:18:01.957304Z 12,2%), (2022-06-25T12:18:06.957474Z 11,8%), (2022-06-25T12:18:11.953689Z 15,8%), (2022-06-25T12:18:16.956634Z 13,1%), (2022-06-25T12:18:21.956681Z 13,0%), (2022-06-25T12:18:26.956230Z 13,6%)","availableProcessors":12},"clientCfgs":{"id":1,"machineId":"uuid:dea84019-34f2-4369-8b57-dcb481beab0a","connectionMode":"DIRECT","numberOfClients":1,"connCfg":{"rntbd":null,"gw":"(cps:1000, nrto:PT1M, icto:PT1M, p:false)","other":"(ed: false, cs: false)"},"consistencyCfg":"(consistency: null, mm: true, prgns: [])"}}}]

Additional Information

No response

[fabriciorby]

After 2 days of trying I was able to make it work.

It seems like it's some issue with the SDK or something like that, but here's the dirty fix to make it work with testContainers:

Creating the emulator:

    static {
        try {
            Consumer<CreateContainerCmd> cmd =
                    e -> e.withPortBindings(
                            new PortBinding(Ports.Binding.bindPort(8081), new ExposedPort(8081)),
                            new PortBinding(Ports.Binding.bindPort(10251), new ExposedPort(10251)),
                            new PortBinding(Ports.Binding.bindPort(10252), new ExposedPort(10252)),
                            new PortBinding(Ports.Binding.bindPort(10253), new ExposedPort(10253)),
                            new PortBinding(Ports.Binding.bindPort(10254), new ExposedPort(10254))
                    );
            emulator = new CosmosDBEmulatorContainer(
                DockerImageName.parse("mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator"))
                    .withCreateContainerCmdModifier(cmd)
                    .withExposedPorts(8081, 10251, 10252, 10253, 10254)
                    .withEnv("AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE", InetAddress.getLocalHost().getHostAddress())
                    .withEnv("AZURE_COSMOS_EMULATOR_PARTITION_COUNT", "3")
                    .withEnv("AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE", "true");
            emulator.start();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

Creating the client:

    public static CosmosAsyncClient createClient(final DirectConnectionConfig directConnectionConfig) {
        return new CosmosClientBuilder()
                .endpointDiscoveryEnabled(false)
                .endpoint(emulator.getEmulatorEndpoint())
                .key(emulator.getEmulatorKey())
                .directMode(directConnectionConfig)
                .contentResponseOnWriteEnabled(false)
                .buildAsyncClient();
    }

Would it be valuable for the project it I contribute by raising a PR even using this deprecated e.withPortBindings method? If not, at least a note about this on the Testcontainers Azure module page would be nice.

[fabriciorby]

I had to map to fixed ports, probably related to this issue on cosmos-emulator repo https://github.com/Azure/azure-cosmos-db-emulator-docker/issues/50

[eddumelendez]

Hi @fabriciorby

No need to do all of that. I found that the sdk is looking for port 10251 which is not open on cosmodbemulator and there is no a way to configure it to a random port via env variable, for example. However, you can do something like this.

Add the following dependency com.github.terma:javaniotcpproxy:1.5

CosmosDBEmulatorContainer emulator = new CosmosDBEmulatorContainer(
                DockerImageName.parse("mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest")
)
.withExposedPorts(8081, 10251)
.withEnv("AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE", InetAddress.getLocalHost().getHostAddress())
emulator.start();

StaticTcpProxyConfig tcpProxyConfig = new StaticTcpProxyConfig(10251, emulator.getHost(), emulator.getMappedPort(10251));
            tcpProxyConfig.setWorkerCount(1);
            TcpProxy tcpProxy = new TcpProxy(tcpProxyConfig);
            tcpProxy.start();

I hope this can help. I have provided feedback via email to the cosmodb team about providing a way to override the port via env variable in the sdk.

[fabriciorby]

Hey @eddumelendez, thanks for your reply!

I've tried the code above and it did not work. I tried running by exposing only the port 8081 and 10251 with my solution and it did not work as well.

Tried again proxying 10251, 10252, 10253, 10254 and it was successful, I think it's needed to expose all 4 ports.

I liked this solution, so I'll keep it. 😁👍🏻

By the way, nice to know about this lib.

[eddumelendez]

I think there is a different setup on your side which requires more ports. TBH, don't know much about cosmodb and the suggestion is only a workaround due to it will be fixing ports. We will be taking this issue as an enhancement to support direct mode.

[rodrigolopes]

I had same issue and Fabricio's solution works for me. @fabriciorby to avoid deprecated methods, use e -> e.getHostConfig().withPortBindings