azure-cosmos-db-emulator-docker icon indicating copy to clipboard operation
azure-cosmos-db-emulator-docker copied to clipboard

Published 20 hours ago • verified publisher icon Azure

Container is not ready after startup

Open HeneryHawk opened this issue 1 year ago • 14 comments

Hi I have the problem that often the Linux emulator container is not finished booting, although the console says Started and all partitions have been created according to the console. It doesn't happen every time the container is started, but at least 50% of the time.

I could observe the following: When the container is started and all partitions are created immediately, the container is immediately ready and, for example, the Explorer can be accessed. However, if the creation of the last partition is not completed immediately, the container is not immediately ready, even though Started is printed to the console. It then takes at least 30 seconds or more for the Explorer to be accessible, for example.

The behavior could be observed on several dev notebooks, as well as in our GitLab CI.

We noticed the behavior in connection with Java Testcontainers and the official Azure module contained therein. Our integration tests start a Cosmos emulator container via it. In the official Azure module, the Cosmos container is implemented in such a way that it waits for the Started log to appear in the console before the tests are executed or the emulator certificate is downloaded and added to the Java Keystore. Unfortunately, this leads to errors if Started is printed on the console but the container cannot yet be called and therefore the certificate cannot be downloaded and no connection from the Cosmos Java SDK to the emulator can be established. Since the behavior has been noticed both when using the Java Testconainers Framework and when the container is started via Docker CLI, I assume that this is not related to Testcontainers. And since our integration tests are executed in the GitLab CI, this leads to countless faulty pipelines.

Is this a known error? Is there a way to get more logs? What could be the reason for this?

Screenshots fast startup and container is ready immediately fast startup and container is ready immediately

slow startup and container is not immediately ready slow startup and container is not immediately ready

Desktop (please complete the following information):

  • OS: Linux (Debian, Ubuntu)
  • Version: latest

Docker Images Used:

  • Linux

Arguments && Environment variables to start Docker: docker run -p 8081:8081 -p 10250-10255:10250-10255 -it --rm -e AZURE_COSMOS_EMULATOR_PARTITION_COUNT=3 mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest

Docker Environment

  • Standalone

HeneryHawk avatar Jan 31 '24 07:01 HeneryHawk

Hi, currently the emulator is marked UP as soon as all the partitions are created. The Service (Gateway) which serves the REST APIs and explorer, sometime takes longer to boot, and this could cause the problem you face.

We will try to improve this behavior in next releases. For now please use following script for checking Emulator startup - check_emulator_startup.sh

v1k1 avatar Feb 09 '24 12:02 v1k1

Hi @v1k1, thanks for your reply. We didn't notice this behavior before, only since the end of January we have noticed this. Furthermore, the container starts very slowly, which makes the implementation of tests very time-consuming, as each execution of the tests takes several minutes. Most of the time is taken up by waiting for the container to start successfully. This is very annoying. Please address this issue so that the container is available again more quickly.

HeneryHawk avatar Feb 27 '24 14:02 HeneryHawk

We're seeing the same as @HeneryHawk. We are using the CosmosDB Emulator in our integration tests, sometime early this year we noticed that the runtime of our integration tests had been significantly increased. Further investigation shows that the added runtime for our tests is that the CosmosDB Emulator now takes a lot longer to get ready (i.e. the Gateway is ready to accept requests).

I've always thought that the CosmosDB Emulator was a bit slow to start (up to 30 seconds), but now I'm seeing startup time of about 1m45s on my dev machine for CosmosDB Emulator, so something has changed somewhere to cause this. We are running the CosmosDB Emulator in docker using mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator with 4 partitions.

andersosthus avatar Mar 01 '24 11:03 andersosthus

We're seeing the same as @HeneryHawk. We are using the CosmosDB Emulator in our integration tests, sometime early this year we noticed that the runtime of our integration tests had been significantly increased. Further investigation shows that the added runtime for our tests is that the CosmosDB Emulator now takes a lot longer to get ready (i.e. the Gateway is ready to accept requests).

I've always thought that the CosmosDB Emulator was a bit slow to start (up to 30 seconds), but now I'm seeing startup time of about 1m45s on my dev machine for CosmosDB Emulator, so something has changed somewhere to cause this. We are running the CosmosDB Emulator in docker using mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator with 4 partitions.

I am seeing the same thing. Start up has increased significantly recently.

seanke avatar Mar 28 '24 20:03 seanke

The issue that I am facing is that randomly slows down. This is really time consuming when testing. Please fix this behavior!

pvpmartins avatar Apr 17 '24 19:04 pvpmartins

We also face rather long stratup times. I wonder if there is a way to increase the log level that we at least can see that it is still doing something.

Troubleshooting is hardly possible. I'm trying to startup the emulator for some end-to-end testing in a K8S cluster and cosmosdb is the only big pain point (TLS/SSL only needing special ingress treatment, not even started even after 15mins, no logs about the real status, constant CPU load...).

Danielku15 avatar Jun 26 '24 11:06 Danielku15

@v1k1 @niteshvijay1995 Can you look at this ?

sajeetharan avatar Jul 04 '24 06:07 sajeetharan

Not sure if my problem aligns with the problems of others, but to share some insights:

I am deploying the a pod via this file (I also tried keeping port 8081 - the custom port is just an attempt to see if it changes something).

deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cosmosdb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cosmosdb
  template:
    metadata:
      labels:
        app: cosmosdb
    spec:
      nodeSelector:
        "kubernetes.io/os": linux
      containers:
        - name: cosmosdb
          image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest
          ports:
            - containerPort: 8081
          env:
            - name: AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE
              value: 127.0.0.1
            - name: AZURE_COSMOS_EMULATOR_ARGS
              value: /port=10000
Output of `env` 
root@cosmosdb-7fc76bfb64-fqwgw:/usr/local/bin/cosmos# env
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_SERVICE_PORT=443
HOSTNAME=cosmosdb-7fc76bfb64-fqwgw
PWD=/usr/local/bin/cosmos
AZURE_COSMOS_EMULATOR_ARGS=/port=10000
HOME=/root
KUBERNETES_PORT_443_TCP=tcp://10.xx.0.1:443
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
TERM=xterm
SHLVL=1
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_ADDR=10.xx.0.1
AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE=127.0.0.1
KUBERNETES_SERVICE_HOST=10.xx.0.1
KUBERNETES_PORT=tcp://10.xx.0.1:443
KUBERNETES_PORT_443_TCP_PORT=443
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/env
Output of `kubectl logs` ``` This is an evaluation version. There are [101] days left in the evaluation period. Starting Started 1/11 partitions Started 2/11 partitions Started 3/11 partitions Started 4/11 partitions Started 5/11 partitions Started 6/11 partitions Started 7/11 partitions Started 8/11 partitions Started 9/11 partitions Started 10/11 partitions Started 11/11 partitions Started ```
contents of `/tmp/cosmosdb/gateway.log` 
root@cosmosdb-7fc76bfb64-dkcw7:/usr/local/bin/cosmos# cat /tmp/cosmos/appdata/gateway.log 
DocDBTrace Information: 0 : Starting Gateway WinFab host
DocDBTrace Information: 0 : Starting GatewayService, type Microsoft.Azure.Cosmos.RoutingGateway.EntryPoint.GatewayService
DocDBTrace Information: 0 : GatewayService: BEGIN Initialize
DocDBTrace Information: 0 : GatewayService Initialize
DocDBTrace Information: 0 : BEGIN: GatewayService InitializeServices()
DocDBTrace Information: 0 : Loading configuration package for service manifest GatewayService, code package name GatewayService.Code, configuration package name GatewayService.Config and package version 1.0
DocDBTrace Information: 0 : FabricClientFacade: Acquired 'null' credentials from EmulatorFabricClientCredentialsProvider.
DocDBTrace Information: 0 : FabricClientFacade: Initialized client for endpoint(s) 127.0.0.1:0 for component 'GatewayService'; FriendlyName: 'GatewayService-Tfgt4'; Credentials: <Undefined>
DocDBTrace Information: 0 : GatewayService.InitializeServices aadAuthLoginEndpoint = , aadAuthGraphEndpoint = , aadAuthFirstPartyApplicationId =
DocDBTrace Warning: 0 : AadTokenProviderFactory: Certficate location AzureClientCertificate  was not present in the configuration provider...
DocDBTrace Warning: 0 : AadTokenProviderFactory: Certficate location AzureCosmosDBToAzureNRPAuthCertificate  was not present in the configuration provider...
DocDBTrace Information: 0 : Account : DocumentDBTest, Namespace : Onebox, Tenant: EmulatorFederation, Role: EmulatorNode, RoleInstance: EmulatorNode_IN_1
DocDBTrace Information: 0 : Initializing MDM with MonitoringAccount=DocumentDBTest Namespace=Onebox RegionalMonitoringAccountPrefix=CdbOnebox isRegionalMDMSupported=True.
DocDBTrace Information: 0 : MDM RuntimeEventSource Initialization is Successful
DocDBTrace Information: 0 : MDM ConfigStoreEventSource Initialization is Successful
DocDBTrace Information: 0 : END: GatewayService InitializeServices()
DocDBTrace Information: 0 : Emitting GatewayInit metric with dimensions as ProcessName: Microsoft.Azure.Cosmos.GatewayService, AssemblyName: DocumentDB.GatewayService, AssemblyVersion: 2.14.0
DocDBTrace Information: 0 : GatewayService: END Initialize
DocDBTrace Information: 0 : GatewayService: BEGIN OpenAsync
DocDBTrace Information: 0 : ConfigurationStoreConfigurationProvider: initialization started. CallingComponent: GatewayService, federationId: emulatorfederation, disableConfigurationRefreshTimer: False
DocDBTrace Information: 0 : CompositeConfigurationReader.ReadConfigurationStoreConfigurationAsync, callingComponent = ConfigurationStoreConfigurationProvider
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: ConfigurationStoreConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>
DocDBTrace Information: 0 : CompositeConfigurationReader.ListConfigurationStoreAccountNamesAsync, callingComponent = ConfigurationStoreConfigurationProvider
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumerateSubNamesAsync, ConfigurationLevel: ConfigurationStoreAccountName, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>   
DocDBTrace Information: 0 : ConfigurationStoreConfigurationProvider: Log ConfigurationStoreConfiguration. hasConfigurationChanged False, UseNamingServiceForWritingFederationConfiguration: True, UseCosmosStoreForWritingFederationConfiguration: False, PerformReadValidationForFederationConfiguration: False, UseCosmosStoreForReadingFederationConfiguration: False, EnableFallbackForReadingFederationConfiguration: False, UseNamingServiceForWritingDatabaseAccountConfiguration: True, UseCosmosStoreForWritingDatabaseAccountConfiguration: False, PerformReadValidationForDatabaseAccountConfiguration: False, UseCosmosStoreForReadingDatabaseAccountConfiguration: False, EnableFallbackForReadingDatabaseAccountConfiguration: False, ConfigurationStoreConfigurationRefreshIntervalInSeconds: 300, federationId: emulatorfederation, UseCosmosStoreForWritingFederationVisibilityConfiguration: False, PerformReadValidationForFederationVisibilityConfiguration: False, UseCosmosStoreForReadingFederationVisibilityConfiguration: False, EnableFallbackForReadingFederationVisibilityConfiguration: False, SkipExceptionsFromSecondaryStoreForFederationConfiguration: True, SkipExceptionsFromSecondaryStoreForFederationVisibilityConfiguration: True, SkipExceptionsFromSecondaryStoreForDatabaseAccountConfiguration: True, ConfigurationStoreAccountNames:
DocDBTrace Information: 0 : ConfigurationStoreConfigurationProvider initialization ended GatewayService, federationId: emulatorfederation, disableConfigurationRefreshTimer: False
DocDBTrace Information: 0 : GatewayService: OpenInternalAsync: ConfigurationStoreConfigurationProvider initialized successfully
DocDBTrace Information: 0 : ReadAllFederationLevelConfigs: Invoked from InitializeAsync
DocDBTrace Information: 0 : CompositeConfigurationReader.ReadFederationConfigAsync, federationId = emulatorfederation, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: FederationConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>       
DocDBTrace Information: 0 : ReadFederationLevelConfigFromConfigStoreAsync: Generate compositeETag ySVnvyI88H+1wQCDn0pI4Phbao6Jj7MCHSqmaiXnVP4= for local federation decoupleAvailableBindableService True
DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: ReadFederationLevelConfigFromConfigStoreAsync, IsSuccess: True
DocDBTrace Information: 0 : FederationConfiguration initialized property SecondarySystemKeyReadWriteEncrypted
DocDBTrace Information: 0 : FederationConfiguration initialized property federationEndpoint = http://127.0.0.1/
DocDBTrace Information: 0 : Decrypting message with subject key identifier(s): N/A
DocDBTrace Information: 0 : FederationConfiguration initialized primary compute key with hash = 7B76D81B9CB723B1D0FEF518331D8DC853E8899AC0D13AF87FD1BE357CC835C8
DocDBTrace Information: 0 : FederationConfiguration initialized property UsePrimarySystemKeyReadWrite = true
DocDBTrace Information: 0 : Decrypting message with subject key identifier(s): N/A
DocDBTrace Information: 0 : FederationConfiguration initialized secondary compute key SecondaryComputeGatewayKeyEncrypted with hash = 38C6CD57974A7E24C49321C44F7D94463F21DAB1FC1C99B2164184486AC09ABA
DocDBTrace Information: 0 : FederationConfiguration initialized property resourceCacheRefreshIntervalInSeconds = 60
DocDBTrace Information: 0 : FederationConfiguration initialized property availableBindableServices
DocDBTrace Information: 0 : FederationConfiguration initialized property UsePrimarySystemKeyReadOnly = true
DocDBTrace Information: 0 : FederationConfiguration initialized property serviceCosmosDBGatewayServiceRunningMode = Idle
DocDBTrace Information: 0 : FederationConfiguration initialized property replicatorUpgradePhase = 5
DocDBTrace Information: 0 : FederationConfiguration initialized property IsGremlinEndpointEnabled = false
DocDBTrace Information: 0 : FederationConfiguration initialized property IsMongoTcpEndpointEnabled = false
DocDBTrace Information: 0 : FederationConfiguration initialized property federationGenerationKind = gen3
DocDBTrace Information: 0 : FederationConfiguration initialized property nodeCountOfRoleHostingGateway = 1
DocDBTrace Information: 0 : FederationConfiguration initialized property isStrictServiceStateCheckEnabled = true
DocDBTrace Information: 0 : FederationConfiguration initialized property federationName = emulatorfederation
DocDBTrace Information: 0 : FederationConfiguration initialized property configurationRefreshIntervalInSec = 30
DocDBTrace Information: 0 : FederationConfiguration initialized property UsePrimarySystemKeyAll = true
DocDBTrace Information: 0 : FederationConfiguration initialized property gatewayNamingServiceConfigRefreshIntervalInSeconds = 30
DocDBTrace Information: 0 : FederationConfiguration initialized property IsSqlComputeEndpointEnabled = false
DocDBTrace Information: 0 : FederationConfiguration initialized property IsTableEndpointEnabled = false
DocDBTrace Information: 0 : FederationConfiguration initialized property region = South Central US
DocDBTrace Information: 0 : FederationConfiguration initialized property enableRG = false
DocDBTrace Information: 0 : FederationConfiguration initialized property reservedDnsName = https://localhost:443/
DocDBTrace Information: 0 : FederationConfiguration initialized property PrimarySystemKeyReadOnlyEncrypted
DocDBTrace Information: 0 : FederationConfiguration initialized property isServiceReuseEnabled = true
DocDBTrace Information: 0 : FederationConfiguration initialized property PrimarySystemKeyAllEncrypted
DocDBTrace Information: 0 : FederationConfiguration initialized property isAvailabilityZoneFederation = false
DocDBTrace Information: 0 : FederationConfiguration initialized property siblingFederations = [{"SiblingFederationId":"emulatorfederation","FabricClientEndpoints":["127.0.0.1:9000"],"FabricClientAdminAccessCertificateCommonName":"","EnabledForCrossFederationDatabaseAccountScaleout":true}]
DocDBTrace Information: 0 : FederationConfiguration initialized property IsCassandraTcpEndpointEnabled = false
DocDBTrace Information: 0 : FederationConfiguration initialized property isCrossFederationDatabaseAccountScaleoutEnabled = true
DocDBTrace Information: 0 : FederationConfiguration initialized property IsDataTransferControlPlaneEndpointEnabled = false
DocDBTrace Information: 0 : FederationConfiguration initialized property SecondarySystemKeyAllEncrypted
DocDBTrace Information: 0 : FederationConfiguration initialized property SecondarySystemKeyReadOnlyEncrypted
DocDBTrace Information: 0 : FederationConfiguration initialized property PrimarySystemKeyReadWriteEncrypted
DocDBTrace Information: 0 : ReadAllFederationLevelConfigs: ETag: old , current ySVnvyI88H+1wQCDn0pI4Phbao6Jj7MCHSqmaiXnVP4=
DocDBTrace Information: 0 : Decrypting message with subject key identifier(s): N/A
DocDBTrace Information: 0 : Decrypting message with subject key identifier(s): N/A
DocDBTrace Information: 0 : Decrypting message with subject key identifier(s): N/A
DocDBTrace Warning: 0 : InitializeKeysInFederationConfiguration: Failed querying federationVisibilityConfigurationWithAllKeys for sibling federation emulatorfederation
DocDBTrace Information: 0 : InitializeKeysInFederationConfiguration: Completed initializing systemkeys for local federation
DocDBTrace Information: 0 : GetSiblingFederations : Skipping sibling federation config fetch in gateway boot ReadAllFederationLevelConfigsAsync
DocDBTrace Information: 0 : CompositeConfigurationReader.ListPrivateEndpointMapsAsync, region = southcentralus, callingComponent = GatewayService-GatewayConfigurationReaderProvider
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: PrivateEndpointMapsConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>
DocDBTrace Information: 0 : GetSiblingFederations : Skipping sibling federation config fetch in gateway boot RefreshAllAvailableBindableServicesAsync
DocDBTrace Information: 0 : InvokeCallbackForAllFederationsInRegionAsync invoked, taskName RefreshAllAvailableBindableServicesAsync, callingComponent InitializeAsync federations emulatorfederation
DocDBTrace Information: 0 : CompositeConfigurationReader ReadFederationPropertyAsync: federationId = emulatorfederation, propertyKey = availableBindableServices, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: GetPropertyAsync, ConfigurationLevel: FederationConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>
DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: RefreshAllAvailableBindableServicesAsync, IsSuccess: True
DocDBTrace Information: 0 : GetSiblingFederations : Skipping sibling federation config fetch in gateway boot RefreshAllServiceManagerConfigurationsAsync
DocDBTrace Information: 0 : InvokeCallbackForAllFederationsInRegionAsync invoked, taskName RefreshAllServiceManagerConfigurationsAsync, callingComponent InitializeAsync federations emulatorfederation
DocDBTrace Information: 0 : CompositeConfigurationReader.ReadFabricServiceManagerConfigurationsAsync, serviceTypeName = ServerService, isEmulated = True, federationId = emulatorfederation, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumerateSubNamesAsync, ConfigurationLevel: FabricServiceConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>      
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: FabricServiceConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>    
DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: RefreshAllServiceManagerConfigurationsAsync, IsSuccess: True
DocDBTrace Information: 0 : RefreshFederationCapacityInformation. Using default value of all federations emulatorfederation,emulatorfederation
DocDBTrace Information: 0 : RequestSamplingConfigurations changed. enableRequestSampling: true, minimumRequestSamplingLatencyThresholdInMilliseconds: 2500, requestSamplingCategories: Document,DocumentFeed, requestSamplingPercent: 30
DocDBTrace Information: 0 : CompositeConfigurationReader ListOffersConfigurationAsync callingComponent = GatewayService-GatewayConfigurationReaderProvider
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumerateSubNamesAsync, ConfigurationLevel: OffersConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: OffersConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>
DocDBTrace Information: 0 : Starting periodic timer for callingComponent PeriodicallyRefreshFederationConfigurationAsync with refresh time(sec) 38
DocDBTrace Information: 0 : Starting periodic timer for callingComponent StartPrivateEndpointMapConfigurationRefreshTimerAsync with refresh time(sec) 30
DocDBTrace Information: 0 : GatewayService: OpenInternalAsync: ConfigurationProvider initialized local federation configuration successfully
DocDBTrace Information: 0 : GatewayService: OpenInternalAsync: ConfigurationProvider initialized successfully
DocDBTrace Information: 0 : GatewayWarmpathAggregator: requestaggregation is disabled
DocDBTrace Information: 0 : Initializing MutualTlsClient with the following configuration:
                cloudName=Microsoft
                enableMtlsAuthentication=True
                allowedSubjectNames=cosmosdbemulatormtls.localhost,client.mtls.cdb.core.azure-test.net
                deniedThumbprints=
                allowedThumbprints=4E30B33D02ABE0CFED415A722F45FBF361BE243F
                optionalBase64EncodedAllowedIssuersCount=0
DocDBTrace Information: 0 : Refresh private endpoint map start
DocDBTrace Information: 0 : CompositeConfigurationReader.ListPrivateEndpointMapsAsync, region = southcentralus, callingComponent = GatewayService-GatewayConfigurationReaderProvider
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: PrivateEndpointMapsConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>
DocDBTrace Warning: 0 : Private Endpoint Map Refresh fails. Will retry in the next cycle.
DocDBTrace Information: 0 : Starting periodic timer for callingComponent StartPrivateEndpointMapConfigurationRefreshTimerAsync with refresh time(sec) 30
DocDBTrace Information: 0 : GatewayService: OpenInternalAsync: MutualTlsClient initialized: True
DocDBTrace Information: 0 : NetworkIPRangePolicy Intializing IP ranges for 0, with C:\Packages\GatewayService\GatewayService.Code\TestNetworksFromVM.xml last written at: 05/21/2024 13:29:12.
DocDBTrace Information: 0 : Networks.xml file path is C:\Packages\GatewayService\GatewayService.Code\TestNetworksFromVM.xml
DocDBTrace Information: 0 : NetworkIPRangePolicy.SetRegionalIPRanges() Count = 676
DocDBTrace Information: 0 : NetworkIPRangePolicy.SetAllAzureIpRanges() Count = 10417
DocDBTrace Information: 0 : regionName = South Central US, IsEmulated = True, regionId is set to 35
DocDBTrace Information: 0 : GatewayService: OpenInternalAsync: NetworkIPRangePolicy initialized successfully
DocDBTrace Information: 0 : GatewayService OpenAsync configurations - RequestTimeoutInSeconds: 10, MaxConcurrentConnectionOpenRequestsPerProcessor: 25, ProcessorCount: 4, BackendConnectionOpenTimeoutInSeconds: 1, IdleConnectionTimeoutInSeconds: 1800, TransportTimerPoolGranularityInSeconds: 1, MaxRntbdChannels: 65535, RntbdPartitionCount: 1, MaxRequestsPerRntbdChannel: 12, PortResuseMode: PrivatePortPool, RntbdPortPoolReuseThreshold: 256, RntbdPortPoolBindAttempts: 5, RntbdChannelReceiveHangDetectionTimeSeconds: 65, RntbdChannelSendHangDetectionTimeSeconds: 10, GatewayClientRequestTimeoutInSeconds: 60, UseGatewayToReadDatabaseAccountConfiguration: False, HttpEndPointPort: 10000, HttpEndPointProbePort: 81, FabricNameResolutionTimeoutInSeconds: -1, NameresolutionTimeoutInSeconds: 10, IsGatewayLocalhostOnlyString: false, IPv6ListenersEnabledForEmulator: False. IsMongoProxyEnabled: FalseHttpEndPointProbePathSettingName: /probe, RegionName: South Central US, ProxyEndPointProbePathSettingName: /proxyprobe, MaxServicePoolProbeCount: 25InitialRetryIntervalMilliseconds: 10, MaximumRetryIntervalMilliseconds: 1000, RandomSaltMaxValueMilliseconds: 5, TotalWaitTimeMilliseconds: 30000, EnableStorageServiceConfigurationLoadOnBoot False IsKeyBasedEncryptionEnabledForStorage: False, EnableMutualTlsServerSide: True, EnableMutualTlsServerSideAuthorization: True, EnableMutualTlsClientSideUsage: False
DocDBTrace Information: 0 : Setting up Distributed tracing
DocDBTrace Information: 0 : Enabling/Disabling Distributed Tracing for SDK: True
DocDBTrace Information: 0 : CertificateUtil: Returning certificate CN=localhost(4E30B33D02ABE0CFED415A722F45FBF361BE243F) for FindByThumbprint '4E30B33D02ABE0CFED415A722F45FBF361BE243F' from LocalMachine\My
DocDBTrace Information: 0 : Creating RNTBD TransportClient with options Rntbd.TransportClient.Options
  OpenTimeout: 00:00:01
  RequestTimeout: 00:00:10
  TimerPoolResolution: 00:00:01
  MaxChannels: 65535
  PartitionCount: 1
  MaxRequestsPerChannel: 12
  ReceiveHangDetectionTime: 00:01:05
  SendHangDetectionTime: 00:00:10
  IdleTimeout: 00:30:00
  UserAgent: Microsoft.Azure.Documents.Common/2.14.0RoutingGateway Suffix: RoutingGateway
  CertificateHostNameOverride: localhost
  LocalRegionTimeout: 00:00:01
  EnableChannelMultiplexing: False
  MaxConcurrentOpeningConnectionCount: 65535
  Use_RecyclableMemoryStream: False
  Use_CustomDnsResolution: False
  IsDistributedTracingEnabled: True

DocDBTrace Information: 0 : TimerPool Created with minSupportedTimerDelayInSeconds = 1
DocDBTrace Information: 0 : TimerPool Created with minSupportedTimerDelayInSeconds = 30
DocDBTrace Information: 0 : Creating RNTBD TransportClient with options Rntbd.TransportClient.Options
  OpenTimeout: 00:00:01
  RequestTimeout: 00:00:10
  TimerPoolResolution: 00:00:01
  MaxChannels: 65535
  PartitionCount: 1
  MaxRequestsPerChannel: 1
  ReceiveHangDetectionTime: 00:01:05
  SendHangDetectionTime: 00:00:10
  IdleTimeout: 00:30:00
  UserAgent: Microsoft.Azure.Documents.Common/2.14.0RoutingGateway Suffix: RoutingGateway
  CertificateHostNameOverride: localhost
  LocalRegionTimeout: 00:00:01
  EnableChannelMultiplexing: False
  MaxConcurrentOpeningConnectionCount: 65535
  Use_RecyclableMemoryStream: False
  Use_CustomDnsResolution: False
  IsDistributedTracingEnabled: True

DocDBTrace Information: 0 : TimerPool Created with minSupportedTimerDelayInSeconds = 1
DocDBTrace Information: 0 : TimerPool Created with minSupportedTimerDelayInSeconds = 30
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: CreateNameAsync, ConfigurationLevel: <empty>, IsWriteRequest: True, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>
DocDBTrace Information: 0 : GatewayService: OpenInternalAsync: Transaction log manager initialized successfully
DocDBTrace Information: 0 : Started listening on address: https://0.0.0.0:10000/
DocDBTrace Information: 0 : Started listening on probe address: http://0.0.0.0:81/
DocDBTrace Information: 0 : BEGIN: StartAsync()
DocDBTrace Information: 0 : Listening on: https://0.0.0.0:10000/
DocDBTrace Information: 0 : MaxAttachmentContentSize config setting is 0
DocDBTrace Information: 0 : SingleBlobUploadThresholdInBytes config setting is 8388608 bytes
DocDBTrace Information: 0 : AzureBlobUploadTimeoutSeconds config setting is 00:01:30 seconds
DocDBTrace Information: 0 : MaxAttachmentContentSize config setting is 0
DocDBTrace Information: 0 : SingleBlobUploadThresholdInBytes config setting is 8388608 bytes
DocDBTrace Information: 0 : AzureBlobUploadTimeoutSeconds config setting is 00:01:30 seconds
DocDBTrace Information: 0 : SingleBlobUploadThresholdInBytes config setting is 8388608 bytes
DocDBTrace Information: 0 : AzureBlobUploadTimeoutSeconds config setting is 00:01:30 seconds
DocDBTrace Information: 0 : AzureBlobDownloadTimeoutSeconds config setting is 00:01:30 seconds
DocDBTrace Information: 0 : HttpRequestDispatcher.UpdateServerTimeouts: Updating DrainEntityBody from default to 120
DocDBTrace Information: 0 : HttpRequestDispatcher.UpdateServerTimeouts: Updating EntityBody from default to 120
DocDBTrace Information: 0 : HttpRequestDispatcher.UpdateServerTimeouts: Updating MinSendBytesPerSecond from default to 8192
DocDBTrace Information: 0 : HttpRequestDispatcher.UpdateServerTimeouts: Updating RequestQueue from default to 120
DocDBTrace Information: 0 : HttpRequestDispatcher.UpdateServerTimeouts: Updating HeaderWait from default to 30
DocDBTrace Information: 0 : HttpRequestDispatcher.UpdateServerTimeouts: Updating IdleConnection from default to 120
DocDBTrace Information: 0 : HttpRequestDispatcher.UpdateServerTimeouts: New timeouts: HttpTimeoutLimitInfo { DrainEntityBodyInSeconds = 120, EntityBodyInSeconds = 120, HeaderWaitInSeconds = 30, IdleConnectionInSeconds = 120, MinSendBytesPerSecond = 8192, RequestQueueInSeconds = 120 }
DocDBTrace Information: 0 : Enabling DistributedTracing for HttpRequestDispatcher, Master Switch ON
DocDBTrace Information: 0 : HttpRequestDispatcher: HttpListener started on 1 prefixes. (HttpTimeoutLimitInfo { DrainEntityBodyInSeconds = 120, EntityBodyInSeconds = 120, HeaderWaitInSeconds = 30, IdleConnectionInSeconds = 120, MinSendBytesPerSecond = 8192, RequestQueueInSeconds = 120 })
DocDBTrace Information: 0 : END: StartAsync()
DocDBTrace Information: 0 : GatewayRequestProcessor useGatewayToReadDatabaseAccountConfiguration = False
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: CreateNameAsync, ConfigurationLevel: <empty>, IsWriteRequest: True, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: CreateNameAsync, ConfigurationLevel: <empty>, IsWriteRequest: True, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>
DocDBTrace Information: 0 : GatewayService: OpenInternalAsync: GatewayRequestProcessor initialized successfully
DocDBTrace Information: 0 : Emitting NumberOfEntriesInGatewayCache metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, CacheProviderTypeName: GatewayDatabaseAccountConfigurationProviderV2
DocDBTrace Information: 0 : Emitting NumberOfEntriesInGatewayCache metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, CacheProviderTypeName: GatewayDatabaseAccountBackendResourceProvider
DocDBTrace Information: 0 : Emitting NumberOfEntriesInGatewayCache metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, CacheProviderTypeName: GatewayDatabaseAccountConfigurationProvider
DocDBTrace Information: 0 : Emitting NumberOfEntriesInGatewayCache metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, CacheProviderTypeName: DatabaseAccountRequestProcessorFactory
DocDBTrace Information: 0 : GatewayService: OpenInternalAsync: GatewayCacheMetricsPublisher initialized successfully
DocDBTrace Information: 0 : Emitting RoutingGatewayOpen metric with dimensions as IsSuccess: True
DocDBTrace Information: 0 : Emitting RoutingGatewayOpenLatency metric with no additional dimensions.
DocDBTrace Information: 0 : GatewayService: END OpenAsync
DocDBTrace Information: 0 : Emitting FederationConfigurationRefresh metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore
DocDBTrace Information: 0 : ReadAllFederationLevelConfigs: Invoked from PeriodicallyRefreshFederationConfigurationAsync
DocDBTrace Information: 0 : CompositeConfigurationReader.ReadFederationConfigAsync, federationId = emulatorfederation, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: FederationConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>       
DocDBTrace Information: 0 : ReadFederationLevelConfigFromConfigStoreAsync: Generate compositeETag ySVnvyI88H+1wQCDn0pI4Phbao6Jj7MCHSqmaiXnVP4= for local federation decoupleAvailableBindableService True
DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: ReadFederationLevelConfigFromConfigStoreAsync, IsSuccess: True
DocDBTrace Information: 0 : ReadAllFederationLevelConfigs: ETag: old ySVnvyI88H+1wQCDn0pI4Phbao6Jj7MCHSqmaiXnVP4=, current ySVnvyI88H+1wQCDn0pI4Phbao6Jj7MCHSqmaiXnVP4=
DocDBTrace Information: 0 : ReadAllFederationLevelConfigs: Skipping sibling federation config refresh
DocDBTrace Information: 0 : InvokeCallbackForAllFederationsInRegionAsync invoked, taskName RefreshAllAvailableBindableServicesAsync, callingComponent PeriodicallyRefreshFederationConfigurationAsync federations emulatorfederation,emulatorfederation
DocDBTrace Information: 0 : CompositeConfigurationReader ReadFederationPropertyAsync: federationId = emulatorfederation, propertyKey = availableBindableServices, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: GetPropertyAsync, ConfigurationLevel: FederationConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>
DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: RefreshAllAvailableBindableServicesAsync, IsSuccess: True
DocDBTrace Information: 0 : CompositeConfigurationReader ReadFederationPropertyAsync: federationId = emulatorfederation, propertyKey = availableBindableServices, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: GetPropertyAsync, ConfigurationLevel: FederationConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>
DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: RefreshAllAvailableBindableServicesAsync, IsSuccess: True
DocDBTrace Information: 0 : InvokeCallbackForAllFederationsInRegionAsync invoked, taskName RefreshAllServiceManagerConfigurationsAsync, callingComponent PeriodicallyRefreshFederationConfigurationAsync federations emulatorfederation,emulatorfederation
DocDBTrace Information: 0 : CompositeConfigurationReader.ReadFabricServiceManagerConfigurationsAsync, serviceTypeName = ServerService, isEmulated = True, federationId = emulatorfederation, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumerateSubNamesAsync, ConfigurationLevel: FabricServiceConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>      
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: FabricServiceConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>    
DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: RefreshAllServiceManagerConfigurationsAsync, IsSuccess: True
DocDBTrace Information: 0 : CompositeConfigurationReader.ReadFabricServiceManagerConfigurationsAsync, serviceTypeName = ServerService, isEmulated = True, federationId = emulatorfederation, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumerateSubNamesAsync, ConfigurationLevel: FabricServiceConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>      
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: FabricServiceConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>    
DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: RefreshAllServiceManagerConfigurationsAsync, IsSuccess: True
DocDBTrace Information: 0 : Updating federation: emulatorfederation with service type: ServerService and bindable capacity: 5
DocDBTrace Information: 0 : Updating federation: emulatorfederation with service type: MasterService and bindable capacity: 5
DocDBTrace Information: 0 : CompositeConfigurationReader ListOffersConfigurationAsync callingComponent = GatewayService-GatewayConfigurationReaderProvider
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumerateSubNamesAsync, ConfigurationLevel: OffersConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: OffersConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>
DocDBTrace Information: 0 : Refreshing Mutual TLS configurations.
DocDBTrace Information: 0 : Starting periodic timer for callingComponent PeriodicallyRefreshFederationConfigurationAsync with refresh time(sec) 38
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: cpu-usage, Type: Mean, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: working-set, Type: Mean, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: gc-heap-size, Type: Mean, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: gen-0-gc-count, Type: Sum, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: gen-1-gc-count, Type: Sum, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: gen-2-gc-count, Type: Sum, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: threadpool-thread-count, Type: Mean, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: monitor-lock-contention-count, Type: Sum, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: threadpool-queue-length, Type: Mean, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: threadpool-completed-items-count, Type: Sum, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: alloc-rate, Type: Sum, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: active-timer-count, Type: Mean, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: gc-fragmentation, Type: Mean, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: gc-committed, Type: Mean, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: exception-count, Type: Sum, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: time-in-gc, Type: Mean, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: gen-0-size, Type: Mean, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: gen-1-size, Type: Mean, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: gen-2-size, Type: Mean, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: loh-size, Type: Mean, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: poh-size, Type: Mean, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: assembly-count, Type: Mean, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: il-bytes-jitted, Type: Mean, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: methods-jitted-count, Type: Mean, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Emitting ServiceEventCounter metric with dimensions as Name: time-in-jit, Type: Sum, ServiceName: GatewayService, ProcessId: 452
DocDBTrace Information: 0 : Refresh private endpoint map start
DocDBTrace Information: 0 : CompositeConfigurationReader.ListPrivateEndpointMapsAsync, region = southcentralus, callingComponent = GatewayService-GatewayConfigurationReaderProvider
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: PrivateEndpointMapsConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>
DocDBTrace Warning: 0 : Private Endpoint Map Refresh fails. Will retry in the next cycle.
DocDBTrace Information: 0 : Starting periodic timer for callingComponent StartPrivateEndpointMapConfigurationRefreshTimerAsync with refresh time(sec) 30
DocDBTrace Information: 0 : Emitting FederationConfigurationRefresh metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore
DocDBTrace Information: 0 : ReadAllFederationLevelConfigs: Invoked from PeriodicallyRefreshFederationConfigurationAsync
DocDBTrace Information: 0 : CompositeConfigurationReader.ReadFederationConfigAsync, federationId = emulatorfederation, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: FederationConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>       
DocDBTrace Information: 0 : ReadFederationLevelConfigFromConfigStoreAsync: Generate compositeETag ySVnvyI88H+1wQCDn0pI4Phbao6Jj7MCHSqmaiXnVP4= for local federation decoupleAvailableBindableService True
DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: ReadFederationLevelConfigFromConfigStoreAsync, IsSuccess: True
DocDBTrace Information: 0 : ReadAllFederationLevelConfigs: ETag: old ySVnvyI88H+1wQCDn0pI4Phbao6Jj7MCHSqmaiXnVP4=, current ySVnvyI88H+1wQCDn0pI4Phbao6Jj7MCHSqmaiXnVP4=
DocDBTrace Information: 0 : ReadAllFederationLevelConfigs: Skipping sibling federation config refresh
DocDBTrace Information: 0 : InvokeCallbackForAllFederationsInRegionAsync invoked, taskName RefreshAllAvailableBindableServicesAsync, callingComponent PeriodicallyRefreshFederationConfigurationAsync federations emulatorfederation,emulatorfederation
DocDBTrace Information: 0 : CompositeConfigurationReader ReadFederationPropertyAsync: federationId = emulatorfederation, propertyKey = availableBindableServices, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: GetPropertyAsync, ConfigurationLevel: FederationConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>
DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: RefreshAllAvailableBindableServicesAsync, IsSuccess: True
DocDBTrace Information: 0 : CompositeConfigurationReader ReadFederationPropertyAsync: federationId = emulatorfederation, propertyKey = availableBindableServices, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: GetPropertyAsync, ConfigurationLevel: FederationConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>
DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: RefreshAllAvailableBindableServicesAsync, IsSuccess: True
DocDBTrace Information: 0 : InvokeCallbackForAllFederationsInRegionAsync invoked, taskName RefreshAllServiceManagerConfigurationsAsync, callingComponent PeriodicallyRefreshFederationConfigurationAsync federations emulatorfederation,emulatorfederation
DocDBTrace Information: 0 : CompositeConfigurationReader.ReadFabricServiceManagerConfigurationsAsync, serviceTypeName = ServerService, isEmulated = True, federationId = emulatorfederation, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumerateSubNamesAsync, ConfigurationLevel: FabricServiceConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>      
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: FabricServiceConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>    
DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: RefreshAllServiceManagerConfigurationsAsync, IsSuccess: True
DocDBTrace Information: 0 : CompositeConfigurationReader.ReadFabricServiceManagerConfigurationsAsync, serviceTypeName = ServerService, isEmulated = True, federationId = emulatorfederation, callingComponent = ConfigurationProvider-GatewayConfigurationReaderProvider
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumerateSubNamesAsync, ConfigurationLevel: FabricServiceConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>      
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: FabricServiceConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>    
DocDBTrace Information: 0 : Emitting ReadFederationConfigurationResult metric with dimensions as _microsoft_metrics_namespace: Onebox/ConfigStore, FederationName: emulatorfederation, CallingComponent: RefreshAllServiceManagerConfigurationsAsync, IsSuccess: True
DocDBTrace Information: 0 : Updating federation: emulatorfederation with service type: ServerService and bindable capacity: 5
DocDBTrace Information: 0 : Updating federation: emulatorfederation with service type: MasterService and bindable capacity: 5
DocDBTrace Information: 0 : CompositeConfigurationReader ListOffersConfigurationAsync callingComponent = GatewayService-GatewayConfigurationReaderProvider
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumerateSubNamesAsync, ConfigurationLevel: OffersConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>
DocDBTrace Information: 0 : Emitting ManagedFabricCallVolume metric with dimensions as Component: GatewayService, FabricModule: PropertyManager, FabricMethod: EnumeratePropertiesAsync, ConfigurationLevel: OffersConfiguration, IsWriteRequest: False, FabricEndpoints: 127.0.0.1:0, Status: True, errorCode: 0, exceptionType: <empty>
DocDBTrace Information: 0 : Refreshing Mutual TLS configurations.
DocDBTrace Information: 0 : Starting periodic timer for callingComponent PeriodicallyRefreshFederationConfigurationAsync with refresh time(sec) 38
Output of lsof -i 
root@cosmosdb-775768b69b-h78h5:/usr/local/bin/cosmos# lsof -i
COMMAND   PID USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
cosmosdb-  19 root   73u  IPv4 951851706      0t0  TCP *:10251 (LISTEN)
cosmosdb-  19 root   75u  IPv4 951853360      0t0  TCP *:10253 (LISTEN)

lsof -i :8081 (or port 10000) doesn't list anything. curl --insecure https://localhost:8081 (or port 10000) fails to call. But this doesn't seem to be a general issue. On our Rancher K8S Cluster it doesn't work, on docker-desktop CosmosDB starts listening.

Using top I can see the cosmosdb emulator running and consuming 10-20% CPU.

Danielku15 avatar Jul 04 '24 12:07 Danielku15

@v1k1 @niteshvijay1995 Can you look at this ?

Is there an ETA?

aratz-lasa avatar Jul 10 '24 10:07 aratz-lasa

It's really slow to be ready ... after the container shows started it still takes a while. I wrote this little bash oneliner to ensure Cosmos DB has started completely.

echo -n "Waiting for Cosmos DB emulator ..."; while true; do result=$(curl -s -k https://localhost:8081 | jq -r '.code'); if [ "$result" = "Unauthorized" ]; then break; fi; echo -n "."; sleep 2; done; echo " ready."

And it produces this output when run with time command:

Waiting for Cosmos DB emulator ........................................................ ready.

real    1m47.743s
user    0m0.941s
sys     0m0.098s

And that's on a i7 12700k, the database is basically empty. (2 datasets).

@aratz-lasa ETA would be 2 minutes then

HEllRZA avatar Jul 19 '24 07:07 HEllRZA

I experience the same really slow startup. It is a real pain as sometimes I have to wait 5 to 10 minutes until I can debug one line of code. Reducing the partition count to 1 did improve the time to start a bit but not really as much as I hoped.

Current setup:

  • CPU: AMD Ryzen 7 3700X 8-Core 4.05GHz
  • RAM: 4x8GB DDR4 2133MHz
  • SSD: Samsung SSD 860 QVO 2TB (Read 550MB/s, Write 520MB/s)
  • Docker Desktop - 4.32.0 -> No other containers are running
  • WSL 2
  • Windows 11 - No Outstanding Updates
  • Visual Studio 2022 17.10.4 64-bit

I'm using Aspire 8.0.2 and I have a small application that uses CosmosDB for persistence. I want to run IntegrationTests using the DistributedApplicationTestingBuilder. I currently have to poll the "/alive" endpoint and added AzureCosmosDB-HealthCheck that has the "live"-Tag, so the application is only considered alive / running when the cosmos emulator instance is accessible.

On the other hand the Azurite-Instance that I have to run as well is up almost immediately.

using Projects;

var builder = DistributedApplication.CreateBuilder(args);

var azureStorage = builder.AddAzureStorage("storage")
                    .RunAsEmulator(r =>
                        r.WithArgs("azurite", "-l", "/data", "--blobHost", "0.0.0.0", "--queueHost", "0.0.0.0", "--tableHost", "0.0.0.0", "--skipApiVersionCheck")
                    );

// ## Synchronization

// Aspire currently only supports Azure Storage for Orleans.
var synchronizationClusterStore = azureStorage.AddTables("synchronization-cluster-table");
var synchronizationStateStore = azureStorage.AddBlobs("synchronization-grains-state");

var synchronizationOrleans = builder.AddOrleans("synchronization-cluster")
                     .WithClustering(synchronizationClusterStore)
                     .WithGrainStorage(synchronizationStateStore)
                     ;

var synchronizationInboxDatabase = builder.AddAzureCosmosDB("SynchronizationInboxDatabase")
    .RunAsEmulator(r => r
        .WithEnvironment("AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE", "127.0.0.1")
        .WithEnvironment("AZURE_COSMOS_EMULATOR_PARTITION_COUNT", "1")
    );

builder.AddProject<Synchronization_Silo>("synchronization")
    .WithReference(synchronizationOrleans)
    .WithReference(synchronizationInboxDatabase)
    .WithExternalHttpEndpoints()
    .WithReplicas(1)
    .WithEnvironment("ASPNETCORE_ENVIRONMENT", builder.Environment.EnvironmentName)
    .WithEnvironment("DOTNET_ENVIRONMENT", builder.Environment.EnvironmentName)
    ;

using var app = builder.Build();
await app.RunAsync();

Cotspheer avatar Jul 21 '24 17:07 Cotspheer

Regarding the very slow startup, we have implemented a workaround that pings /_explorer/emulator.pem until this URL is accessible. This has now worked well for a while, but further errors have been occurring for a few weeks now. For inexplicable reasons, our Gradle tests lose the connection to the emulator during execution in the pipeline. So the tests fail and so does the pipeline.

ERROR  c.a.c.i.GlobalEndpointManager - Fail to reach global gateway [https://172.17.0.2:8081/]
ERROR  c.a.c.i.GlobalEndpointManager - startRefreshLocationTimerAsync() - Unable to refresh database account from any location.

We have not yet found a workaround for this and the Gradle tests regularly break in the pipeline. It sometimes takes 5 - 10 executions until the test job is successful.

Is there an update for this issue? Or is there an update on the way? Cosmos DB is the flagship database on Azure and is advertised by Microsoft. It is a great pity that the associated emulator really does not work well. I would expect an emulator that is regularly maintained, starts quickly, works without major problems and fulfills expectations. Unfortunately, this is not currently the case. The instability of the Linux emulator is causing us more and more problems, our development is slowing down and we are losing more and more money as a result. Please Microsoft, do something about it and satisfy the expectations of the developers. That should also be your own standard.

@sajeetharan @v1k1

HeneryHawk avatar Jul 22 '24 15:07 HeneryHawk

We also experience same problem now: on Ubuntu 20 host cosmosDB emulator prints Started in console, but would never be able to serve the certificate for Java to connect to it. However, in my case (I am executing tests in emulator on pretty beefy VMs using Ubuntu 20 as OS) problem was in disabled IPv6. AFter enabling it - emulator started pretty fast

eptw4by avatar Aug 21 '24 16:08 eptw4by