loki Loki with S3 not able to push chunks on S3 bucket with compactor error "NoSuchKeys"

Loki with S3 not able to push chunks on S3 bucket with compactor error "NoSuchKeys"

Open aksonlyaks opened this issue 3 years ago • 31 comments

Dear Team,

Issue details: I'm trying to setup Loki for our IOT setup on AWS. This does not include Kubernates just logging of Broker service. Broker and Loki are running on two different instances in same VPC. When Loki is running, I can see the logs in grafana But only for around 2 Hours. Not more than that. In S3 bucket I can see there are only two folders, fake and index. I feel chunk folder should also be present to store past logs. On logs I can see compactor is throwing an error "NoSuchKeys". All the logs and details kindly see below:

Loki Config: auth_enabled: false

  server:
    http_listen_port: 3100
  
    #distributor#:
    #ring:
    #  kvstore:
    #    store: memberlist
  
  ingester:
    lifecycler:
      ring:
        kvstore:
          store: inmemory
        replication_factor: 1
        #heartbeat_timeout: 10m
      final_sleep: 0s
    chunk_idle_period: 1h
    chunk_retain_period: 30s
    chunk_encoding: snappy
    max_chunk_age: 2h
    chunk_block_size: 262144
    chunk_target_size: 1536000
    wal:
      enabled: true
  
      #memberlist:
      #abort_if_cluster_join_fails: false
  
    # Expose this port on all distributor, ingester
    # and querier replicas.
    #bind_port: 7946
  
    # You can use a headless k8s service for all distributor,
    # ingester and querier components.
    #join_members:
    # - loki-memberlist        
        #- loki-gossip-ring.loki.svc.cluster.local:7946
  
        #max_join_backoff: 1m
        #max_join_retries: 10
        #min_join_backoff: 1s
  
  schema_config:
    configs:
    - from: 2020-05-15
      store: boltdb-shipper
      object_store: aws
      schema: v11
      index:
        prefix: loki_index_
        period: 24h
  
  compactor:
    working_directory: /data/compactor
    shared_store: aws
    compaction_interval: 1m
    retention_enabled: true


storage_config:
 boltdb_shipper:
   active_index_directory: /loki/index
   cache_location: /loki/index_cache
   shared_store: s3
   cache_ttl: 24h
   resync_interval: 5s
 aws:
   s3: https://xxxxxx/
   bucketnames: xxxxx
   endpoint:  https://xxxxxx/
   insecure: false
   #sse_encryption: false
   http_config:
     idle_conn_timeout: 90s
     response_header_timeout: 0s
     insecure_skip_verify: false
   s3forcepathstyle: true
 filesystem:
    directory: /loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  max_streams_per_user: 100000
  ingestion_rate_mb: 64
  ingestion_burst_size_mb: 16
  per_stream_rate_limit: "10MB"
  per_stream_rate_limit_burst: "50MB"
  retention_period: 2520h

  #http_server_read_timeout: 10m
  #http_server_write_timeout: 10m


query_range:
  split_queries_by_interval: 0

frontend:
  max_outstanding_per_tenant: 4096
  compress_responses: true

querier:
  max_concurrent: 1024

Loki Logs:

  level=info ts=2022-08-31T21:56:35.103515209Z caller=main.go:103 msg="Starting Loki" version="(version=2.6.1, branch=HEAD, revision=6bd05c9a4)"
  level=info ts=2022-08-31T21:56:35.103890713Z caller=server.go:288 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
  level=info ts=2022-08-31T21:56:35.104488269Z caller=modules.go:736 msg="RulerStorage is not configured in single binary mode and will not be started."
  level=warn ts=2022-08-31T21:56:35.10545968Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - chunksfifocache"
  level=info ts=2022-08-31T21:56:35.107380682Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.38µs distinct_users_len=0
  level=info ts=2022-08-31T21:56:35.107449242Z caller=shipper.go:124 msg="starting index shipper in RW mode"
  level=info ts=2022-08-31T21:56:35.107620804Z caller=table_manager.go:226 msg="loading table loki_index_19235"
  level=info ts=2022-08-31T21:56:35.108184301Z caller=table.go:319 msg="handing over indexes to shipper loki_index_19235"
  level=info ts=2022-08-31T21:56:35.108238481Z caller=table.go:335 msg="finished handing over table loki_index_19235"
  level=info ts=2022-08-31T21:56:35.108298962Z caller=shipper_index_client.go:79 msg="starting boltdb shipper in RW mode"
  level=info ts=2022-08-31T21:56:35.108787787Z caller=modules.go:761 msg="RulerStorage is nil.  Not starting the ruler."
  level=info ts=2022-08-31T21:56:35.110411195Z caller=table_manager.go:167 msg="handing over indexes to shipper"
  level=info ts=2022-08-31T21:56:35.110503326Z caller=table.go:319 msg="handing over indexes to shipper loki_index_19235"
  level=info ts=2022-08-31T21:56:35.110536087Z caller=table.go:335 msg="finished handing over table loki_index_19235"
  level=info ts=2022-08-31T21:56:35.111068743Z caller=table_manager.go:134 msg="uploading tables"
  level=info ts=2022-08-31T21:56:35.111176864Z caller=index_set.go:86 msg="uploading table loki_index_19235"
  level=info ts=2022-08-31T21:56:35.112545679Z caller=worker.go:112 msg="Starting querier worker using query-scheduler and scheduler ring for addresses"
  level=info ts=2022-08-31T21:56:35.118339773Z caller=module_service.go:82 msg=initialising module=server
  level=info ts=2022-08-31T21:56:35.121440028Z caller=module_service.go:82 msg=initialising module=memberlist-kv
  level=info ts=2022-08-31T21:56:35.122847873Z caller=module_service.go:82 msg=initialising module=query-frontend-tripperware
  level=info ts=2022-08-31T21:56:35.124948997Z caller=module_service.go:82 msg=initialising module=store
  level=info ts=2022-08-31T21:56:35.125057648Z caller=module_service.go:82 msg=initialising module=ring
  level=info ts=2022-08-31T21:56:35.12525476Z caller=ring.go:263 msg="ring doesn't exist in KV store yet"
  level=info ts=2022-08-31T21:56:35.125409362Z caller=client.go:255 msg="value is nil" key=collectors/ring index=1
  level=info ts=2022-08-31T21:56:35.125518873Z caller=module_service.go:82 msg=initialising module=ingester-querier
  level=info ts=2022-08-31T21:56:35.125631245Z caller=module_service.go:82 msg=initialising module=usage-report
  level=info ts=2022-08-31T21:56:35.126033029Z caller=module_service.go:82 msg=initialising module=ingester
  level=info ts=2022-08-31T21:56:35.12609502Z caller=ingester.go:401 msg="recovering from checkpoint"
  level=info ts=2022-08-31T21:56:35.126233171Z caller=recovery.go:39 msg="no checkpoint found, treating as no-op"
  level=info ts=2022-08-31T21:56:35.126304582Z caller=module_service.go:82 msg=initialising module=query-scheduler
  level=info ts=2022-08-31T21:56:35.126586685Z caller=ring.go:263 msg="ring doesn't exist in KV store yet"
  level=info ts=2022-08-31T21:56:35.126705097Z caller=ingester.go:417 msg="recovered WAL checkpoint recovery finished" elapsed=630.297µs errors=false
  level=info ts=2022-08-31T21:56:35.126769807Z caller=ingester.go:423 msg="recovering from WAL"
  level=info ts=2022-08-31T21:56:35.126762867Z caller=client.go:255 msg="value is nil" key=collectors/scheduler index=2
  level=info ts=2022-08-31T21:56:35.126583215Z caller=basic_lifecycler.go:261 msg="instance not found in the ring" instance=ip-172-31-4-123 ring=scheduler
  level=info ts=2022-08-31T21:56:35.12700163Z caller=basic_lifecycler_delegates.go:63 msg="not loading tokens from file, tokens file path is empty"
  level=info ts=2022-08-31T21:56:35.126600025Z caller=module_service.go:82 msg=initialising module=compactor
  level=info ts=2022-08-31T21:56:35.126630786Z caller=module_service.go:82 msg=initialising module=distributor
  level=info ts=2022-08-31T21:56:35.128076822Z caller=lifecycler.go:547 msg="not loading tokens from file, tokens file path is empty"
  level=info ts=2022-08-31T21:56:35.128374735Z caller=lifecycler.go:576 msg="instance not found in ring, adding with no tokens" ring=distributor
  level=info ts=2022-08-31T21:56:35.128574508Z caller=lifecycler.go:416 msg="auto-joining cluster after timeout" ring=distributor
  level=info ts=2022-08-31T21:56:35.127793569Z caller=scheduler.go:617 msg="waiting until scheduler is JOINING in the ring"
  level=info ts=2022-08-31T21:56:35.12879867Z caller=scheduler.go:621 msg="scheduler is JOINING in the ring"
  level=info ts=2022-08-31T21:56:35.128928332Z caller=scheduler.go:631 msg="waiting until scheduler is ACTIVE in the ring"
  level=info ts=2022-08-31T21:56:35.129101473Z caller=basic_lifecycler.go:261 msg="instance not found in the ring" instance=ip-172-31-4-123 ring=compactor
  level=info ts=2022-08-31T21:56:35.129174624Z caller=basic_lifecycler_delegates.go:63 msg="not loading tokens from file, tokens file path is empty"
  level=info ts=2022-08-31T21:56:35.129551448Z caller=compactor.go:307 msg="waiting until compactor is JOINING in the ring"
  level=info ts=2022-08-31T21:56:35.129630399Z caller=compactor.go:311 msg="compactor is JOINING in the ring"
  level=info ts=2022-08-31T21:56:35.129759691Z caller=compactor.go:321 msg="waiting until compactor is ACTIVE in the ring"
  level=info ts=2022-08-31T21:56:35.130122975Z caller=ingester.go:439 msg="WAL segment recovery finished" elapsed=4.047535ms errors=false
  level=info ts=2022-08-31T21:56:35.130173015Z caller=ingester.go:387 msg="closing recoverer"
  level=info ts=2022-08-31T21:56:35.130202926Z caller=ingester.go:395 msg="WAL recovery finished" time=4.127996ms
  level=info ts=2022-08-31T21:56:35.130295327Z caller=lifecycler.go:547 msg="not loading tokens from file, tokens file path is empty"
  level=info ts=2022-08-31T21:56:35.130362797Z caller=lifecycler.go:576 msg="instance not found in ring, adding with no tokens" ring=ingester
  level=info ts=2022-08-31T21:56:35.130467499Z caller=lifecycler.go:416 msg="auto-joining cluster after timeout" ring=ingester
  level=info ts=2022-08-31T21:56:35.13062923Z caller=wal.go:156 msg=started component=wal
  level=info ts=2022-08-31T21:56:35.22854364Z caller=index_set.go:107 msg="finished uploading table loki_index_19235"
  level=info ts=2022-08-31T21:56:35.228734802Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table loki_index_19235"
  level=info ts=2022-08-31T21:56:35.311633974Z caller=compactor.go:325 msg="compactor is ACTIVE in the ring"
  level=info ts=2022-08-31T21:56:35.316843102Z caller=scheduler.go:635 msg="scheduler is ACTIVE in the ring"
  level=info ts=2022-08-31T21:56:35.317055855Z caller=module_service.go:82 msg=initialising module=querier
  level=info ts=2022-08-31T21:56:35.317061875Z caller=module_service.go:82 msg=initialising module=query-frontend
  level=info ts=2022-08-31T21:56:35.317385418Z caller=loki.go:374 msg="Loki started"
  level=info ts=2022-08-31T21:56:38.318075331Z caller=worker.go:209 msg="adding connection" addr=172.31.4.123:9095
  level=info ts=2022-08-31T21:56:38.321002903Z caller=scheduler.go:682 msg="this scheduler is in the ReplicationSet, will now accept requests."
  level=info ts=2022-08-31T21:56:40.110411401Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:56:40.110479131Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.43µs distinct_users_len=0
  level=info ts=2022-08-31T21:56:40.312729302Z caller=compactor.go:386 msg="this instance has been chosen to run the compactor, starting compactor"
  level=info ts=2022-08-31T21:56:40.312847834Z caller=compactor.go:413 msg="waiting 1m0s for ring to stay stable and previous compactions to finish before starting compactor"
  level=info ts=2022-08-31T21:56:45.111020189Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:56:45.1110861Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.72µs distinct_users_len=0
  level=info ts=2022-08-31T21:56:45.317601097Z caller=frontend_scheduler_worker.go:101 msg="adding connection to scheduler" addr=172.31.4.123:9095
  level=info ts=2022-08-31T21:56:50.110845388Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:56:50.110889138Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.14µs distinct_users_len=0
  level=info ts=2022-08-31T21:56:55.110750115Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:56:55.110802026Z caller=table_manager.go:252 msg="query readiness setup completed" duration=1.88µs distinct_users_len=0
  level=info ts=2022-08-31T21:57:00.111290655Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:57:00.111365756Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.25µs distinct_users_len=0
  level=info ts=2022-08-31T21:57:05.110476892Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:57:05.110530883Z caller=table_manager.go:252 msg="query readiness setup completed" duration=1.96µs distinct_users_len=0
  level=info ts=2022-08-31T21:57:10.110994099Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:57:10.11108952Z caller=table_manager.go:252 msg="query readiness setup completed" duration=3.01µs distinct_users_len=0
  level=info ts=2022-08-31T21:57:15.111400575Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:57:15.111518696Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.64µs distinct_users_len=0
  level=info ts=2022-08-31T21:57:20.111343325Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:57:20.111420576Z caller=table_manager.go:252 msg="query readiness setup completed" duration=3.2µs distinct_users_len=0
  level=info ts=2022-08-31T21:57:25.11136429Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:57:25.111418541Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.55µs distinct_users_len=0
  level=info ts=2022-08-31T21:57:30.111529898Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:57:30.111610629Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.66µs distinct_users_len=0
  level=info ts=2022-08-31T21:57:35.11043739Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:57:35.110498631Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.05µs distinct_users_len=0
  level=info ts=2022-08-31T21:57:35.111543913Z caller=table_manager.go:167 msg="handing over indexes to shipper"
  level=info ts=2022-08-31T21:57:35.111648264Z caller=table.go:319 msg="handing over indexes to shipper loki_index_19235"
  level=info ts=2022-08-31T21:57:35.111691114Z caller=table.go:335 msg="finished handing over table loki_index_19235"
  level=info ts=2022-08-31T21:57:35.229397992Z caller=table_manager.go:134 msg="uploading tables"
  level=info ts=2022-08-31T21:57:35.229476553Z caller=index_set.go:86 msg="uploading table loki_index_19235"
  level=info ts=2022-08-31T21:57:35.229503773Z caller=index_set.go:107 msg="finished uploading table loki_index_19235"
  level=info ts=2022-08-31T21:57:35.229526674Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table loki_index_19235"
  level=info ts=2022-08-31T21:57:40.111138528Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:57:40.111192558Z caller=table_manager.go:252 msg="query readiness setup completed" duration=1.75µs distinct_users_len=0
  level=info ts=2022-08-31T21:57:40.313896107Z caller=compactor.go:418 msg="compactor startup delay completed"
  level=info ts=2022-08-31T21:57:40.313958298Z caller=compactor.go:469 msg="compactor started"
  level=info ts=2022-08-31T21:57:40.313971148Z caller=marker.go:178 msg="mark processor started" workers=150 delay=2h0m0s
  level=info ts=2022-08-31T21:57:40.31408398Z caller=compactor.go:426 msg="applying retention with compaction"
  level=info ts=2022-08-31T21:57:40.3141143Z caller=expiration.go:69 msg="overall smallest retention period 1652911060.314, default smallest retention period 1652911060.314"
  ts=2022-08-31T21:57:40.31412924Z caller=spanlogger.go:80 level=info msg="building index list cache"
  level=info ts=2022-08-31T21:57:40.314580255Z caller=marker.go:203 msg="no marks file found"
  ts=2022-08-31T21:57:40.3366666Z caller=spanlogger.go:80 level=info msg="index list cache built" duration=22.50015ms
  level=error ts=2022-08-31T21:57:40.336772992Z caller=cached_client.go:65 msg="failed to build cache" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: 45HGCW579N8WP80J, host id: 5BcwGslXQ+cM3FDeSd7Z3Q2fQu4E6fjZwCO07w1bMSQbS3dvyYfoQSd6UyaI8M3admoSmFkHcg+79elPUbLmgQ=="
  level=error ts=2022-08-31T21:57:40.336857663Z caller=compactor.go:432 msg="failed to run compaction" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: 45HGCW579N8WP80J, host id: 5BcwGslXQ+cM3FDeSd7Z3Q2fQu4E6fjZwCO07w1bMSQbS3dvyYfoQSd6UyaI8M3admoSmFkHcg+79elPUbLmgQ=="
  level=info ts=2022-08-31T21:57:45.110841563Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:57:45.110893724Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.07µs distinct_users_len=0
  level=info ts=2022-08-31T21:57:50.111245558Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:57:50.111303449Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.26µs distinct_users_len=0
  level=info ts=2022-08-31T21:57:55.110657024Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:57:55.110707175Z caller=table_manager.go:252 msg="query readiness setup completed" duration=1.83µs distinct_users_len=0
  level=info ts=2022-08-31T21:58:00.110510574Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:58:00.110594655Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.78µs distinct_users_len=0
  level=info ts=2022-08-31T21:58:05.11078013Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:58:05.1108363Z caller=table_manager.go:252 msg="query readiness setup completed" duration=1.99µs distinct_users_len=0
  level=info ts=2022-08-31T21:58:10.111471838Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:58:10.111525948Z caller=table_manager.go:252 msg="query readiness setup completed" duration=1.9µs distinct_users_len=0
  level=info ts=2022-08-31T21:58:15.111474397Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:58:15.111530117Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.45µs distinct_users_len=0
  level=info ts=2022-08-31T21:58:20.110724183Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:58:20.110778724Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.25µs distinct_users_len=0
  level=info ts=2022-08-31T21:58:25.11069941Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:58:25.110776711Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.74µs distinct_users_len=0
  level=info ts=2022-08-31T21:58:30.110460508Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:58:30.110550279Z caller=table_manager.go:252 msg="query readiness setup completed" duration=3.31µs distinct_users_len=0
  level=info ts=2022-08-31T21:58:35.110749343Z caller=table_manager.go:167 msg="handing over indexes to shipper"
  level=info ts=2022-08-31T21:58:35.110999416Z caller=table.go:319 msg="handing over indexes to shipper loki_index_19235"
  level=info ts=2022-08-31T21:58:35.111048566Z caller=table.go:335 msg="finished handing over table loki_index_19235"
  level=info ts=2022-08-31T21:58:35.110849004Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:58:35.111203758Z caller=table_manager.go:252 msg="query readiness setup completed" duration=10.44µs distinct_users_len=0
  level=info ts=2022-08-31T21:58:35.229165189Z caller=table_manager.go:134 msg="uploading tables"
  level=info ts=2022-08-31T21:58:35.22925081Z caller=index_set.go:86 msg="uploading table loki_index_19235"
  level=info ts=2022-08-31T21:58:35.22927658Z caller=index_set.go:107 msg="finished uploading table loki_index_19235"
  level=info ts=2022-08-31T21:58:35.22931547Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table loki_index_19235"
  level=info ts=2022-08-31T21:58:40.111069776Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:58:40.111130057Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.64µs distinct_users_len=0
  level=info ts=2022-08-31T21:58:40.314690541Z caller=marker.go:203 msg="no marks file found"
  level=info ts=2022-08-31T21:58:40.337382623Z caller=compactor.go:426 msg="applying retention with compaction"
  level=info ts=2022-08-31T21:58:40.337451994Z caller=expiration.go:69 msg="overall smallest retention period 1652911120.337, default smallest retention period 1652911120.337"
  ts=2022-08-31T21:58:40.337470494Z caller=spanlogger.go:80 level=info msg="building index list cache"
  ts=2022-08-31T21:58:40.364565235Z caller=spanlogger.go:80 level=info msg="index list cache built" duration=27.078011ms
  level=error ts=2022-08-31T21:58:40.364696337Z caller=cached_client.go:65 msg="failed to build cache" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: 4CCXSS078WTKWHDA, host id: MBpvbHI7Kzb0qDRN3fmwalj6K/GQwja6Q9ZAvv9ajN0JPbmvrF45sHi6G/sI487VWMn8mk64ozUS7F9EoThA+g=="
  level=error ts=2022-08-31T21:58:40.364744597Z caller=compactor.go:432 msg="failed to run compaction" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: 4CCXSS078WTKWHDA, host id: MBpvbHI7Kzb0qDRN3fmwalj6K/GQwja6Q9ZAvv9ajN0JPbmvrF45sHi6G/sI487VWMn8mk64ozUS7F9EoThA+g=="
  level=info ts=2022-08-31T21:58:45.11080886Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:58:45.110889941Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.87µs distinct_users_len=0
  level=info ts=2022-08-31T21:58:50.110718706Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:58:50.110808937Z caller=table_manager.go:252 msg="query readiness setup completed" duration=3.3µs distinct_users_len=0
  level=info ts=2022-08-31T21:58:55.111317316Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:58:55.111405647Z caller=table_manager.go:252 msg="query readiness setup completed" duration=5.75µs distinct_users_len=0
  level=info ts=2022-08-31T21:59:00.110992353Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:59:00.111072474Z caller=table_manager.go:252 msg="query readiness setup completed" duration=3.11µs distinct_users_len=0
  level=info ts=2022-08-31T21:59:05.110684188Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:59:05.110765248Z caller=table_manager.go:252 msg="query readiness setup completed" duration=3.85µs distinct_users_len=0
  level=info ts=2022-08-31T21:59:10.111020027Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:59:10.111079288Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.1µs distinct_users_len=0
  level=info ts=2022-08-31T21:59:15.111473161Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:59:15.111535972Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.14µs distinct_users_len=0
  level=info ts=2022-08-31T21:59:20.110629903Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:59:20.110709754Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.84µs distinct_users_len=0
  level=info ts=2022-08-31T21:59:25.111182688Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:59:25.111267919Z caller=table_manager.go:252 msg="query readiness setup completed" duration=3.79µs distinct_users_len=0
  level=info ts=2022-08-31T21:59:30.111403812Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:59:30.111470813Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.42µs distinct_users_len=0
  level=info ts=2022-08-31T21:59:35.110985477Z caller=table_manager.go:167 msg="handing over indexes to shipper"
  level=info ts=2022-08-31T21:59:35.111134878Z caller=table.go:319 msg="handing over indexes to shipper loki_index_19235"
  level=info ts=2022-08-31T21:59:35.111003437Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:59:35.111158979Z caller=table.go:335 msg="finished handing over table loki_index_19235"
  level=info ts=2022-08-31T21:59:35.111176239Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.28µs distinct_users_len=0
  level=info ts=2022-08-31T21:59:35.228958303Z caller=table_manager.go:134 msg="uploading tables"
  level=info ts=2022-08-31T21:59:35.229073914Z caller=index_set.go:86 msg="uploading table loki_index_19235"
  level=info ts=2022-08-31T21:59:35.229099225Z caller=index_set.go:107 msg="finished uploading table loki_index_19235"
  level=info ts=2022-08-31T21:59:35.229114605Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table loki_index_19235"
  level=info ts=2022-08-31T21:59:40.110987349Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-08-31T21:59:40.11108507Z caller=table_manager.go:252 msg="query readiness setup completed" duration=3.09µs distinct_users_len=0
  level=info ts=2022-08-31T21:59:40.314657925Z caller=marker.go:203 msg="no marks file found"
  level=info ts=2022-08-31T21:59:40.337049821Z caller=compactor.go:426 msg="applying retention with compaction"
  level=info ts=2022-08-31T21:59:40.337106532Z caller=expiration.go:69 msg="overall smallest retention period 1652911180.337, default smallest retention period 1652911180.337"
  ts=2022-08-31T21:59:40.337124692Z caller=spanlogger.go:80 level=info msg="building index list cache"
  ts=2022-08-31T21:59:40.364796376Z caller=spanlogger.go:80 level=info msg="index list cache built" duration=27.655884ms
  level=error ts=2022-08-31T21:59:40.364947387Z caller=cached_client.go:65 msg="failed to build cache" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: K8HY6PR6WVTSEQJT, host id: Mw+aszR77OLhDj+eFHkltNhovmKHSgfiVvEb8+/UU1CB8zGStC34eJSYh1hQZeBRKLZRpYF2jyw="
  level=error ts=2022-08-31T21:59:40.365002688Z caller=compactor.go:432 msg="failed to run compaction" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: K8HY6PR6WVTSEQJT, host id: Mw+aszR77OLhDj+eFHkltNhovmKHSgfiVvEb8+/UU1CB8zGStC34eJSYh1hQZeBRKLZRpYF2jyw="

S3 bucket:

Loki Version used:

    loki, version 2.6.1 (branch: HEAD, revision: 6bd05c9a4)
    build user:       root@af90ed01061f
    build date:       2022-07-18T08:41:09Z
    go version:       go1.17.6
    platform:         linux/amd64

Kindly help me with this issue. Struggling since quite a few days.

Thanks and Regards Abhishek

Sep 01 '22 08:09 aksonlyaks

@aksonlyaks thanks for reporting the issue.

I feel chunk folder should also be present to store past logs

I quickly checked in one of our internal AWS setup. and chunks are usually stored under tenant-id prefix. In your case it's "fake" (which is default tenant-id for running in single tenant mode).

Q1. May I know you see any files inside "fake" directory? or is it just empty.

Q2: Do you see any errors during querying? (permission denied, unable to access the storage, timed-out or similar?)

Also there shouldn't be filesystem config there.

 filesystem:
    directory: /loki/chunks

Sep 01 '22 09:09 kavirajk

There are files present in it, below is a screenshot:

Screenshot of Grafana: It never shows data for beyond past 2hrs.

Sep 01 '22 09:09 aksonlyaks

So looks like chunks are getting flushed just fine.

May I know what query you using it for that dashboard?

Sep 01 '22 09:09 kavirajk

Removed filesystem tag but still log looks same as above I have reported

Q2: Do you see any errors during querying? (permission denied, unable to access the storage, timed-out or similar?)

logs when Query is performed: query is: {job="emqx_connection"} |= `` for last 6 hours.

  level=info ts=2022-09-01T09:15:11.171798294Z caller=marker.go:203 msg="no marks file found"
  ts=2022-09-01T09:15:11.171801434Z caller=spanlogger.go:80 level=info msg="building index list cache"
  ts=2022-09-01T09:15:11.197582011Z caller=spanlogger.go:80 level=info msg="index list cache built" duration=25.728927ms
  level=error ts=2022-09-01T09:15:11.197652671Z caller=cached_client.go:65 msg="failed to build cache" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: 06T7WA7KKFNY5FNF, host id: staR4iv/gnkq2XNUF9jTw0R3B6wMHX5jYoxAju2psfRYwkUr43zPUTOmp3ip0lWu00Oyh+m/kaj5qghihgIrNA=="
  level=error ts=2022-09-01T09:15:11.197685862Z caller=compactor.go:432 msg="failed to run compaction" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: 06T7WA7KKFNY5FNF, host id: staR4iv/gnkq2XNUF9jTw0R3B6wMHX5jYoxAju2psfRYwkUr43zPUTOmp3ip0lWu00Oyh+m/kaj5qghihgIrNA=="
  level=info ts=2022-09-01T09:15:15.982550988Z caller=table_manager.go:213 msg="syncing tables"
  level=info ts=2022-09-01T09:15:15.982618388Z caller=table_manager.go:252 msg="query readiness setup completed" duration=3.17µs distinct_users_len=0
  level=info ts=2022-09-01T09:15:19.780266877Z caller=table_manager.go:180 msg="downloading all files for table loki_index_19236"
  level=error ts=2022-09-01T09:15:19.819465013Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF66PHG7KNMJDVZE, host id: s5UhRDZI0Pa1h0ukzKPKSf5KY5mae7AlCT3PYAc24BxtdS7d/2zzYBytGdvImqNmZavYtQoDeLk="
  level=error ts=2022-09-01T09:15:19.831339184Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6C7ZQH5JDCTMYK, host id: Li0b3VggR1+dqwdxKf5gtTLg+Wiki1xJ0tbqXQ11rwL6gXhUQw0veBOtfLkgwXyyL4Ov0rTaJxc="
  level=error ts=2022-09-01T09:15:19.8318453Z caller=index_set.go:104 table-name=loki_index_19236 user-id=fake msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6C7ZQH5JDCTMYK, host id: Li0b3VggR1+dqwdxKf5gtTLg+Wiki1xJ0tbqXQ11rwL6gXhUQw0veBOtfLkgwXyyL4Ov0rTaJxc="
  level=error ts=2022-09-01T09:15:19.832266215Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set fake" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6C7ZQH5JDCTMYK, host id: Li0b3VggR1+dqwdxKf5gtTLg+Wiki1xJ0tbqXQ11rwL6gXhUQw0veBOtfLkgwXyyL4Ov0rTaJxc="
  level=error ts=2022-09-01T09:15:19.832388756Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set fake has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6C7ZQH5JDCTMYK, host id: Li0b3VggR1+dqwdxKf5gtTLg+Wiki1xJ0tbqXQ11rwL6gXhUQw0veBOtfLkgwXyyL4Ov0rTaJxc="
  ts=2022-09-01T09:15:19.83276921Z caller=spanlogger.go:80 table-name=loki_index_19236 user-id=fake org_id=fake level=info msg="downloaded index set at query time" duration=52.048858ms
  level=error ts=2022-09-01T09:15:19.848099201Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6B0EHDZTS6MKA9, host id: g/4JzWs199PgIQqWRqSiEQ+97YxL6VKPLP+TTbrxbX6xt37FhCFyen1SN5ixM1kZIDonhmmsIlM="
  level=error ts=2022-09-01T09:15:19.859809431Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6APJZVBPPY8ZHY, host id: CZXpXzYR3zYUEQMzmDg1+4KZvYZ+4GVqd6hgtDGZoON7KzjvyaocQN7dieRX78s6Vjgf0M32bNY="
  level=error ts=2022-09-01T09:15:19.859947962Z caller=index_set.go:104 table-name=loki_index_19236 msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6APJZVBPPY8ZHY, host id: CZXpXzYR3zYUEQMzmDg1+4KZvYZ+4GVqd6hgtDGZoON7KzjvyaocQN7dieRX78s6Vjgf0M32bNY="
  level=error ts=2022-09-01T09:15:19.860035383Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set " err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6APJZVBPPY8ZHY, host id: CZXpXzYR3zYUEQMzmDg1+4KZvYZ+4GVqd6hgtDGZoON7KzjvyaocQN7dieRX78s6Vjgf0M32bNY="
  level=error ts=2022-09-01T09:15:19.862817414Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set  has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6APJZVBPPY8ZHY, host id: CZXpXzYR3zYUEQMzmDg1+4KZvYZ+4GVqd6hgtDGZoON7KzjvyaocQN7dieRX78s6Vjgf0M32bNY="
  ts=2022-09-01T09:15:19.863064697Z caller=spanlogger.go:80 table-name=loki_index_19236 org_id=fake level=info msg="downloaded index set at query time" duration=28.845731ms
  level=info ts=2022-09-01T09:15:19.866288953Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=16m0s step=1m0s duration=75.828612ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=4.290957ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.868857221Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=16m0s step=1m0s duration=89.910289ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=78.911µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.869148565Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=15m19.671s step=10s duration=88.405213ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=82.031µs subqueries=1
  level=info ts=2022-09-01T09:15:19.869444138Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=15m19.671s step=10s duration=89.985409ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=47.081µs subqueries=1
  level=info ts=2022-09-01T09:15:19.869861212Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=16m0s step=1m0s duration=89.933319ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=95.892µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.870089195Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=16m0s step=1m0s duration=95.623472ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=106.101µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.870239537Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=14m0s step=1m0s duration=70.626575ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=38.791µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.870284877Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=15m19.671s step=10s duration=89.99183ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=88.221µs subqueries=1
  level=info ts=2022-09-01T09:15:19.87056602Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=16m0s step=1m0s duration=92.874832ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=92.061µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.870765803Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=16m0s step=1m0s duration=88.779906ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=124.781µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.870918824Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=16m0s step=1m0s duration=93.726891ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=96.101µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.871003755Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=92.822982ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=105.971µs subqueries=1
  level=info ts=2022-09-01T09:15:19.871132607Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=95.926176ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=60.6µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.871645082Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=15m19.671s step=10s duration=95.139647ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=105.481µs subqueries=1
  level=info ts=2022-09-01T09:15:19.871258098Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=16m0s step=1m0s duration=95.317119ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=96.951µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.872432031Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=16m0s step=1m0s duration=95.564782ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=107.581µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.872618813Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=15m19.671s step=10s duration=95.759904ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=41.031µs subqueries=1
  level=info ts=2022-09-01T09:15:19.872875546Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=16m0s step=1m0s duration=96.047387ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=103.861µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.873112729Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=16m0s step=1m0s duration=87.773166ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=173.832µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.873292411Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=15m19.671s step=10s duration=89.169761ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=153.332µs subqueries=1
  level=info ts=2022-09-01T09:15:19.873788646Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=16m0s step=1m0s duration=88.746906ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=145.901µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.87417449Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=15m19.671s step=10s duration=89.138061ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=135.502µs subqueries=1
  level=info ts=2022-09-01T09:15:19.876444016Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=14m40.329s step=10s duration=71.11034ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=116.191µs subqueries=1
  level=info ts=2022-09-01T09:15:19.877664449Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=76.732622ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=24.91µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.880326589Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=58.895464ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=88.651µs subqueries=1
  level=info ts=2022-09-01T09:15:19.88136635Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=80.047099ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=113.651µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.881917976Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=80.244972ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=110.101µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.882043528Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=15m19.671s step=10s duration=88.131339ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=2.761321ms subqueries=1
  level=info ts=2022-09-01T09:15:19.881936297Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=16m0s step=1m0s duration=88.866337ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=35.821µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.88222474Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=15m19.671s step=10s duration=89.506384ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=3.290966ms subqueries=1
  level=info ts=2022-09-01T09:15:19.882393512Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=15m19.671s step=10s duration=86.938426ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=125.892µs subqueries=1
  level=info ts=2022-09-01T09:15:19.882411252Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=76.946165ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=123.322µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.882583584Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=78.465772ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=133.762µs subqueries=1
  level=info ts=2022-09-01T09:15:19.882651065Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=15m19.671s step=10s duration=87.871026ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=144.372µs subqueries=1
  level=info ts=2022-09-01T09:15:19.882780896Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=16m0s step=1m0s duration=89.469594ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=2.646119ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.883020349Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=78.496122ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=97.841µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.883035899Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=16m0s step=1m0s duration=87.869086ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=123.461µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.883700436Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=79.182479ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=31.201µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.883972129Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=16m0s step=1m0s duration=90.136912ms status=200 limit=1000 returned_lines=0 throughput=6.6MB total_bytes=593kB total_entries=14 queue_time=129.112µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.884997651Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=15m19.671s step=10s duration=84.294586ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=104.381µs subqueries=1
  level=info ts=2022-09-01T09:15:19.88489959Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=15m19.671s step=10s duration=90.417395ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=2.742631ms subqueries=1
  level=info ts=2022-09-01T09:15:19.885194713Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=15m19.671s step=10s duration=85.166716ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=120.781µs subqueries=1
  level=info ts=2022-09-01T09:15:19.885350805Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=15m19.671s step=10s duration=85.038485ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=100.691µs subqueries=1
  level=info ts=2022-09-01T09:15:19.885349365Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=73.099662ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=123.28µs subqueries=1
  level=info ts=2022-09-01T09:15:19.885561237Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=76.737083ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=121.912µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.885637178Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=85.251707ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=129.931µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.886904552Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=75.923094ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=34.6µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.886951362Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=84.402868ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=125.26µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.887765801Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=76.583171ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=959.96µs subqueries=1
  level=info ts=2022-09-01T09:15:19.888308107Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=76.391709ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=141.641µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.888595211Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=74.069213ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=132.691µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.890280579Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=77.756413ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=132.212µs subqueries=1
  level=info ts=2022-09-01T09:15:19.890569683Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=78.189979ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=981.731µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.890838756Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=77.717233ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=270.693µs subqueries=1
  level=info ts=2022-09-01T09:15:19.891722065Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=77.191397ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=142.942µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.892741697Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=75.320547ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=102.841µs subqueries=1
  level=info ts=2022-09-01T09:15:19.892985689Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=75.208656ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=256.623µs subqueries=1
  level=info ts=2022-09-01T09:15:19.893207312Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=73.952401ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=109.371µs subqueries=1
  level=info ts=2022-09-01T09:15:19.893648397Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=75.506619ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=708.427µs subqueries=1
  level=info ts=2022-09-01T09:15:19.893879279Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=74.879052ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=121.381µs subqueries=1
  level=info ts=2022-09-01T09:15:19.894102882Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=72.843079ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=27.65µs subqueries=1
  level=info ts=2022-09-01T09:15:19.894786229Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=73.673079ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=341.724µs subqueries=1
  level=info ts=2022-09-01T09:15:19.895027192Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=73.199394ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=328.683µs subqueries=1
  level=info ts=2022-09-01T09:15:19.906067585Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=15m19.671s step=10s duration=106.845667ms status=200 limit=1000 returned_lines=801 throughput=5.5MB total_bytes=593kB total_entries=801 queue_time=99.351µs subqueries=1
  level=error ts=2022-09-01T09:15:19.914502888Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6CZCN57XKGN7BY, host id: Vd1Nh5KVAOEVX4zvzJ0a2vtRKN6HBw9pjd2UkKQyKXfyZ3BOyQOSt79bgZQeO2RPp1lsDrHWcFo="
  level=error ts=2022-09-01T09:15:19.922595558Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6D7PAE4G72CAK7, host id: XP7uSgZJv79zo+QXU20InytFnzyOwv5Cx99lvpg1j+mXb6Ks6ZZrGbEIYpQjYvHbja7N9syPW3o="
  level=error ts=2022-09-01T09:15:19.92277898Z caller=index_set.go:104 table-name=loki_index_19236 user-id=fake msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6D7PAE4G72CAK7, host id: XP7uSgZJv79zo+QXU20InytFnzyOwv5Cx99lvpg1j+mXb6Ks6ZZrGbEIYpQjYvHbja7N9syPW3o="
  level=error ts=2022-09-01T09:15:19.922887462Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set fake" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6D7PAE4G72CAK7, host id: XP7uSgZJv79zo+QXU20InytFnzyOwv5Cx99lvpg1j+mXb6Ks6ZZrGbEIYpQjYvHbja7N9syPW3o="
  level=error ts=2022-09-01T09:15:19.922996473Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set fake has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6D7PAE4G72CAK7, host id: XP7uSgZJv79zo+QXU20InytFnzyOwv5Cx99lvpg1j+mXb6Ks6ZZrGbEIYpQjYvHbja7N9syPW3o="
  ts=2022-09-01T09:15:19.923181255Z caller=spanlogger.go:80 table-name=loki_index_19236 user-id=fake org_id=fake level=info msg="downloaded index set at query time" duration=21.682231ms
  level=error ts=2022-09-01T09:15:19.934392379Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6BEP98CRGPFQ5G, host id: L7P+Ynl9BdKUQoz92erl8w957vOKmbVGDlmmxPXjZg5Hml5pJDg4ZFAyDVMcvybH03ODNBt+/Ds="
  level=error ts=2022-09-01T09:15:19.944134388Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6CCSE9WBZ7BT2D, host id: EmYeGcudqDUWHF7zZpj4Cc9we+OpBSrS/QPDVKf4x9N9K9j6cZJj2ysYfioPGZ8Qp2mn8kMAF7k="
  level=error ts=2022-09-01T09:15:19.950182425Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6BCJGVYA07K9AF, host id: zSzYGJxNrGCqvgPWHqjTgUBcNKSI1CQU/Au7S2cJL+xFiF2d7ygdBiS45H2ebIxvCIDwRKZvIEQ="
  level=error ts=2022-09-01T09:15:19.950302917Z caller=index_set.go:104 table-name=loki_index_19236 msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6BCJGVYA07K9AF, host id: zSzYGJxNrGCqvgPWHqjTgUBcNKSI1CQU/Au7S2cJL+xFiF2d7ygdBiS45H2ebIxvCIDwRKZvIEQ="
  level=error ts=2022-09-01T09:15:19.95058255Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set " err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6BCJGVYA07K9AF, host id: zSzYGJxNrGCqvgPWHqjTgUBcNKSI1CQU/Au7S2cJL+xFiF2d7ygdBiS45H2ebIxvCIDwRKZvIEQ="
  level=error ts=2022-09-01T09:15:19.95063299Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set  has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6BCJGVYA07K9AF, host id: zSzYGJxNrGCqvgPWHqjTgUBcNKSI1CQU/Au7S2cJL+xFiF2d7ygdBiS45H2ebIxvCIDwRKZvIEQ="
  ts=2022-09-01T09:15:19.951166266Z caller=spanlogger.go:80 table-name=loki_index_19236 org_id=fake level=info msg="downloaded index set at query time" duration=27.84523ms
  level=info ts=2022-09-01T09:15:19.951692022Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=33.593634ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=128.902µs subqueries=1
  level=info ts=2022-09-01T09:15:19.953831986Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=37.208334ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=108.501µs subqueries=1
  level=info ts=2022-09-01T09:15:19.954660175Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=14m0s step=1m0s duration=36.952781ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=134.411µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.955040809Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=37.269685ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=229.233µs subqueries=1
  level=info ts=2022-09-01T09:15:19.955396913Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=14m0s step=1m0s duration=37.580798ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=127.162µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.955741767Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=34.286842ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=64.921µs subqueries=1
  level=info ts=2022-09-01T09:15:19.956355614Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=14m0s step=1m0s duration=37.470627ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=640.567µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.956953031Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=37.217614ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=33.301µs subqueries=1
  level=info ts=2022-09-01T09:15:19.957300924Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=14m0s step=1m0s duration=36.938211ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=157.822µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.957647148Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=14m0s step=1m0s duration=35.854698ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=74.271µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.958423137Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=14m0s step=1m0s duration=36.85652ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=401.055µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.959018444Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=36.927321ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=74.671µs subqueries=1
  level=info ts=2022-09-01T09:15:19.959127965Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=55.335656ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=134.502µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.951690602Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=37.504097ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=45.39µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.959409068Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=55.003522ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=43.19µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.960056355Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=54.604937ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=32.52µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.96226696Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=50.522751ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=3.875723ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.962641994Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=50.560092ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=3.938164ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.964350593Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=64.328815ms status=200 limit=1000 returned_lines=0 throughput=16MB total_bytes=1.0MB total_entries=30 queue_time=83.171µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:19.965264843Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF66GX9TM84SEWHM, host id: ZZVId2hEqwgBjFNhg6guXKsScao8VaK7E5arE0vb7DdiIBviJYq8DM/NALNDhA1d7ZByPEHGV80="
  level=error ts=2022-09-01T09:15:19.965376834Z caller=index_set.go:104 table-name=loki_index_19236 user-id=fake msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF66GX9TM84SEWHM, host id: ZZVId2hEqwgBjFNhg6guXKsScao8VaK7E5arE0vb7DdiIBviJYq8DM/NALNDhA1d7ZByPEHGV80="
  level=error ts=2022-09-01T09:15:19.965475955Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set fake" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF66GX9TM84SEWHM, host id: ZZVId2hEqwgBjFNhg6guXKsScao8VaK7E5arE0vb7DdiIBviJYq8DM/NALNDhA1d7ZByPEHGV80="
  level=error ts=2022-09-01T09:15:19.965568976Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set fake has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF66GX9TM84SEWHM, host id: ZZVId2hEqwgBjFNhg6guXKsScao8VaK7E5arE0vb7DdiIBviJYq8DM/NALNDhA1d7ZByPEHGV80="
  ts=2022-09-01T09:15:19.965787669Z caller=spanlogger.go:80 table-name=loki_index_19236 user-id=fake org_id=fake level=info msg="downloaded index set at query time" duration=40.283158ms
  level=info ts=2022-09-01T09:15:19.966514937Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=59.33913ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=132.591µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.966935182Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=66.282706ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=81.781µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.967493048Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=73.950132ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=84.671µs subqueries=1
  level=info ts=2022-09-01T09:15:19.96773871Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=68.38765ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=33.571µs subqueries=1
  level=info ts=2022-09-01T09:15:19.968013053Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=55.617569ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=3.957844ms subqueries=1
  level=info ts=2022-09-01T09:15:19.968322797Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=59.122107ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=3.966534ms subqueries=1
  level=info ts=2022-09-01T09:15:19.970303869Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=62.729627ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=137.661µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.970602742Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=64.517877ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=86.591µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.970625523Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=70.966379ms status=200 limit=1000 returned_lines=1000 throughput=6.9MB total_bytes=486kB total_entries=1000 queue_time=95.771µs subqueries=1
  level=error ts=2022-09-01T09:15:19.976197634Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6B797H6WZE61DD, host id: Lx1k7lH7cFgqc/CWFWbLfGXk9EsdjIUN5s0stks1EkRkXQ/zT2SYfIKh9l/mAcmUXNUxmEx7CF8="
  level=error ts=2022-09-01T09:15:19.984917071Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF67RCXGBCFP657R, host id: Etl18w7iGQbCCXLKlxGQ3l7yZKgtKX7hKeeVQwgQ2HLbu16bUIDxPMXBxavGWDoWuLh+CPtKXrY="
  level=error ts=2022-09-01T09:15:19.984996012Z caller=index_set.go:104 table-name=loki_index_19236 msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF67RCXGBCFP657R, host id: Etl18w7iGQbCCXLKlxGQ3l7yZKgtKX7hKeeVQwgQ2HLbu16bUIDxPMXBxavGWDoWuLh+CPtKXrY="
  level=error ts=2022-09-01T09:15:19.985039633Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set " err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF67RCXGBCFP657R, host id: Etl18w7iGQbCCXLKlxGQ3l7yZKgtKX7hKeeVQwgQ2HLbu16bUIDxPMXBxavGWDoWuLh+CPtKXrY="
  level=error ts=2022-09-01T09:15:19.985078853Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set  has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF67RCXGBCFP657R, host id: Etl18w7iGQbCCXLKlxGQ3l7yZKgtKX7hKeeVQwgQ2HLbu16bUIDxPMXBxavGWDoWuLh+CPtKXrY="
  ts=2022-09-01T09:15:19.985367426Z caller=spanlogger.go:80 table-name=loki_index_19236 org_id=fake level=info msg="downloaded index set at query time" duration=19.377915ms
  level=info ts=2022-09-01T09:15:19.985608199Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=14m40.329s step=10s duration=60.27432ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=634.007µs subqueries=1
  level=error ts=2022-09-01T09:15:19.987449889Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6CC6BFHH9QNN6E, host id: lrE25W2sgCw5oOa+W3ROAbmsLhVypnuKNIRZEMb8poF44GqdsZp2Z7jL7s2NFclwY+DOt0ARg1I="
  level=info ts=2022-09-01T09:15:19.988603342Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=14m0s step=1m0s duration=61.325672ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.446936ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.989923177Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=14m40.329s step=10s duration=62.187861ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.457166ms subqueries=1
  level=info ts=2022-09-01T09:15:19.990480543Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=14m40.329s step=10s duration=62.536235ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.452876ms subqueries=1
  level=info ts=2022-09-01T09:15:19.990758356Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=14m40.329s step=10s duration=62.603036ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.006201ms subqueries=1
  level=info ts=2022-09-01T09:15:19.991680686Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=63.194423ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=862.52µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.992409215Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=63.091821ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=862.779µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.992999051Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=14m40.329s step=10s duration=63.667508ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=864.119µs subqueries=1
  level=info ts=2022-09-01T09:15:19.993266444Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=63.426054ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=839.389µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.993516457Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=14m40.329s step=10s duration=63.265293ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=722.758µs subqueries=1
  level=info ts=2022-09-01T09:15:19.99376839Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=63.234852ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=676.677µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.994038383Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=63.193142ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=657.638µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.994550058Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=56.60664ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=8.831708ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.994838122Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=56.059933ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=9.304013ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.995089964Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=56.130814ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=174.422µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.995337407Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=14m40.329s step=10s duration=56.226895ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=8.214651ms subqueries=1
  level=info ts=2022-09-01T09:15:19.99556286Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=14m40.329s step=10s duration=56.169184ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=6.721674ms subqueries=1
  level=info ts=2022-09-01T09:15:19.995774522Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=14m40.329s step=10s duration=56.138183ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=929.37µs subqueries=1
  level=info ts=2022-09-01T09:15:19.996957735Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=14m40.329s step=10s duration=56.007542ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=44.941µs subqueries=1
  level=info ts=2022-09-01T09:15:19.997253308Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=55.849311ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=103.472µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.997488621Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=55.734469ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=111.911µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.997699523Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=14m40.329s step=10s duration=55.648528ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=134.271µs subqueries=1
  level=info ts=2022-09-01T09:15:19.997913466Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=14m40.329s step=10s duration=55.488186ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=75.681µs subqueries=1
  level=info ts=2022-09-01T09:15:19.998123198Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=14m40.329s step=10s duration=55.284385ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=35.731µs subqueries=1
  level=info ts=2022-09-01T09:15:19.998389161Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=54.628607ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=83.271µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.998697374Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=14m40.329s step=10s duration=54.551386ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=79.461µs subqueries=1
  level=info ts=2022-09-01T09:15:19.998947447Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=54.165082ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=76.611µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.99920012Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=54.00148ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=144.171µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:19.999655105Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=14m40.329s step=10s duration=53.904169ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=684.678µs subqueries=1
  level=info ts=2022-09-01T09:15:19.999969109Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=52.876317ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=714.828µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.008677215Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6CGWENWD84GMNJ, host id: oN2r8avjwzGhbDa6NCzSPQm6HT+xobMa8Gi79m4AEM2/GhdOp08sNuKY/VrE1oEIRyPpAxMGkEk="
  level=error ts=2022-09-01T09:15:20.008753876Z caller=index_set.go:104 table-name=loki_index_19236 user-id=fake msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6CGWENWD84GMNJ, host id: oN2r8avjwzGhbDa6NCzSPQm6HT+xobMa8Gi79m4AEM2/GhdOp08sNuKY/VrE1oEIRyPpAxMGkEk="
  level=error ts=2022-09-01T09:15:20.008795057Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set fake" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6CGWENWD84GMNJ, host id: oN2r8avjwzGhbDa6NCzSPQm6HT+xobMa8Gi79m4AEM2/GhdOp08sNuKY/VrE1oEIRyPpAxMGkEk="
  level=error ts=2022-09-01T09:15:20.008834607Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set fake has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: QF6CGWENWD84GMNJ, host id: oN2r8avjwzGhbDa6NCzSPQm6HT+xobMa8Gi79m4AEM2/GhdOp08sNuKY/VrE1oEIRyPpAxMGkEk="
  ts=2022-09-01T09:15:20.008883698Z caller=spanlogger.go:80 table-name=loki_index_19236 user-id=fake org_id=fake level=info msg="downloaded index set at query time" duration=33.673614ms
  level=info ts=2022-09-01T09:15:20.012973893Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=65.588518ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=55.67µs subqueries=1
  level=info ts=2022-09-01T09:15:20.016554423Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=71.017519ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=26.66µs subqueries=1
  level=info ts=2022-09-01T09:15:20.01719359Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=71.447794ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=54.621µs subqueries=1
  level=info ts=2022-09-01T09:15:20.017759856Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=71.002109ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=33.43µs subqueries=1
  level=info ts=2022-09-01T09:15:20.017964878Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=70.812877ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=55.65µs subqueries=1
  level=info ts=2022-09-01T09:15:20.018156061Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=70.830197ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=717.798µs subqueries=1
  level=error ts=2022-09-01T09:15:20.026415292Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HP959WHEANDFZS, host id: R5Tr9kpvg6neJXoD6phjjW49LlYPmNwG1ZVpauTg6j6rod+1h7mxNGScPSJp9Qg18l09ebsuIbI="
  level=error ts=2022-09-01T09:15:20.027340413Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HQEZ3J0WW7YYPX, host id: TbEQhypY2epk2muszZb6IR29hXFGLAP7hS2x4j1CKP6g8XlU8aRYd2pnD45aE4IdYxdDi6gvk6g="
  level=error ts=2022-09-01T09:15:20.039217625Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HVDVTZEYSW4G9Z, host id: V4mkS6vnJ23mCnN/60hQ91tvBRoO/Pq93DCAaUEKfqWzik8dADLa/fd8itSuSG8sX2g7eaLI668="
  level=error ts=2022-09-01T09:15:20.039405057Z caller=index_set.go:104 table-name=loki_index_19236 user-id=fake msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HVDVTZEYSW4G9Z, host id: V4mkS6vnJ23mCnN/60hQ91tvBRoO/Pq93DCAaUEKfqWzik8dADLa/fd8itSuSG8sX2g7eaLI668="
  level=error ts=2022-09-01T09:15:20.039533518Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set fake" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HVDVTZEYSW4G9Z, host id: V4mkS6vnJ23mCnN/60hQ91tvBRoO/Pq93DCAaUEKfqWzik8dADLa/fd8itSuSG8sX2g7eaLI668="
  level=error ts=2022-09-01T09:15:20.039656709Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set fake has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HVDVTZEYSW4G9Z, host id: V4mkS6vnJ23mCnN/60hQ91tvBRoO/Pq93DCAaUEKfqWzik8dADLa/fd8itSuSG8sX2g7eaLI668="
  ts=2022-09-01T09:15:20.039850832Z caller=spanlogger.go:80 table-name=loki_index_19236 user-id=fake org_id=fake level=info msg="downloaded index set at query time" duration=27.811719ms
  level=error ts=2022-09-01T09:15:20.040394398Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HH4QMT61B3S043, host id: u7EAqekertWqMyOTSH+Azg1hRQkaLP3IKN2zQO83U/YPKqYUDHALc7sl4y4nER1yaKZS5Jisy6U="
  level=error ts=2022-09-01T09:15:20.040524519Z caller=index_set.go:104 table-name=loki_index_19236 msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HH4QMT61B3S043, host id: u7EAqekertWqMyOTSH+Azg1hRQkaLP3IKN2zQO83U/YPKqYUDHALc7sl4y4nER1yaKZS5Jisy6U="
  level=error ts=2022-09-01T09:15:20.04064408Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set " err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HH4QMT61B3S043, host id: u7EAqekertWqMyOTSH+Azg1hRQkaLP3IKN2zQO83U/YPKqYUDHALc7sl4y4nER1yaKZS5Jisy6U="
  level=error ts=2022-09-01T09:15:20.040754812Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set  has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HH4QMT61B3S043, host id: u7EAqekertWqMyOTSH+Azg1hRQkaLP3IKN2zQO83U/YPKqYUDHALc7sl4y4nER1yaKZS5Jisy6U="
  ts=2022-09-01T09:15:20.040920703Z caller=spanlogger.go:80 table-name=loki_index_19236 org_id=fake level=info msg="downloaded index set at query time" duration=31.847183ms
  level=error ts=2022-09-01T09:15:20.041311068Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.041415979Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.041956915Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.042155577Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.042279049Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.042458611Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.042918966Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.043706354Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.044737186Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.04510525Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.045311322Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.045493294Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.045674346Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.045761707Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=info ts=2022-09-01T09:15:20.045748537Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=17.403423ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=538.546µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.060503871Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.060599382Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.060657433Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.060718253Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.060773654Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.0622024Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=info ts=2022-09-01T09:15:20.060790234Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=48.983795ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=52.931µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.062873947Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=84.729852ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=137.981µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.064171062Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=info ts=2022-09-01T09:15:20.064352464Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=84.517329ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=235.782µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.064465395Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.064560186Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.065111532Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.065211083Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.065290424Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.065380635Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.065460656Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.065553057Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.06578655Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.066094473Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.066181364Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.066264395Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.066351776Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.067657871Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.067782112Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.067858723Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.067933334Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.068010144Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.068081765Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.068170216Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.068245177Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=info ts=2022-09-01T09:15:20.06944922Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=85.891125ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.542537ms subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.070460292Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.070567303Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.070648084Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=info ts=2022-09-01T09:15:20.070686194Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=68.123996ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=657.447µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.071017888Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.071113639Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.071254421Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.071669395Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.071962488Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.0720639Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.072537315Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.072623726Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.072704977Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.072783808Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.072866378Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.072949499Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=info ts=2022-09-01T09:15:20.072934809Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=69.532593ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.301574ms subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.047139053Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.047304464Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.047489636Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=info ts=2022-09-01T09:15:20.047464356Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=14m0s step=1m0s duration=12.193076ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.038462ms subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.047662948Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.04783304Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.048073963Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.048342916Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=info ts=2022-09-01T09:15:20.048362476Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=12.067114ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=7.112879ms subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.050009704Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.050110506Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.050700402Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=info ts=2022-09-01T09:15:20.051634613Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=14.587432ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=7.117039ms subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.051894965Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.053524904Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=info ts=2022-09-01T09:15:20.053745096Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=14.142577ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.03038ms subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.054849878Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.055077561Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.055703818Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.056523667Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.057047803Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=info ts=2022-09-01T09:15:20.057235185Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=17.832588ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.032991ms subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.057279695Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.057442597Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.057582129Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=info ts=2022-09-01T09:15:20.058411438Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=18.970751ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.041922ms subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.058980664Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.059219147Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=info ts=2022-09-01T09:15:20.073840919Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=69.679164ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=42.12µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.075572028Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.075877702Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.075950533Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=info ts=2022-09-01T09:15:20.076732931Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=72.012881ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=90.811µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.077222527Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=71.14738ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=160.122µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.077733292Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=70.20236ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=163.822µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.078816605Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=98.96602ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=251.973µs subqueries=1
  level=error ts=2022-09-01T09:15:20.079429061Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.079554293Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.079649844Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.079735175Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.079845606Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.079931737Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.080030288Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.080124679Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.08023716Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.080334881Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.080431562Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.080519773Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.080597044Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.080676525Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.080750646Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.08111808Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.081205971Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.081292362Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.081359463Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.081739367Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.081822058Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.081911409Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.081991Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.082077921Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.082148062Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.082223752Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.082294043Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.082369844Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.082440535Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.082522736Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.082767088Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.082842269Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.08291583Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.082984821Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.083059412Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.083130932Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.083607338Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.083697479Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.08378495Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.084181124Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.08466526Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.084756801Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.084919152Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.085268976Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.085352347Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.085539589Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.086933315Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.041398419Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=info ts=2022-09-01T09:15:20.090798728Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=119.080893ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=42.741µs subqueries=1
  level=error ts=2022-09-01T09:15:20.091315203Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.091815699Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.092217003Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.092364455Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=info ts=2022-09-01T09:15:20.092662198Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=89.749827ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=670.048µs subqueries=1
  level=error ts=2022-09-01T09:15:20.093054043Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.093176354Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.093297905Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.093423077Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.093531728Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.093835121Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=info ts=2022-09-01T09:15:20.093838571Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=105.27987ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=170.741µs subqueries=1
  level=info ts=2022-09-01T09:15:20.094094374Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=92.268305ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=629.127µs subqueries=1
  level=error ts=2022-09-01T09:15:20.095016545Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.095126026Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=info ts=2022-09-01T09:15:20.095151126Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=115.038138ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=255.692µs subqueries=1
  level=error ts=2022-09-01T09:15:20.095716492Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.047025561Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
  level=error ts=2022-09-01T09:15:20.102110543Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HKHBNDA0QM2XFV, host id: IZAZg4rBuud6lQ4VPlCj8KULDDgZCcLXXjf55446uSaNTVQKmNr5tuwuo9ckefxPLeujt3I/m/8="
  level=error ts=2022-09-01T09:15:20.112354367Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HPCV27SC6SEZE4, host id: IP/stnuu8rdVa4/r+LAXWhQ0I/NoKUeoMXsw3Ks8HIFxuW8xSuuwBe8sMsa9WEWDJrGTpr6TEzk="
  level=error ts=2022-09-01T09:15:20.11265064Z caller=index_set.go:104 table-name=loki_index_19236 user-id=fake msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HPCV27SC6SEZE4, host id: IP/stnuu8rdVa4/r+LAXWhQ0I/NoKUeoMXsw3Ks8HIFxuW8xSuuwBe8sMsa9WEWDJrGTpr6TEzk="
  level=error ts=2022-09-01T09:15:20.112782822Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set fake" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HPCV27SC6SEZE4, host id: IP/stnuu8rdVa4/r+LAXWhQ0I/NoKUeoMXsw3Ks8HIFxuW8xSuuwBe8sMsa9WEWDJrGTpr6TEzk="
  level=error ts=2022-09-01T09:15:20.112871093Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set fake has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HPCV27SC6SEZE4, host id: IP/stnuu8rdVa4/r+LAXWhQ0I/NoKUeoMXsw3Ks8HIFxuW8xSuuwBe8sMsa9WEWDJrGTpr6TEzk="
  ts=2022-09-01T09:15:20.1135344Z caller=spanlogger.go:80 table-name=loki_index_19236 user-id=fake org_id=fake level=info msg="downloaded index set at query time" duration=31.52119ms
  level=info ts=2022-09-01T09:15:20.119304384Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=81.707918ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.318155ms subqueries=1
  level=error ts=2022-09-01T09:15:20.119712139Z caller=scheduler_processor.go:182 org_id=fake msg="error notifying frontend about finished query" err="rpc error: code = Canceled desc = context canceled" frontend=172.31.4.123:9095
  level=error ts=2022-09-01T09:15:20.119767549Z caller=scheduler_processor.go:137 org_id=fake msg="error notifying scheduler about finished query" err=EOF addr=172.31.4.123:9095
  level=error ts=2022-09-01T09:15:20.123846475Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HS5ZD9D29W04QW, host id: 9tv2PltR4yYuaYfmdyFz2tnzqfUPHWlxpZZU0vuyVxf/dBfraOmZK9elCSFwTP5O5juLoyuaDVk="
  level=error ts=2022-09-01T09:15:20.128373075Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJATCE0FTJ0QGB, host id: zDfZgvsxG+1SvXY/KEqiBBAwRW72JNQqeBSzy1OrhU985Rv3yRf9MJFv6c0JRhFlc+suafkA+zI="
  level=info ts=2022-09-01T09:15:20.129178254Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=91.87885ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=7.113929ms subqueries=1
  level=error ts=2022-09-01T09:15:20.129351666Z caller=scheduler_processor.go:182 org_id=fake msg="error notifying frontend about finished query" err="rpc error: code = Canceled desc = context canceled" frontend=172.31.4.123:9095
  level=error ts=2022-09-01T09:15:20.129645119Z caller=scheduler_processor.go:137 org_id=fake msg="error notifying scheduler about finished query" err=EOF addr=172.31.4.123:9095
  level=error ts=2022-09-01T09:15:20.129364436Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="context canceled"
  level=info ts=2022-09-01T09:15:20.133029477Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="{job=\"emqx_connection\"} |= \"\"" query_type=filter range_type=range length=30m0s step=10s duration=24.155848ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=119.681µs subqueries=1
  level=error ts=2022-09-01T09:15:20.13328991Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="context canceled"
  level=error ts=2022-09-01T09:15:20.133411841Z caller=scheduler_processor.go:182 org_id=fake msg="error notifying frontend about finished query" err="rpc error: code = Canceled desc = context canceled" frontend=172.31.4.123:9095
  level=error ts=2022-09-01T09:15:20.133460252Z caller=scheduler_processor.go:137 org_id=fake msg="error notifying scheduler about finished query" err=EOF addr=172.31.4.123:9095
  level=error ts=2022-09-01T09:15:20.13508787Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HSJH6RP8217VW2, host id: LWUgF0J2+1Rt3TeLUVVzFF0FrSGmlBevmgYnHY5mZE4igqJRy6zZ1piSlJsB86Gh7+K8I8H8scE="
  level=error ts=2022-09-01T09:15:20.142811036Z caller=index_set.go:104 table-name=loki_index_19236 msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HSJH6RP8217VW2, host id: LWUgF0J2+1Rt3TeLUVVzFF0FrSGmlBevmgYnHY5mZE4igqJRy6zZ1piSlJsB86Gh7+K8I8H8scE="
  level=error ts=2022-09-01T09:15:20.147184834Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set " err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HSJH6RP8217VW2, host id: LWUgF0J2+1Rt3TeLUVVzFF0FrSGmlBevmgYnHY5mZE4igqJRy6zZ1piSlJsB86Gh7+K8I8H8scE="
  level=error ts=2022-09-01T09:15:20.147266685Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set  has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HSJH6RP8217VW2, host id: LWUgF0J2+1Rt3TeLUVVzFF0FrSGmlBevmgYnHY5mZE4igqJRy6zZ1piSlJsB86Gh7+K8I8H8scE="
  ts=2022-09-01T09:15:20.147494828Z caller=spanlogger.go:80 table-name=loki_index_19236 org_id=fake level=info msg="downloaded index set at query time" duration=33.765565ms
  level=info ts=2022-09-01T09:15:20.14775526Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=14m0s step=1m0s duration=37.136332ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=140.302µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.145079981Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="context canceled"
  level=error ts=2022-09-01T09:15:20.138966963Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HV6ZSRBM0V30V1, host id: jeAYrrbaptycrSYDbrTONV+6t87iNzC1VI1Xmr7CfaK2Dxs7y8oP6zEdVPHgwWrZB8hYDWQnkXw="
  level=error ts=2022-09-01T09:15:20.150981866Z caller=index_set.go:104 table-name=loki_index_19236 user-id=fake msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HV6ZSRBM0V30V1, host id: jeAYrrbaptycrSYDbrTONV+6t87iNzC1VI1Xmr7CfaK2Dxs7y8oP6zEdVPHgwWrZB8hYDWQnkXw="
  level=error ts=2022-09-01T09:15:20.151053107Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set fake" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HV6ZSRBM0V30V1, host id: jeAYrrbaptycrSYDbrTONV+6t87iNzC1VI1Xmr7CfaK2Dxs7y8oP6zEdVPHgwWrZB8hYDWQnkXw="
  level=error ts=2022-09-01T09:15:20.151114238Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set fake has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HV6ZSRBM0V30V1, host id: jeAYrrbaptycrSYDbrTONV+6t87iNzC1VI1Xmr7CfaK2Dxs7y8oP6zEdVPHgwWrZB8hYDWQnkXw="
  ts=2022-09-01T09:15:20.151509702Z caller=spanlogger.go:80 table-name=loki_index_19236 user-id=fake org_id=fake level=info msg="downloaded index set at query time" duration=33.407241ms
  level=info ts=2022-09-01T09:15:20.156467867Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=14m0s step=1m0s duration=50.368479ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=134.952µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.157229936Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=14m0s step=1m0s duration=48.803382ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.030391ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.157619Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=14m0s step=1m0s duration=48.438188ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=139.051µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.142723615Z caller=metrics.go:133 component=frontend org_id=fake latency=fast query="{job=\"emqx_connection\"} |= ``" query_type=filter range_type=range length=6h0m0s step=10s duration=287.652126ms status=200 limit=1000 returned_lines=0 throughput=3.8MB total_bytes=1.1MB total_entries=1000 queue_time=14.050061ms subqueries=2
  level=error ts=2022-09-01T09:15:20.162489204Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HMMNZBHASRT7R4, host id: CGifRpGnurNGkwTGXC4Tv/a3BUY4kPeV/XjFOJUWPPTvqVSZ9lsWWSqHSwBtUVYkpIpJH5dH1T0="
  level=error ts=2022-09-01T09:15:20.173451746Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HT2Y3T2J2JDHE1, host id: LX+1F3mcKP+ZDzOJXzYFGE2okd5tmFz/c6n4HPvaGdOPEJcPSLqbb3COrc4K2V7etQGlTp+/R88="
  level=error ts=2022-09-01T09:15:20.173545277Z caller=index_set.go:104 table-name=loki_index_19236 msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HT2Y3T2J2JDHE1, host id: LX+1F3mcKP+ZDzOJXzYFGE2okd5tmFz/c6n4HPvaGdOPEJcPSLqbb3COrc4K2V7etQGlTp+/R88="
  level=error ts=2022-09-01T09:15:20.173583577Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set " err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HT2Y3T2J2JDHE1, host id: LX+1F3mcKP+ZDzOJXzYFGE2okd5tmFz/c6n4HPvaGdOPEJcPSLqbb3COrc4K2V7etQGlTp+/R88="
  level=error ts=2022-09-01T09:15:20.173657018Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set  has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HT2Y3T2J2JDHE1, host id: LX+1F3mcKP+ZDzOJXzYFGE2okd5tmFz/c6n4HPvaGdOPEJcPSLqbb3COrc4K2V7etQGlTp+/R88="
  ts=2022-09-01T09:15:20.173928141Z caller=spanlogger.go:80 table-name=loki_index_19236 org_id=fake level=info msg="downloaded index set at query time" duration=22.226627ms
  level=info ts=2022-09-01T09:15:20.174189204Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=14m0s step=1m0s duration=56.392916ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=337.344µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.174641159Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=14m0s step=1m0s duration=54.173392ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=323.684µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.175181055Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=14m0s step=1m0s duration=54.091561ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=331.634µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.176998375Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=48.984774ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=46.21µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.177350339Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=37.619308ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=197.992µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.178696774Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=55.683489ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=45.881µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.179289821Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=59.905595ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=328.743µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.179584114Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=48.892943ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=174.662µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.185319738Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=49.194296ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=156.182µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.18729918Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HHV7E6JCKWQV4M, host id: V67v3NkmhBvZ9HXln7uiMiG07NBZQAjHOzv8aUb69A4TpxBIBGmc0wAfBJMi/+GR+xUh7N10vDs="
  level=info ts=2022-09-01T09:15:20.188387692Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=58.163516ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=101.501µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.190268303Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=47.841802ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=128.3µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.191740689Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=59.356609ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=226.303µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.192934392Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=53.95283ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=182.162µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.193561899Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=54.344853ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=786.388µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.193832542Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=54.390905ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=203.352µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.194097345Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=54.086811ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=179.892µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.195143877Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HHYWW9GPD0QC90, host id: 4wL3DQp3/rFEOnUI13xq7tM3t7MwRtveEFW8+MyzP85ORUTu99HvB4LYk/ICQLsLs1Qn5Lt5+EA="
  level=error ts=2022-09-01T09:15:20.195201228Z caller=index_set.go:104 table-name=loki_index_19236 user-id=fake msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HHYWW9GPD0QC90, host id: 4wL3DQp3/rFEOnUI13xq7tM3t7MwRtveEFW8+MyzP85ORUTu99HvB4LYk/ICQLsLs1Qn5Lt5+EA="
  level=error ts=2022-09-01T09:15:20.195295379Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set fake" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HHYWW9GPD0QC90, host id: 4wL3DQp3/rFEOnUI13xq7tM3t7MwRtveEFW8+MyzP85ORUTu99HvB4LYk/ICQLsLs1Qn5Lt5+EA="
  level=error ts=2022-09-01T09:15:20.1953854Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set fake has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HHYWW9GPD0QC90, host id: 4wL3DQp3/rFEOnUI13xq7tM3t7MwRtveEFW8+MyzP85ORUTu99HvB4LYk/ICQLsLs1Qn5Lt5+EA="
  ts=2022-09-01T09:15:20.195541901Z caller=spanlogger.go:80 table-name=loki_index_19236 user-id=fake org_id=fake level=info msg="downloaded index set at query time" duration=26.863048ms
  level=error ts=2022-09-01T09:15:20.207495684Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HKB8ZMMB51MZ7C, host id: JQZcE3BexzQhxccWv/UtQA7X3YKGJBIII6LhFEGc3tWa/fdAKUctBqZ/z+mbblWZOZvCeFFY0S8="
  level=error ts=2022-09-01T09:15:20.211208195Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HKXS1DN2J8RTXD, host id: tMxufyHYPqAQ9ftYLd6Y28+GBzW8M2Zh3y3VFh/pxE9pMWq8z9BbwuOIZ72Udh7dcbDH7jKQAf8="
  level=error ts=2022-09-01T09:15:20.218025011Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HVRFKHHYETN16W, host id: IGeA94cM3WxNXIIZ42KtkgbD/F0TxVUwn1yYcxg/sO4Om0pEF0eOzMF9FPHA+G11qVnyhUDn3vg="
  level=error ts=2022-09-01T09:15:20.218366855Z caller=index_set.go:104 table-name=loki_index_19236 user-id=fake msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HVRFKHHYETN16W, host id: IGeA94cM3WxNXIIZ42KtkgbD/F0TxVUwn1yYcxg/sO4Om0pEF0eOzMF9FPHA+G11qVnyhUDn3vg="
  level=error ts=2022-09-01T09:15:20.218643658Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set fake" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HVRFKHHYETN16W, host id: IGeA94cM3WxNXIIZ42KtkgbD/F0TxVUwn1yYcxg/sO4Om0pEF0eOzMF9FPHA+G11qVnyhUDn3vg="
  level=error ts=2022-09-01T09:15:20.218718169Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set fake has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HVRFKHHYETN16W, host id: IGeA94cM3WxNXIIZ42KtkgbD/F0TxVUwn1yYcxg/sO4Om0pEF0eOzMF9FPHA+G11qVnyhUDn3vg="
  ts=2022-09-01T09:15:20.219008802Z caller=spanlogger.go:80 table-name=loki_index_19236 user-id=fake org_id=fake level=info msg="downloaded index set at query time" duration=20.377656ms
  level=error ts=2022-09-01T09:15:20.218304284Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  level=error ts=2022-09-01T09:15:20.219066213Z caller=index_set.go:104 table-name=loki_index_19236 msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  level=error ts=2022-09-01T09:15:20.219105503Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set " err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  level=error ts=2022-09-01T09:15:20.219367606Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set  has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  ts=2022-09-01T09:15:20.219460097Z caller=spanlogger.go:80 table-name=loki_index_19236 org_id=fake level=info msg="downloaded index set at query time" duration=23.721424ms
  level=error ts=2022-09-01T09:15:20.219515618Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  level=warn ts=2022-09-01T09:15:20.219596239Z caller=grpc_logging.go:64 method=/logproto.Querier/QuerySample duration=10.914262ms err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=" msg=gRPC
  level=info ts=2022-09-01T09:15:20.220426338Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=53.905589ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=3.278816ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.221244027Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=66.039654ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=100.122µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.22152289Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=18.387774ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.673089ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.221770533Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=22.53148ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=499.024µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.222786154Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unknown desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=\n"
  level=info ts=2022-09-01T09:15:20.228633569Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=72.092021ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=673.397µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.230713952Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=44.416144ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=129.882µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.230911114Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=74.751561ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=671.168µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.230795923Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=75.152335ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=58.37µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.231954606Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=54.152402ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=567.056µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.232159358Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  level=warn ts=2022-09-01T09:15:20.23235226Z caller=grpc_logging.go:64 method=/logproto.Querier/QuerySample duration=19.581577ms err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=" msg=gRPC
  level=error ts=2022-09-01T09:15:20.232516072Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  level=warn ts=2022-09-01T09:15:20.232879306Z caller=grpc_logging.go:64 method=/logproto.Querier/QuerySample duration=20.445987ms err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=" msg=gRPC
  level=error ts=2022-09-01T09:15:20.233006888Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  level=warn ts=2022-09-01T09:15:20.233123919Z caller=grpc_logging.go:64 method=/logproto.Querier/QuerySample duration=21.080804ms err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=" msg=gRPC
  level=info ts=2022-09-01T09:15:20.233146669Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=75.873403ms status=200 limit=1000 returned_lines=0 throughput=15MB total_bytes=1.1MB total_entries=30 queue_time=109.701µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.23326693Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=60.144718ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=60.101µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.233582324Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  level=warn ts=2022-09-01T09:15:20.233706835Z caller=grpc_logging.go:64 method=/logproto.Querier/QuerySample duration=23.096866ms err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=" msg=gRPC
  level=info ts=2022-09-01T09:15:20.233385652Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=50.906355ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=531.086µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.234275622Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  level=warn ts=2022-09-01T09:15:20.234420813Z caller=grpc_logging.go:64 method=/logproto.Querier/QuerySample duration=24.131238ms err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=" msg=gRPC
  level=info ts=2022-09-01T09:15:20.234976369Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=22.698492ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=72.911µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.235290303Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  level=warn ts=2022-09-01T09:15:20.235796499Z caller=grpc_logging.go:64 method=/logproto.Querier/QuerySample duration=26.642836ms err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=" msg=gRPC
  level=error ts=2022-09-01T09:15:20.23593461Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  level=warn ts=2022-09-01T09:15:20.236254134Z caller=grpc_logging.go:64 method=/logproto.Querier/QuerySample duration=27.299834ms err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=" msg=gRPC
  level=error ts=2022-09-01T09:15:20.236383355Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  level=warn ts=2022-09-01T09:15:20.236505226Z caller=grpc_logging.go:64 method=/logproto.Querier/QuerySample duration=27.606826ms err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=" msg=gRPC
  level=error ts=2022-09-01T09:15:20.236629198Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  level=info ts=2022-09-01T09:15:20.236733839Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=55.310734ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=412.195µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.23677905Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  level=warn ts=2022-09-01T09:15:20.237200714Z caller=grpc_logging.go:64 method=/logproto.Querier/QuerySample duration=28.79621ms err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=" msg=gRPC
  level=info ts=2022-09-01T09:15:20.237309935Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=55.092192ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=525.305µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.237342676Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  level=warn ts=2022-09-01T09:15:20.237623009Z caller=grpc_logging.go:64 method=/logproto.Querier/QuerySample duration=30.228166ms err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=" msg=gRPC
  level=info ts=2022-09-01T09:15:20.237645629Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=26.797257ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=103.852µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.23776465Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=25.741046ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=69.02µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.238223826Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  level=warn ts=2022-09-01T09:15:20.238348477Z caller=grpc_logging.go:64 method=/logproto.Querier/QuerySample duration=31.180957ms err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=" msg=gRPC
  level=error ts=2022-09-01T09:15:20.238472528Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  level=warn ts=2022-09-01T09:15:20.23859248Z caller=grpc_logging.go:64 method=/logproto.Querier/QuerySample duration=31.521891ms err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=" msg=gRPC
  level=error ts=2022-09-01T09:15:20.238739601Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  level=warn ts=2022-09-01T09:15:20.236750629Z caller=grpc_logging.go:64 method=/logproto.Querier/QuerySample duration=30.800462ms err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=" msg=gRPC
  level=error ts=2022-09-01T09:15:20.238919373Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  level=warn ts=2022-09-01T09:15:20.239036645Z caller=grpc_logging.go:64 method=/logproto.Querier/QuerySample duration=39.089345ms err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=" msg=gRPC
  level=warn ts=2022-09-01T09:15:20.239196236Z caller=grpc_logging.go:64 method=/logproto.Querier/QuerySample duration=40.918674ms err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=" msg=gRPC
  level=error ts=2022-09-01T09:15:20.239332058Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  level=warn ts=2022-09-01T09:15:20.239453959Z caller=grpc_logging.go:64 duration=39.745801ms method=/logproto.Querier/QuerySample err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=" msg=gRPC
  level=info ts=2022-09-01T09:15:20.23951609Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=42.26627ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=134.371µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.239623111Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  level=warn ts=2022-09-01T09:15:20.239742562Z caller=grpc_logging.go:64 method=/logproto.Querier/QuerySample duration=35.300112ms err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=" msg=gRPC
  level=error ts=2022-09-01T09:15:20.239870874Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8="
  level=error ts=2022-09-01T09:15:20.239959365Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HH1SFQA06XE7QD, host id: ozfFI3/8z+a3Rwd3RL/c+3Pdh5tTs13kmY8bbYhbUhaFgF//lDFd0SPR3QPnflBHlR14mJazn4Q="
  level=warn ts=2022-09-01T09:15:20.239984895Z caller=grpc_logging.go:64 duration=40.5213ms method=/logproto.Querier/QuerySample err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=" msg=gRPC
  level=info ts=2022-09-01T09:15:20.241025207Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=32.108877ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=69.391µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.24131847Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=49.50579ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=51.58µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.241436341Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=37.127013ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=369.014µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.2413185Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=36.684307ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=323.074µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.241663914Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=37.721969ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=318.813µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.241893646Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=38.63392ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=809.569µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.241891616Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=44.252902ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=132.512µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.242121539Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=45.209172ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=129.622µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.24221136Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=39.407528ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.673058ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.242429692Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=38.66212ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=148.782µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.242543774Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=40.437929ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.518697ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.242429712Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=39.980725ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.668118ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.242769456Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=36.859199ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=308.303µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.243596835Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=41.302079ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=16.76µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.245285524Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unknown desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=\n"
  level=error ts=2022-09-01T09:15:20.245409485Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unknown desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=\n"
  level=error ts=2022-09-01T09:15:20.245484396Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unknown desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=\n"
  level=error ts=2022-09-01T09:15:20.248098565Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unknown desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=\n"
  level=error ts=2022-09-01T09:15:20.248283857Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unknown desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=\n"
  level=error ts=2022-09-01T09:15:20.248391209Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unknown desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=\n"
  level=error ts=2022-09-01T09:15:20.248868284Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unknown desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=\n"
  level=error ts=2022-09-01T09:15:20.24941269Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unknown desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=\n"
  level=error ts=2022-09-01T09:15:20.249711303Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unknown desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=\n"
  level=error ts=2022-09-01T09:15:20.250045297Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unknown desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=\n"
  level=error ts=2022-09-01T09:15:20.250155658Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unknown desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=\n"
  level=error ts=2022-09-01T09:15:20.250264349Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unknown desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=\n"
  level=error ts=2022-09-01T09:15:20.250384621Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unknown desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=\n"
  level=error ts=2022-09-01T09:15:20.250731295Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unknown desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=\n"
  level=error ts=2022-09-01T09:15:20.250894366Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unknown desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=\n"
  level=error ts=2022-09-01T09:15:20.251006928Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unknown desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=\n"
  level=error ts=2022-09-01T09:15:20.251125209Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unknown desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=\n"
  level=error ts=2022-09-01T09:15:20.25123177Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unknown desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJYE6NEZWYWHEQ, host id: l+uwszjiEjdQPbzLzN0qxxtwWDf0lQB4JmVASVr7k+Ho3CQtQfKV4ZuFwX+TuEAprpcL467oQL8=\n"
  level=error ts=2022-09-01T09:15:20.25483625Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HSN5EGPPH808P3, host id: E+J6bP1nIfpf4uuT6ch2FubDCHjKh7NCVB1EYK6/OPrAA9QWY9sVHNypkT5N/WqDDh+3fDJ4JgA="
  level=error ts=2022-09-01T09:15:20.254991712Z caller=index_set.go:104 table-name=loki_index_19236 user-id=fake msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HSN5EGPPH808P3, host id: E+J6bP1nIfpf4uuT6ch2FubDCHjKh7NCVB1EYK6/OPrAA9QWY9sVHNypkT5N/WqDDh+3fDJ4JgA="
  level=error ts=2022-09-01T09:15:20.255169174Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set fake" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HSN5EGPPH808P3, host id: E+J6bP1nIfpf4uuT6ch2FubDCHjKh7NCVB1EYK6/OPrAA9QWY9sVHNypkT5N/WqDDh+3fDJ4JgA="
  level=error ts=2022-09-01T09:15:20.255204634Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set fake has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HSN5EGPPH808P3, host id: E+J6bP1nIfpf4uuT6ch2FubDCHjKh7NCVB1EYK6/OPrAA9QWY9sVHNypkT5N/WqDDh+3fDJ4JgA="
  level=error ts=2022-09-01T09:15:20.255589779Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HSN5EGPPH808P3, host id: E+J6bP1nIfpf4uuT6ch2FubDCHjKh7NCVB1EYK6/OPrAA9QWY9sVHNypkT5N/WqDDh+3fDJ4JgA="
  level=info ts=2022-09-01T09:15:20.256050484Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=1.628718ms status=500 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=7.387492ms subqueries=1 source=logvolhist
  ts=2022-09-01T09:15:20.257216527Z caller=spanlogger.go:80 table-name=loki_index_19236 user-id=fake org_id=fake level=info msg="downloaded index set at query time" duration=29.70277ms
  level=error ts=2022-09-01T09:15:20.259424271Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HSN5EGPPH808P3, host id: E+J6bP1nIfpf4uuT6ch2FubDCHjKh7NCVB1EYK6/OPrAA9QWY9sVHNypkT5N/WqDDh+3fDJ4JgA=\n"
  level=error ts=2022-09-01T09:15:20.260520603Z caller=series_index_store.go:527 org_id=fake msg="error querying storage" err="context canceled"
  level=error ts=2022-09-01T09:15:20.268409271Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HNEN5ERB0RPVG1, host id: CnHmWvD7vY8a3WHIDYKfHHMx2GEc/cWUR4LWVB9OeThl+JCAArhgKt/o2y87fLU7WluPWWw6zjY="
  level=error ts=2022-09-01T09:15:20.268958587Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HKCQB6SVCYT24K, host id: qgzfyZwJC6BZ/htWylyWw5BEr8BGZDhpj4vX/XeG0VwYFlwQ0tgNE8mzI2+gG4LdNN5P94IsUTc="
  level=error ts=2022-09-01T09:15:20.277572493Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HPB7D0Z5HY3E68, host id: KC5yIkDKR95hvz4AsYQk2fwZpf/u55t0z3WSDZHFaZMqkbjWgje5NmLJ/4xtNAdWSG9/lRwrw+U="
  level=error ts=2022-09-01T09:15:20.277662794Z caller=index_set.go:104 table-name=loki_index_19236 msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HPB7D0Z5HY3E68, host id: KC5yIkDKR95hvz4AsYQk2fwZpf/u55t0z3WSDZHFaZMqkbjWgje5NmLJ/4xtNAdWSG9/lRwrw+U="
  level=error ts=2022-09-01T09:15:20.277711074Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set " err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HPB7D0Z5HY3E68, host id: KC5yIkDKR95hvz4AsYQk2fwZpf/u55t0z3WSDZHFaZMqkbjWgje5NmLJ/4xtNAdWSG9/lRwrw+U="
  level=error ts=2022-09-01T09:15:20.277794565Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set  has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HPB7D0Z5HY3E68, host id: KC5yIkDKR95hvz4AsYQk2fwZpf/u55t0z3WSDZHFaZMqkbjWgje5NmLJ/4xtNAdWSG9/lRwrw+U="
  ts=2022-09-01T09:15:20.278370602Z caller=spanlogger.go:80 table-name=loki_index_19236 org_id=fake level=info msg="downloaded index set at query time" duration=21.477528ms
  level=info ts=2022-09-01T09:15:20.279682046Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=39.157425ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=49.141µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.28000317Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HZRN2HNAR8Q8S9, host id: OBrP3C7NrpO4aI9zyMHMgYkRJ6MnRoAySzrvdFLOmorpmJIKLA6o+PY75t0siYlj8J3+D921k7M="
  level=error ts=2022-09-01T09:15:20.28004731Z caller=index_set.go:104 table-name=loki_index_19236 user-id=fake msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HZRN2HNAR8Q8S9, host id: OBrP3C7NrpO4aI9zyMHMgYkRJ6MnRoAySzrvdFLOmorpmJIKLA6o+PY75t0siYlj8J3+D921k7M="
  level=error ts=2022-09-01T09:15:20.280091441Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set fake" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HZRN2HNAR8Q8S9, host id: OBrP3C7NrpO4aI9zyMHMgYkRJ6MnRoAySzrvdFLOmorpmJIKLA6o+PY75t0siYlj8J3+D921k7M="
  level=info ts=2022-09-01T09:15:20.282118893Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=57.843943ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=36.15µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.280132311Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set fake has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HZRN2HNAR8Q8S9, host id: OBrP3C7NrpO4aI9zyMHMgYkRJ6MnRoAySzrvdFLOmorpmJIKLA6o+PY75t0siYlj8J3+D921k7M="
  ts=2022-09-01T09:15:20.282562818Z caller=spanlogger.go:80 table-name=loki_index_19236 user-id=fake org_id=fake level=info msg="downloaded index set at query time" duration=26.049249ms
  level=info ts=2022-09-01T09:15:20.282588258Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=54.412665ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=329.784µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.28276494Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=52.859327ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=60.111µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.282464107Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=51.956777ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=314.984µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.293614881Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HYAWW5EVYH8YBQ, host id: nWFJI90bzTldOiPMFH+mA0nTc2WdiBgwKI4l8Z1mp/e8AFGXF/RZDvNat7LRuPz1vjJCDQRRpXc="
  level=error ts=2022-09-01T09:15:20.294607532Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HW0VX3R362R2DD, host id: HSp5JNkExqK1ztaX8S0OTVVLNT6XBYmlfWsidRjP8TbeC7PLAFr9eBtCl6pO3eaDnLZ8we9rjBY="
  level=error ts=2022-09-01T09:15:20.301437948Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJCF805AS9CP9G, host id: Lq3m0PBtJO5wNYzr3SWpkkRDZSYO8KI/0eubIP0vV3GN8l2eN/ioUWr7G6qHDiwCIQQYqvewEH4="
  level=error ts=2022-09-01T09:15:20.301498509Z caller=index_set.go:104 table-name=loki_index_19236 msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJCF805AS9CP9G, host id: Lq3m0PBtJO5wNYzr3SWpkkRDZSYO8KI/0eubIP0vV3GN8l2eN/ioUWr7G6qHDiwCIQQYqvewEH4="
  level=error ts=2022-09-01T09:15:20.301547509Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set " err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJCF805AS9CP9G, host id: Lq3m0PBtJO5wNYzr3SWpkkRDZSYO8KI/0eubIP0vV3GN8l2eN/ioUWr7G6qHDiwCIQQYqvewEH4="
  level=error ts=2022-09-01T09:15:20.30158816Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set  has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HJCF805AS9CP9G, host id: Lq3m0PBtJO5wNYzr3SWpkkRDZSYO8KI/0eubIP0vV3GN8l2eN/ioUWr7G6qHDiwCIQQYqvewEH4="
  ts=2022-09-01T09:15:20.301857323Z caller=spanlogger.go:80 table-name=loki_index_19236 org_id=fake level=info msg="downloaded index set at query time" duration=18.771248ms
  level=error ts=2022-09-01T09:15:20.301932873Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HSDXS3Y6JG0C8E, host id: uBHlxyG0SqBviZNVcAWGOGlK0rjPDrcAhWMNsGpxVhtbH/Iz+SFYRhOkIVyByEsz5+LzKrrRi/o="
  level=error ts=2022-09-01T09:15:20.301974974Z caller=index_set.go:104 table-name=loki_index_19236 user-id=fake msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HSDXS3Y6JG0C8E, host id: uBHlxyG0SqBviZNVcAWGOGlK0rjPDrcAhWMNsGpxVhtbH/Iz+SFYRhOkIVyByEsz5+LzKrrRi/o="
  level=error ts=2022-09-01T09:15:20.302012124Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set fake" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HSDXS3Y6JG0C8E, host id: uBHlxyG0SqBviZNVcAWGOGlK0rjPDrcAhWMNsGpxVhtbH/Iz+SFYRhOkIVyByEsz5+LzKrrRi/o="
  level=error ts=2022-09-01T09:15:20.302048515Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set fake has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HSDXS3Y6JG0C8E, host id: uBHlxyG0SqBviZNVcAWGOGlK0rjPDrcAhWMNsGpxVhtbH/Iz+SFYRhOkIVyByEsz5+LzKrrRi/o="
  ts=2022-09-01T09:15:20.302098135Z caller=spanlogger.go:80 table-name=loki_index_19236 user-id=fake org_id=fake level=info msg="downloaded index set at query time" duration=18.108351ms
  level=info ts=2022-09-01T09:15:20.302184756Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=28.288264ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=111.681µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.303630642Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=52.397552ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=226.663µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.303630602Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=43.710115ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=528.526µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.304059667Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=40.590201ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=187.702µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.304088207Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=55.632488ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=3.522179ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.30431321Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=53.872148ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=3.212676ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.304481292Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=40.136436ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=52.801µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.305988138Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=40.660392ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=172.912µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.307273683Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=37.547767ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=93.721µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.308204803Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=42.370011ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=181.402µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.308625258Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=42.466122ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=183.582µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.310521919Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=42.429152ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=52.791µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.311162496Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=42.216869ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=104.791µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.311792763Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=47.71119ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=527.776µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.311887954Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=40.941275ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=81.271µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.312098496Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=42.552952ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=103.771µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.312309979Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=42.774035ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=101.882µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.312953146Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=39.739422ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=60.071µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.313137438Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=39.752952ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=53.511µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.313553502Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=61.232861ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=58.751µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.313671134Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=42.30073ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=775.068µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.313735755Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=59.623243ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=3.515289ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.314076028Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=42.145898ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=32.74µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.31425387Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=41.508442ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=89.051µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.314607704Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HQZVDH1HECW9WB, host id: 9ZsE24nznQkNz/jy+plKs+vhSzqI7ZMYSINI/6Y8wz7B21sXDlYb3OSLDvSyJInmUi/Nt5Ls/DM="
  level=info ts=2022-09-01T09:15:20.315442313Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=52.402933ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=524.876µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.317183383Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=56.323076ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.948851ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.317321224Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=55.8016ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=525.635µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.322564303Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HSSWR5Y9YSBN2D, host id: Jhhq1XmkFrNdWbJCckDDOL+q+iJSn8gZ99ZZDDGvJQ8QycsbxAy1PYbNehidoWw70sY0F33NgXM="
  level=error ts=2022-09-01T09:15:20.332416642Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HPRJBZAQWVRNG4, host id: nqrtPjuUI4CX3YxD0UoECBIZt6GAdNimsjf76LLgvg7sxwx4j6iwJHI3UUEESqpjLbHWnfvOSCY="
  level=error ts=2022-09-01T09:15:20.332521173Z caller=index_set.go:104 table-name=loki_index_19236 msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HPRJBZAQWVRNG4, host id: nqrtPjuUI4CX3YxD0UoECBIZt6GAdNimsjf76LLgvg7sxwx4j6iwJHI3UUEESqpjLbHWnfvOSCY="
  level=error ts=2022-09-01T09:15:20.332571294Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set " err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HPRJBZAQWVRNG4, host id: nqrtPjuUI4CX3YxD0UoECBIZt6GAdNimsjf76LLgvg7sxwx4j6iwJHI3UUEESqpjLbHWnfvOSCY="
  level=error ts=2022-09-01T09:15:20.332833117Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set  has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HPRJBZAQWVRNG4, host id: nqrtPjuUI4CX3YxD0UoECBIZt6GAdNimsjf76LLgvg7sxwx4j6iwJHI3UUEESqpjLbHWnfvOSCY="
  ts=2022-09-01T09:15:20.333072869Z caller=spanlogger.go:80 table-name=loki_index_19236 org_id=fake level=info msg="downloaded index set at query time" duration=30.859863ms
  level=info ts=2022-09-01T09:15:20.333355663Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=51.901347ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=36.07µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.336364156Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=50.451451ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=351.384µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.336780711Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=49.293868ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=57.931µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.337444368Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=48.963674ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.631248ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.338362418Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=48.495739ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=2.132284ms subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.338988395Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HP55YEPCTFV3EV, host id: LaMfVGNcLVW1nb39s/U3eZkDxYQuEc2tUFyMpooSctQR454JnEh18RUr5jXUX+sst2jZqTg9l2U="
  level=error ts=2022-09-01T09:15:20.339044446Z caller=index_set.go:104 table-name=loki_index_19236 user-id=fake msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HP55YEPCTFV3EV, host id: LaMfVGNcLVW1nb39s/U3eZkDxYQuEc2tUFyMpooSctQR454JnEh18RUr5jXUX+sst2jZqTg9l2U="
  level=error ts=2022-09-01T09:15:20.339242138Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set fake" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HP55YEPCTFV3EV, host id: LaMfVGNcLVW1nb39s/U3eZkDxYQuEc2tUFyMpooSctQR454JnEh18RUr5jXUX+sst2jZqTg9l2U="
  level=error ts=2022-09-01T09:15:20.339370699Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set fake has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HP55YEPCTFV3EV, host id: LaMfVGNcLVW1nb39s/U3eZkDxYQuEc2tUFyMpooSctQR454JnEh18RUr5jXUX+sst2jZqTg9l2U="
  ts=2022-09-01T09:15:20.339540761Z caller=spanlogger.go:80 table-name=loki_index_19236 user-id=fake org_id=fake level=info msg="downloaded index set at query time" duration=30.915643ms
  level=error ts=2022-09-01T09:15:20.350051128Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HN9DPR8BH69PFN, host id: oZ/SK4d1qp7y4Lp+WMaz6hoh2gNZV9M3YHbyohcTBHNTOv4pzRxl9EAPcVt0e6dbCVlmdWJ/J+c="
  level=error ts=2022-09-01T09:15:20.35021267Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HKQAXHJQWAGS8R, host id: SMrx9ffhLK/w7MOdvwmNxlNeFWSL/nfO9+kmfA9ilRP2/vV8JKUPwU0zvLXCQipixHJcVSyQwq0="
  level=error ts=2022-09-01T09:15:20.359394502Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HPMHXJ6BWYMQCG, host id: +1Pwilxv6t6Ov4ZQGJIoztKyUtcGHo//8CgNqhNHImyTnBeZLgvlly/K1L3cOJaRXJa4PKbaigo="
  level=error ts=2022-09-01T09:15:20.359493153Z caller=index_set.go:104 table-name=loki_index_19236 user-id=fake msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HPMHXJ6BWYMQCG, host id: +1Pwilxv6t6Ov4ZQGJIoztKyUtcGHo//8CgNqhNHImyTnBeZLgvlly/K1L3cOJaRXJa4PKbaigo="
  level=error ts=2022-09-01T09:15:20.359541563Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set fake" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HPMHXJ6BWYMQCG, host id: +1Pwilxv6t6Ov4ZQGJIoztKyUtcGHo//8CgNqhNHImyTnBeZLgvlly/K1L3cOJaRXJa4PKbaigo="
  level=error ts=2022-09-01T09:15:20.359583864Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set fake has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HPMHXJ6BWYMQCG, host id: +1Pwilxv6t6Ov4ZQGJIoztKyUtcGHo//8CgNqhNHImyTnBeZLgvlly/K1L3cOJaRXJa4PKbaigo="
  ts=2022-09-01T09:15:20.359730786Z caller=spanlogger.go:80 table-name=loki_index_19236 user-id=fake org_id=fake level=info msg="downloaded index set at query time" duration=19.059012ms
  level=error ts=2022-09-01T09:15:20.360336642Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HYG513C369A45K, host id: 60FMqjP0ncti6hKEZhNF7S5MioYaQn6VKjMUQ7bY9jm5xNA9pwOGl65B0k4Ei+6XQLJ77pautzE="
  level=error ts=2022-09-01T09:15:20.360426713Z caller=index_set.go:104 table-name=loki_index_19236 msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HYG513C369A45K, host id: 60FMqjP0ncti6hKEZhNF7S5MioYaQn6VKjMUQ7bY9jm5xNA9pwOGl65B0k4Ei+6XQLJ77pautzE="
  level=error ts=2022-09-01T09:15:20.360500354Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set " err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HYG513C369A45K, host id: 60FMqjP0ncti6hKEZhNF7S5MioYaQn6VKjMUQ7bY9jm5xNA9pwOGl65B0k4Ei+6XQLJ77pautzE="
  level=error ts=2022-09-01T09:15:20.365943335Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set  has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HYG513C369A45K, host id: 60FMqjP0ncti6hKEZhNF7S5MioYaQn6VKjMUQ7bY9jm5xNA9pwOGl65B0k4Ei+6XQLJ77pautzE="
  level=info ts=2022-09-01T09:15:20.365977085Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=32.249359ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=36.12µs subqueries=1 source=logvolhist
  ts=2022-09-01T09:15:20.366246168Z caller=spanlogger.go:80 table-name=loki_index_19236 org_id=fake level=info msg="downloaded index set at query time" duration=26.478264ms
  level=info ts=2022-09-01T09:15:20.361129901Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=27.281503ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=150.262µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.361184842Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=19.694049ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=153.392µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.361322753Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=43.029688ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=58.381µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.361336803Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=19.450676ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=157.061µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.361478545Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=42.042527ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=55.98µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.361480195Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=19.527137ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=93.722µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.361630687Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=40.024595ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=59.84µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.361628687Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=19.393665ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=94.851µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.362959721Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=38.6364ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=142.722µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.363004782Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=19.403786ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=57.401µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.363378626Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=20.372926ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=88.611µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.363458187Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=39.496769ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=360.924µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.363617939Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=38.514698ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=34.941µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.363658879Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=19.966422ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=87.041µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.363955692Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=35.758487ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=3.429528ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.363973173Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=19.82212ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=92.681µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.364124664Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=19.121013ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.344675ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.364124644Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=35.465154ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=227.833µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.364341987Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=18.89388ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.260704ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.364370507Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=34.175409ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=5.37569ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.364505069Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=33.712465ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=3.63475ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.364500899Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=18.745318ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=2.439597ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.365154116Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=34.149789ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=80.951µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.365279537Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=25.415062ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=54.701µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.365446989Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=30.270446ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=147.912µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.365446999Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=34.300041ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=2.961843ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.365594071Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=33.529172ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=85.58µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.365594011Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=29.796701ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=148.471µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.365759703Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=33.707075ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=895.6µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.365760153Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=27.128761ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=89.911µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.361011Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=53.415233ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=50.981µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.383825423Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HPP7YSVZWJ14FT, host id: DNCgcu3wSCn9K6efR6VSl5/GJ5y99K6MGthjrxQ/Y5wuNx3EA29UIyxAyjRfXMd3ZSX2HMWtfuA="
  level=error ts=2022-09-01T09:15:20.392693922Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HV1GHW3TQT7JSC, host id: suqqNGAzw3tjOuHcBTDJrdiT4tcZRkqNxWcUGADBkmf0z7RVnHQ7SceRkwsj4j72jLcMF5WCJQw="
  level=error ts=2022-09-01T09:15:20.392761563Z caller=index_set.go:104 table-name=loki_index_19236 user-id=fake msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HV1GHW3TQT7JSC, host id: suqqNGAzw3tjOuHcBTDJrdiT4tcZRkqNxWcUGADBkmf0z7RVnHQ7SceRkwsj4j72jLcMF5WCJQw="
  level=error ts=2022-09-01T09:15:20.392801953Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set fake" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HV1GHW3TQT7JSC, host id: suqqNGAzw3tjOuHcBTDJrdiT4tcZRkqNxWcUGADBkmf0z7RVnHQ7SceRkwsj4j72jLcMF5WCJQw="
  level=error ts=2022-09-01T09:15:20.392844753Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set fake has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HV1GHW3TQT7JSC, host id: suqqNGAzw3tjOuHcBTDJrdiT4tcZRkqNxWcUGADBkmf0z7RVnHQ7SceRkwsj4j72jLcMF5WCJQw="
  ts=2022-09-01T09:15:20.392979855Z caller=spanlogger.go:80 table-name=loki_index_19236 user-id=fake org_id=fake level=info msg="downloaded index set at query time" duration=20.224085ms
  level=error ts=2022-09-01T09:15:20.40154284Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HTVAXBEM383QG5, host id: duB+ak1ZFyPRne4p40pKNwacG9/pEoTH8gmAsjvIPRQzRu/YP01jodjhOO9KtuDZBvJ5QODqDVQ="
  level=error ts=2022-09-01T09:15:20.407471636Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HTC79Y8NT2E2NA, host id: jPDhUGYEoBilBtJ6nAdl4qX2oTR4xsN02o3N/999QNjqjpX0UGhdi/YPj4L7TSkFH7Jhhg6hHFQ="
  level=error ts=2022-09-01T09:15:20.408393446Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HG996CQFCG2S49, host id: enbpiXC3eztVm+vlaaAXD/uafXbM4DvD9vdI0wKQ0xoltgMwEOLgaLGgQty/LNb82NZsbuT6bpw="
  level=error ts=2022-09-01T09:15:20.408438237Z caller=index_set.go:104 table-name=loki_index_19236 msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HG996CQFCG2S49, host id: enbpiXC3eztVm+vlaaAXD/uafXbM4DvD9vdI0wKQ0xoltgMwEOLgaLGgQty/LNb82NZsbuT6bpw="
  level=error ts=2022-09-01T09:15:20.408512058Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set " err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HG996CQFCG2S49, host id: enbpiXC3eztVm+vlaaAXD/uafXbM4DvD9vdI0wKQ0xoltgMwEOLgaLGgQty/LNb82NZsbuT6bpw="
  level=error ts=2022-09-01T09:15:20.408605079Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set  has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HG996CQFCG2S49, host id: enbpiXC3eztVm+vlaaAXD/uafXbM4DvD9vdI0wKQ0xoltgMwEOLgaLGgQty/LNb82NZsbuT6bpw="
  ts=2022-09-01T09:15:20.408999053Z caller=spanlogger.go:80 table-name=loki_index_19236 org_id=fake level=info msg="downloaded index set at query time" duration=15.925327ms
  level=info ts=2022-09-01T09:15:20.409196405Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=26.254162ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.303664ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.41054573Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=29.461677ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.622488ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.411482401Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=31.202887ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=506.285µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.412080987Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=40.312938ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=103.721µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.412252489Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=22.376828ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=508.916µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.412987657Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=38.781021ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.84945ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.413169369Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=31.864994ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=519.995µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.413805906Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=31.948795ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=518.166µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.413804346Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=38.875992ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=420.965µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.414382893Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=37.130033ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.015641ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.414434653Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=28.689759ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.233084ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.414963979Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=26.854299ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=25.871µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.415368774Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=25.270481ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.298714ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.415454565Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=37.839211ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.050822ms subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.4159189Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=27.105521ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=457.085µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.417001942Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=34.549534ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=473.675µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.417257375Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=39.273366ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=739.268µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.417473387Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=40.843873ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=649.897µs subqueries=1 source=logvolhist
  level=error ts=2022-09-01T09:15:20.418739001Z caller=index_set.go:265 table-name=loki_index_19236 user-id=fake msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HQ55D447R74BT0, host id: uFdO38pLMmFd5kbrq6faWQDxeZVYqG/4iS7cuRljCIG6R4i9obYyGBcZPBV7BnRPdjgkTqR3MaQ="
  level=error ts=2022-09-01T09:15:20.418788252Z caller=index_set.go:104 table-name=loki_index_19236 user-id=fake msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HQ55D447R74BT0, host id: uFdO38pLMmFd5kbrq6faWQDxeZVYqG/4iS7cuRljCIG6R4i9obYyGBcZPBV7BnRPdjgkTqR3MaQ="
  level=error ts=2022-09-01T09:15:20.418825552Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set fake" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HQ55D447R74BT0, host id: uFdO38pLMmFd5kbrq6faWQDxeZVYqG/4iS7cuRljCIG6R4i9obYyGBcZPBV7BnRPdjgkTqR3MaQ="
  level=error ts=2022-09-01T09:15:20.418864333Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set fake has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HQ55D447R74BT0, host id: uFdO38pLMmFd5kbrq6faWQDxeZVYqG/4iS7cuRljCIG6R4i9obYyGBcZPBV7BnRPdjgkTqR3MaQ="
  ts=2022-09-01T09:15:20.418917483Z caller=spanlogger.go:80 table-name=loki_index_19236 user-id=fake org_id=fake level=info msg="downloaded index set at query time" duration=20.948793ms
  level=error ts=2022-09-01T09:15:20.426535628Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HMZZ8558NSKFM9, host id: E7x6BY+S+Y9H2dH7b3iTCj7pWUCOBTk0hA0vg6TKUa1KGSFGosZssdAV8La6VunTAaGdpfG/Rao="
  level=error ts=2022-09-01T09:15:20.435966013Z caller=index_set.go:265 table-name=loki_index_19236 msg="sync failed, retrying it" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HMD609KY58X8SY, host id: jHKp7eJb7qXcNZc2bKHceiM0uNXHfRgqPEfY1Cd9Mu9M/gIJBiOSf4GxA7DHqgUkqlmTHl0CZHs="
  level=error ts=2022-09-01T09:15:20.436052283Z caller=index_set.go:104 table-name=loki_index_19236 msg="failed to initialize table loki_index_19236, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HMD609KY58X8SY, host id: jHKp7eJb7qXcNZc2bKHceiM0uNXHfRgqPEfY1Cd9Mu9M/gIJBiOSf4GxA7DHqgUkqlmTHl0CZHs="
  level=error ts=2022-09-01T09:15:20.436094404Z caller=table.go:294 table-name=loki_index_19236 msg="failed to init user index set " err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HMD609KY58X8SY, host id: jHKp7eJb7qXcNZc2bKHceiM0uNXHfRgqPEfY1Cd9Mu9M/gIJBiOSf4GxA7DHqgUkqlmTHl0CZHs="
  level=error ts=2022-09-01T09:15:20.438820964Z caller=table.go:312 table-name=loki_index_19236 org_id=fake msg="index set  has some problem, cleaning it up" err="NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: Y5HMD609KY58X8SY, host id: jHKp7eJb7qXcNZc2bKHceiM0uNXHfRgqPEfY1Cd9Mu9M/gIJBiOSf4GxA7DHqgUkqlmTHl0CZHs="
  ts=2022-09-01T09:15:20.438966376Z caller=spanlogger.go:80 table-name=loki_index_19236 org_id=fake level=info msg="downloaded index set at query time" duration=19.904491ms
  level=info ts=2022-09-01T09:15:20.437814113Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=43.551744ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=38.561µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.437831383Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=42.030607ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=86.311µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.437929484Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=40.593041ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=73.411µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.437929494Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=44.922809ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=380.025µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.43844442Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=43.058108ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=86.811µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.438729443Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=42.805795ms status=200 limit=1000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=19.76µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.438769244Z caller=metrics.go:133 component=querier org_id=fake latency=fast query="sum by(level)(count_over_time({job=\"emqx_connection\"} |= \"\"[1m]))" query_type=metric range_type=range length=29m0s step=1m0s duration=43.911448ms status=200 limit=1000 returned_lines=0 throughput=9.1MB total_bytes=397kB total_entries=12 queue_time=87.421µs subqueries=1 source=logvolhist
  level=info ts=2022-09-01T09:15:20.442458025Z caller=metrics.go:133 component=frontend org_id=fake latency=fast query="sum by (level) (count_over_time({job=\"emqx_connection\"} |= ``[1m]))" query_type=metric range_type=range length=6h0m0s step=1m0s duration=694.334474ms status=200 limit=1000 returned_lines=0 throughput=4.5MB total_bytes=3.1MB total_entries=1 queue_time=143.035682ms subqueries=13 source=logvolhist
  level=info ts=2022-09-01T09:15:20.982070031Z caller=table_manager.go:213 msg="syncing tables"

Sep 01 '22 09:09 aksonlyaks

@kavirajk Are you able to see any issues with the logs above? any suggestions you would like me to try?

Sep 05 '22 10:09 aksonlyaks

@aksonlyaks I am a Loki noobie but maybe you have to change storage_config.boltdb_shipper.shared_store from s3 to aws

Sep 06 '22 12:09 R-Studio

@R-Studio thanks for pointing out the same. However, that too did not helped here..

@kavirajk could you help me with a working config for loki over S3 that I can use to test. Kindly help.

Sep 08 '22 05:09 aksonlyaks

@aksonlyaks this is the config that we use internally and that is working with latest loki version.

storage_config:
    aws:
        region: <region> # e.g: us-east-2
        s3: s3://<s3-url>
        s3forcepathstyle: false
    boltdb_shipper:
        active_index_directory: /data/index
        build_per_tenant_index: true
        cache_location: /data/boltdb-cache
        shared_store: s3
compactor:
    apply_retention_interval: 1h
    compaction_interval: 5m
    retention_delete_worker_count: 500
    retention_enabled: true
    shared_store: s3
    working_directory: /data/compactor

One major difference I see is, you using bucket config instead of direct `s3://`` style URL.

Can you try similar configs on your side and see if that helps?

Sep 08 '22 10:09 kavirajk

@kavirajk I have tried the config file you have shared. Below is the one I've used:

auth_enabled: false

server:
  http_listen_port: 3100

  #distributor#:
  #ring:
  #  kvstore:
  #    store: memberlist
ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
      #heartbeat_timeout: 10m
      #final_sleep: 0s
    #chunk_idle_period: 1h
  #chunk_retain_period: 30s
  #chunk_encoding: snappy
  #max_chunk_age: 2h
  #chunk_block_size: 262144
  #chunk_target_size: 1536000

schema_config:
  configs:
  - from: 2020-05-15
    store: boltdb-shipper
    object_store: aws
    schema: v11
    index:
      prefix: loki_index_
      period: 24h

  - from: 2022-05-16
    store: boltdb-shipper
    object_store: aws
    schema: v12
    index:
      prefix: loki_index_new_
      period: 24h

compactor:
    apply_retention_interval: 1h
    compaction_interval: 5m
    retention_delete_worker_count: 500
    retention_enabled: true
    shared_store: s3
    working_directory: /data/compactor_new

storage_config:
   aws:
       region: ap-south-1 # e.g: us-east-2
       s3: s3://xxxxxxx:[email protected]/tt-xxxxxx
       s3forcepathstyle: false
   boltdb_shipper:
       active_index_directory: /data/index
       build_per_tenant_index: true
       cache_location: /data/boltdb-cache
       shared_store: s3

I'm still not able to see data more than 2.5hrs on Grafana.

While running Loki I can see below errors are occuring time to time.

  level=error ts=2022-09-08T18:50:24.796801701Z caller=flush.go:146 org_id=fake msg="failed to flush user" err="store put chunk: NoSuchBucket: The specified bucket does not exist\n\tstatus code: 404, request id: 3BE9KW4M1E76JVEA, host id: lkwYnvXS25eMiVtKI6LTagr/+jwXUsyG10JUzPzJdyb2xQWguL78TmpnhZwqoRYmyYvasQv2FBE="
  level=error ts=2022-09-08T18:50:24.817808524Z caller=flush.go:146 org_id=fake msg="failed to flush user" err="store put chunk: NoSuchBucket: The specified bucket does not exist\n\tstatus code: 404, request id: 3BE9ZJ64G3W66PR2, host id: yF2bx4qbPYeup1K8sLVyhpxi8nremtnyxI2W8cL29bVBu0iQbux4wRXq0xF3w7A/+K8QRIs8uvg="
  level=error ts=2022-09-08T18:50:24.823670069Z caller=flush.go:146 org_id=fake msg="failed to flush user" err="store put chunk: NoSuchBucket: The specified bucket does not exist\n\tstatus code: 404, request id: 3BEATF6HV5E43P0W, host id: MgMUhSglqZiHAM0zyhjTsPvg8mLqVQaOhxeUxibBbjQWEaU4zDGIPPO1asAidnhYrfG8HvYJnNU="

Sep 08 '22 18:09 aksonlyaks

this link too reports something similar happening with Loki when they used Azure or Cassendra. If I'm having teh similar issue?

Sep 08 '22 19:09 aksonlyaks

@aksonlyaks from the logs you shared, looks like loki couldn't find the right bucket to flush the chunks. If chunks are not flushed then you can query logs only from in-memory chunks (which can last for only around 2hr)

err="store put chunk: NoSuchBucket: The specified bucket does not exist\n\tstatus

I'm almost sure the issue is with either s3 URL or some credentials issues. Can you please double check? The issue you link may not be relevant here AFAIC, mainly because it seems the actual issue is Loki unable to flush chunks because of the same error you added.

Sep 13 '22 08:09 kavirajk

Have you found the reason for this problem? I have the same problem.

Sep 24 '22 05:09 heiying0810

Hi, did you have any solution for this case? I'm having the same problem.

Nov 14 '22 13:11 felipeIntelipost

I'm having the same problem

Nov 21 '22 09:11 tanmx

Did not get any solution on this. I'm using version 2.5.0. I'll try 2.7.0 and see if that'll help.

Dec 02 '22 20:12 aksonlyaks

@aksonlyaks from the logs you shared, looks like loki couldn't find the right bucket to flush the chunks. If chunks are not flushed then you can query logs only from in-memory chunks (which can last for only around 2hr)
err="store put chunk: NoSuchBucket: The specified bucket does not exist\n\tstatus
I'm almost sure the issue is with either s3 URL or some credentials issues. Can you please double check? The issue you link may not be relevant here AFAIC, mainly because it seems the actual issue is Loki unable to flush chunks because of the same error you added.

@kavirajk A proper explanatory example to configure S3 would definitely help many of us. If you could help in creating a tutorial for AWS S3 as a storage with real example?

Dec 02 '22 20:12 aksonlyaks

Hi, i am also facing the same NoSuchKey issue while retrieving old pod logs. @kavirajk Is there a way to check what S3 Key loki look for while getting this error? It will help to figure out if the key is in the bucket or not OR key is there but loki can't get it.

@aksonlyaks for your storage configuration you should use the actual bucket name not the endpoint. I am using like below:

  aws:
    s3: s3://us-east-1/my-grafana-bucket
    s3forcepathstyle: true
    bucketnames: my-grafana-bucket
    region: us-east-1

Dec 29 '22 18:12 evercast-mahesh2021

Hello EveryOne,

I'm able to resolve this issue. Documenting it here for everyone to follow the steps.

Configure the AWS section as shown below. do not use s3forcepathstyle. and mention the details about your bucket as shown below:

storage_config:
  aws:
          
    bucketnames: bucketname
    access_key_id: xxxxxxxxxxxxx
    secret_access_key: xxxxxxxxxxxxxxxxxxxxxxxxxx
    region: ap-south-1
    #s3forcepathstyle: true  <-- This is the main culprit; comment it out
    endpoint:  https://s3.ap-south-1.amazonaws.com

Below I'm showing the path of my bucket

If everything works well, three folder should be created in the root folder of bucket itself. earlier I could see there used to be one more folder bucketname inside bucketname bucket. that was wrong.

Please check if that helps you guys. I have tested this for than a week now. and logs are getting stored in S3. I can search logs for whole now.

@evercast-mahesh2021 @tanmx @kavirajk @R-Studio @heiying0810

Dec 29 '22 20:12 aksonlyaks

Oh great, glad to hear that.

For me, i am still dealing with below error:

level=info ts=2022-12-29T22:16:49.435279835Z caller=metrics.go:143 component=querier org_id=fake latency=fast query="{app=\"janus-1664111251-161-199-67-141-1518\"}" query_type=limited range_type=range length=30m0s start_delta=2281h46m49.435276142s end_delta=2281h16m49.43527633s step=2m0s duration=1.891770734s status=400 limit=10000 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=1.33739987s subqueries=1 cache_chunk_req=1 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_index_req=0 cache_index_hit=0 cache_result_req=0 cache_result_hit=0
level=error ts=2022-12-29T22:16:49.435351022Z caller=batch.go:720 org_id=fake msg="error fetching chunks" err="failed to get s3 object: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: AR0ETNF10NFPQPCF, host id: QtGvULON/11uy7BfZ3KxOV/pk1Eq+yLVgzmEgcI0IDSA4U9HPzTE7SrCubAzTBKF9lfTGtYPy8U="
level=error ts=2022-12-29T22:16:49.4353619Z caller=batch.go:720 org_id=fake msg="error fetching chunks" err="failed to get s3 object: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: AR07B79TRX3F4P8Q, host id: +tonNoTksegznFkky5AGXl6WcWHCpG3pm+livEQCQwo2dvI/gJL5Cs2RkRQle0DzcD24zoyHG5c="

level=info ts=2022-12-29T22:21:03.58125506Z caller=flush.go:168 msg="flushing stream" user=fake fp=9cbbd7ac3ba6f6f3 immediate=false num_chunks=1 labels="{app=\"kube-proxy\", cluster=\"cluster\", container=\"kube-proxy\", filename=\"/var/log/pods/kube-system_kube-proxy-xxxx_18c464f7-7047-422c-bba4-20c44102cd5f/kube-proxy/0.log\", job=\"kube-system/kube-proxy\", namespace=\"kube-system\", node_name=\"ip-x-x-x-x.us-west-2.compute.internal\", pod=\"kube-proxy-xxx\"}"
level=info ts=2022-12-29T22:21:03.58139091Z caller=flush.go:168 msg="flushing stream" user=fake fp=f4a8bd6683945717 immediate=false num_chunks=1 labels="{app=\"kube-proxy\", cluster=\"cluster\", container=\"kube-proxy\", filename=\"/var/log/pods/kube-system_kube-proxy-xxxxx_3d912c6b-3287-474b-ab1a-02525b8e48bc/kube-proxy/0.log\", job=\"kube-system/kube-proxy\", namespace=\"kube-system\", node_name=\"ip-x-x-x-x.us-west-2.compute.internal\", pod=\"kube-proxy-xxxxx\"}"

We were able to get the logs for last couple of days but not the older and than we have applied new loki version to our cluster (loki 2.7.0), which updated the timestamp of the old index and fake files btw. Loki 2.7.0 didn't fix the issue. So, we have upgraded the loki to 2.7.1 and now we are able to get the certain days logs after we deployed 2.7.1 but not all. It's kind of frustrating that we are not sure why we are getting NoSuchKey error, Not able to resolve it or find a root cause of this after trying so many things.

Dec 29 '22 22:12 evercast-mahesh2021

2.7.1 having same issue will share files Using Microservice and noticed there is no compactor svc

Jan 24 '23 13:01 adapasuresh

Finally I got it working

Jan 25 '23 04:01 adapasuresh

below is my configuration file and systemd is getting failed

auth_enabled: false

server:
  http_listen_port: 3100

  #distributor#:
  #ring:
  #  kvstore:
  #    store: memberlist
ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
      #heartbeat_timeout: 10m
      #final_sleep: 0s
    #chunk_idle_period: 1h
  #chunk_retain_period: 30s
  #chunk_encoding: snappy
  #max_chunk_age: 2h
  #chunk_block_size: 262144
  #chunk_target_size: 1536000

schema_config:
  configs:
  - from: 2020-05-15
    store: boltdb-shipper
    object_store: aws
    schema: v11
    index:
      prefix: loki_index_
      period: 24h

  - from: 2022-05-16
    store: boltdb-shipper
    object_store: aws
    schema: v12
    index:
      prefix: loki_index_new_
      period: 24h

compactor:
    apply_retention_interval: 1h
    compaction_interval: 5m
    retention_delete_worker_count: 500
    retention_enabled: true
    shared_store: s3
    working_directory: /data/compactor_new

storage_config:
  aws:
    bucketnames: lokiaccessbucket
    access_key_id: *****
    secret_access_key: ****
    region: ap-south-1
    #s3forcepathstyle: true  <-- This is the main culprit; comment it out
#    endpoint:   https://s3.us-east-1.amazonaws.com`
```

Am getting below error
```
Jan 25 07:21:40 ip-172-31-29-6 loki-linux-amd64[3469]: github.com/grafana/loki/pkg/loki.(*Loki).Run
Jan 25 07:21:40 ip-172-31-29-6 loki-linux-amd64[3469]:         /drone/src/pkg/loki/loki.go:363
Jan 25 07:21:40 ip-172-31-29-6 loki-linux-amd64[3469]: main.main
Jan 25 07:21:40 ip-172-31-29-6 loki-linux-amd64[3469]:         /drone/src/cmd/loki/main.go:105
Jan 25 07:21:40 ip-172-31-29-6 loki-linux-amd64[3469]: runtime.main
Jan 25 07:21:40 ip-172-31-29-6 loki-linux-amd64[3469]:         /usr/local/go/src/runtime/proc.go:250
Jan 25 07:21:40 ip-172-31-29-6 loki-linux-amd64[3469]: runtime.goexit
Jan 25 07:21:40 ip-172-31-29-6 loki-linux-amd64[3469]:         /usr/local/go/src/runtime/asm_amd64.s:1594
Jan 25 07:21:40 ip-172-31-29-6 systemd[1]: loki.service: Main process exited, code=exited, status=1/FAILURE
Jan 25 07:21:40 ip-172-31-29-6 systemd[1]: loki.service: Failed with result 'exit-code'.
```

Jan 25 '23 07:01 shreysway

Please refer the https://github.com/adapasuresh/WhatIsDevOps/tree/main/Observability/yaml

SORRY had to remove the images and files and make repo as private. Will share the yaml files later...

Jan 25 '23 07:01 adapasuresh

Hopefully, this will help others

Jan 25 '23 07:01 adapasuresh

Just wondering if the Loki Index data at S3 and can be store as below: ClusterName:Region:NameSpace:MicroService:Instance:Date-Time.gz ClusterName:Region:NameSpace:MicroService:Instance:Date-Time.log

Help with filters and rulers examples??

Jan 25 '23 08:01 adapasuresh

@adapasuresh How did you get issue resolved?

Mar 09 '23 05:03 earimont-ib

In my case, in lower environment "insecure: true" worked, but in production environment it is more restrictive, where the "PUT" & "DELETE" of "loki_cluster_seed.json" was not happening on http, so had to make "insecure: false",after that communication was on https and it was able to create folders and upload chunks , index files and *.tar.gz / *.zip files to S3.

Mar 09 '23 05:03 adapasuresh

@adapasuresh Ok thanks

Mar 09 '23 14:03 earimont-ib

For anyone using the helm-chart and going crazy like me.. check the bucketnames-variables! It should be the actual name of the bucket, not some sub-bucket. It also doesn't work when you specify the bucket-endpoint, because that seems to conflict with the s3forcepathstyle: false.

This worked on OpenTelekomCloud with OBS:

loki:
  commonConfig:
    replication_factor: 1
  storage:
    type: s3
    s3:
      bucketnames: lokilogs
      access_key_id: ${S3_ACCESS_KEY}
      secret_access_key: ${S3_SECRET_ACCESS_KEY}
      #s3: s3://${S3_ACCESS_KEY}:${S3_SECRET_ACCESS_KEY}@obs.eu-de.otc.t-systems.com/
      endpoint: https://obs.eu-de.otc.t-systems.com
      s3forcepathstyle: false
      sse_encryption: true
      region: eu-de
      # insecure: false
      http_config:
        idle_conn_timeout: 90s
        response_header_timeout: 0s
        insecure_skip_verify: false
  rulerConfig:
    storage:
      type: local
      local:
        directory: /loki/rules
  storage_config:
    hedging:
      at: "250ms"
      max_per_second: 20
      up_to: 3
    boltdb_shipper:
      active_index_directory: /var/loki/index
      cache_location: /var/loki/index_cache
      shared_store: s3

May 04 '23 18:05 strowi

loki:
  commonConfig:
    replication_factor: 1
  storage:
    type: s3
    s3:
      bucketnames: lokilogs

Looking at the heml values file, bucketnames option doesn't exist under s3, it's under storage:

loki:
  storage:
    bucketNames:
      chunks: chunks
      ruler: ruler
      admin: admin
    type: s3
   s3:
...

Jul 28 '23 23:07 batdevis

loki loki copied to clipboard

Loki with S3 not able to push chunks on S3 bucket with compactor error "NoSuchKeys"

loki
loki copied to clipboard