Using thanosconvert to push older prometheus blocks to cortex block storage is not returning older data when querying Cortex

[CoreyCaldwell]

Describe the bug I'm testing out Cortex and have used the Thanosconvert tool to push my local storage Prometheus blocks to the block storage backend and verified they exist and have data. What I'm not seeing is any data older than when I first stood up Cortex (i.e. > 7 days available in Prometheus but only 2 days available in Cortex). I was under the impression Cortex would discover these blocks but I must be missing something. I initially ran Thanosconvert after Cortex was running and even tried uninstalling and reinstalling the chart but the data still does not appear to be available.

To Reproduce Steps to reproduce the behavior:

1. Existing Prometheus install with an attached disk for storage
2. Install cortex-helm/cortex (chart v1.3.0, app v1.11.0)

• helm install cortex --namespace cortex cortex-helm/cortex --values cortex-test-values.yml

3. Setup Prometheus remote-write to point to cortex (http://cortex-nginx.cortex.svc.cluster.local/api/v1/push)
4. Run Thanosconvert pod to push tsdb blocks from local disk storage to block storage with same settings as cortex-blocks
5. Check for delta in the oldest data that returns (i.e. sum(machine_cpu_cores))

Expected behavior Running queries on Prometheus and Cortex returns data older than the initial Cortex installation

Environment:

• Kubernetes (AKS)
• Helm
• Cortex (cortex-helm/cortex - chart v1.3.0, app v1.11.0)
• Azure Storage Account
• Prometheus (kube-prometheus-stack - chart 32.2.1, app 0.54.0)

Storage Engine

• [X] Blocks
• [ ] Chunks

Additional Context cortex-test-values.yaml

config:
  alertmanager_storage:
    backend: azure
    azure:
      account_name: cortextest
      account_key: xxx
      container_name: "cortex-alertmanager"
  storage:
    engine: blocks
  blocks_storage:
    backend: azure
    azure:
      account_name: cortextest
      account_key: xxx
      container_name: "cortex-blocks"
    bucket_store:
      sync_dir: "/data"
    tsdb:
      dir: "/data"
  querier:
    # -- Comma separated list of store-gateway addresses in DNS Service Discovery
    # format. This option should be set when using the blocks storage and the
    # store-gateway sharding is disabled (when enabled, the store-gateway instances
    # form a ring and addresses are picked from the ring).
    store_gateway_addresses: dns+cortex-store-gateway-headless.cortex.svc:9095
  ruler_storage:
    backend: azure
    azure:
      account_name: cortextest
      account_key: xxx
      container_name: "cortex-ruler"

Thanosconvert pod config

apiversion: v1
kind: configmap
metadata:
  name: bucket-config
  namespace: monitoring
data:
  bucket-config.yaml: |
    backend: azure
    azure:
      account_name: cortextest
      account_key: xxx
      container_name: "cortex-blocks"
...
apiVersion: v1
kind: Pod
metadata:
  name: thanosconvert
  namespace: monitoring
spec:
  volumes:
  - name: prometheus-kube-prometheus-stack-prometheus-db
    persistentVolumeClaim:
      claimName: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0
  - name: bucket-config
    configMap:
      name: bucket-config
      items: 
      - key: bucket-config.yaml
        path: bucket-config.yaml
  containers:
  - name: thanosconvert
    image: quay.io/cortexproject/thanosconvert:v1.11.0
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: prometheus-kube-prometheus-stack-prometheus-db
      mountPath: /prometheus
      subPath: prometheus-db
      readOnly: true
    - name: bucket-config
      mountPath: /config/bucket-config.yaml
      subPath: bucket-config.yaml
      readOnly: true
    command: ["/bin/sh"]
    args: ["-c", "cd /prometheus; /thanosconvert -config /config/bucket-config.yaml;"]
  restartPolicy: Never
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
            - key: "kubernetes.io/hostname"
              operator: In
              values:
              - {{same vm as prometheus pod for disk attach}}
      

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.