[Ruler] The ruler evaluated result is incorrect while running in read-write-backend mode

[jecofang01]

Describe the bug

After we upgrade mimir to v2.5 and deployed in read-write-backend mode, I received JVM memory filling up alert as JVM memory is filling up (> 80%) VALUE = 323.98404389138 LABELS = map: [instance: xxx.xxx.xxx.xx], I checked the value by PromQL, the result is 42.3%.

To Reproduce

Steps to reproduce the behavior:

1. Start Mimir (SHA or version) Mimir, version 2.5.0 (branch: release-2.5, revision: 25533fd)
2. Start Mimir in read-write-backend mode
3. Config alert rule:

  - alert: JvmMemoryFillingUp
    expr: (sum by (instance)(jvm_memory_used_bytes{area="heap"}) / sum by (instance)(jvm_memory_max_bytes{area="heap"})) * 100 > 80
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: JVM memory filling up (instance {{ $labels.instance }})
      description: "JVM memory is filling up (> 80%)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

Expected behavior

The rule should execute correctly.

Environment

• Infrastructure: redhat-7.8, s3, memcached 3 - write, 2 - read, 2 -backend

Additional Context

[pracucci]

I checked the value by PromQL, the result is 42.3%.

Could I see the result of the query run as a range query around the time the alert fired?