kuma RateLimitedBackOff set to 8 seconds retries after ~11 seconds

What happened?

We're setting "x-retry-after" to 8 seconds but the delay is longer.

Also, request time was quite small so not sure what happened there:

Jan 16 '23 11:01 slonka

ProxyTemplate to add response header:

apiVersion: kuma.io/v1alpha1
kind: ProxyTemplate
mesh: default
metadata:
  name: custom-template-1
spec:
  selectors:
    - match:
        kuma.io/service: backend_kuma-demo_svc_3001
  conf:
    imports:
      - default-proxy
    modifications:
      - virtualHost:
          operation: remove
          match:
            name: "backend_kuma-demo_svc_3001"
      - virtualHost:
          operation: add
          match:
            routeConfigurationName: "inbound:backend_kuma-demo_svc_3001"
          value: |
            name: "backend_kuma-demo_svc_3001"
            domains:
            - "*"
            routes:
            - match:
                prefix: /
              route:
                cluster: "localhost:3001"
                timeout: "0s"
              responseHeadersToAdd:
                - header:
                    key: "x-retry-after-header"
                    value: "8"

Jan 16 '23 11:01 lobkovilya

Same thing happens on k3d locally (1st retry triggered after ~11 seconds):

Fault injection:

apiVersion: kuma.io/v1alpha1
kind: FaultInjection
mesh: default
metadata:
  name: fi1
spec:
  sources:
    - match:
        kuma.io/service: frontend_kuma-demo_svc_8080
        kuma.io/protocol: http
  destinations:
    - match:
        kuma.io/protocol: http
        kuma.io/service: backend_kuma-demo_svc_3001
  conf:        
    abort:
      httpStatus: 500
      percentage: 80

MeshRetry:

apiVersion: kuma.io/v1alpha1
kind: MeshRetry
metadata:
  name: mesh-retry
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Mesh
  to:
    - targetRef:
        kind: Mesh
      default:
        http: 
          numRetries: 5
          retryOn:
            - 5xx
          rateLimitedBackOff:
            resetHeaders:
              - name: "x-retry-after-header"
                format: "Seconds"

MeshTrace:

apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: default
  namespace: kuma-system
  labels:
    kuma.io/mesh: default # optional, defaults to `default` if unset
spec:
  targetRef:
    kind: Mesh
  default:
    backends:
      - zipkin:
          url: http://jaeger-collector.mesh-observability:9411/api/v2/spans

Default request timeout (timeout-all-default) changed to:

      requestTimeout: 120s

Jan 23 '23 08:01 slonka