Google Managed Prometheus breaks on `partial_reponse` in queries.

[clearclaw]

Pyrra sets partial_response=false on most queries. Google Managed Prometheus fails all such queries with a 400, thus breaking Pyrra.

Replaying a query from Pyrra (recovered from a pcap) against Google Managed Prometheus:

$ curl -g -X POST 'http://localhost:9090/api/v1/query' -d 'partial_response=false&query=sum+by+%28route%29+%28istio_requests%3Aincrease2w%7Breporter%3D%22source%22%2Cslo%3D%22istio-rc%22%7D%29&time=1707934158'
{
  "error": {
    "code": 400,
    "message": "Invalid JSON payload received. Unknown name \"partial_response\": Cannot bind query parameter. Field 'partial_response' could not be found in request message.",
    "status": "INVALID_ARGUMENT",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.BadRequest",
        "fieldViolations": [
          {
            "description": "Invalid JSON payload received. Unknown name \"partial_response\": Cannot bind query parameter. Field 'partial_response' could not be found in request message."
          }
        ]
      }
    ]
  }
}

Same query without partial_response:

$ curl -g -X POST 'http://localhost:9090/api/v1/query' -d 'query=sum+by+%28route%29+%28istio_requests%3Aincrease2w%7Breporter%3D%22source%22%2Cslo%3D%22istio-rc%22%7D%29&time=1707934158' 
{"status":"success","data":{"resultType":"vector","result":[]}}

[metalmatze]

Thanks for reporting. The partial_response query parameter is a Thanos one. I've added it, since it doesn't break Prometheus (which just ignores it). If this causes trouble we can make it optional and only enable it via a flag for those that run against Thanos.

[clearclaw]

Yep, making it optional would be good. GoogleManagedPrometheus borks on partial_response (and largely removes any need to run Thanos), so I'm currently running a rewriting proxy atop Prometheus to filter out that bit of Pyrra's queries -- which is ehhh, grody.

I've also raised an issue with Google on this, but no response as-yet.