"WaitForScrapeInstruction" errors

[saulp]

Hi all,

I am seeing several msg="Error WaitForScrapeInstruction messages in the server log, with corresponding msg="Error reading request:" err="invalid method "<!DOCTYPE"" messages on the client log.

As an example, see the below traces:

Server: Aug 08 16:11:32 hosting.metersense.com proxy[13178]: level=info ts=2019-08-08T20:11:32.822Z caller=coordinator.go:123 msg=WaitForScrapeInstruction fqdn=myclient.harrissmartworks.com

Aug 08 16:11:32 hosting.metersense.com proxy[13178]: level=info ts=2019-08-08T20:11:32.822Z caller=proxy.go:129 msg="Error WaitForScrapeInstruction:" err="request is expired"

Client: level=error ts=2019-08-08T20:11:31.805Z caller=client.go:167 msg="Error reading request:" err="invalid method \"<!DOCTYPE\""

(The client and server are synchronized to within a couple seconds of each other, with a 4-hour timezone difference.)

These errors are present every minute -- except on those minutes when the actual scrape request comes in, which is handled successfully. The current scrape interval is 10m.

I also noticed there is no "WaitForScrapeInstruction" method defined in client.go, but am unable to tell if that is the cause for the errors.

I logged #56 recently, and while I cannot discard the possibility that the two issues are connected, I thought it best to keep them separate for now. As mentioned there I am running the latest version of both the client and proxy software.

[SuperQ]

Somehow you have HTML leaking into your connection. Are you using https in front of your proxy server? This feels like there's some kind of transparent firewall in between your client and the proxy, it is intercepting the connection and attempting to re-write the http connection.

[saulp]

Thank you Ben, and nice catch: there is an apache reverse proxy in between (for both HTTPS and client-IP filtering). The scrapes seem to go through just fine, though...