Hey complains when it receives 503s.

[bushidocodes]

When running hey, 503s result in the following being logged:

Unsolicited response received on idle HTTP channel starting with "HTTP/1.1 503 Service Unavailable\r\n\r\n"; err=<nil>

It is unclear if this is expected behavior or an indication that we are not properly sending 503s.

Run ./runtime/experiments/premption/run.sh to recreate

[emil916]

I am still getting these errors on the client side. Not with 503 though, but mostly with 429 (Admissions Controller denying a requrest) and sometimes with 409 (shedding [killing] a work in the global queue to make some cpu util for a guaranteed request).

[bushidocodes]

Does the hey -disable-keepalive option make a difference? This is an HTTP feature we don't support. This is related to the Connection: close header in the following:

#define HTTP_RESPONSE_429_TOO_MANY_REQUESTS  \
	"HTTP/1.1 429 Too Many Requests\r\n" \
	"Server: SLEdge\r\n"                 \
	"Connection: close\r\n"              \
	"\r\n"

I'd also be curious if this happens to be an HTTP HEAD request.

A comment here suggests that it might be because we write a response before reading the entire HTTP request:

"Basically it means that the upstream server either responded with data before the HTTP request was finished, or it continued to respond with data after it was meant to stop (already wrote the full content length's worth of bytes?)"

[emil916]

Does the hey -disable-keepalive option make a difference?

Nope. I use hey with that option by default.

I'd also be curious if this happens to be an HTTP HEAD request.

I am curious why it happens only a few times, not for all the requests.

"Basically it means that the upstream server either responded with data before the HTTP request was finished, or it continued to respond with data after it was meant to stop (already wrote the full content length's worth of bytes?)"

Yeah I saw that. Not getting how it is happening in Sledge though :/