vbe: Try fetching beresp when sending bereq failed

[dridi]

There is no cancellation mechanism in the middle of an HTTP/1 transaction besides closing the session. If a backend closes the connection before the end of the bereq delivery, it may also send a 4XX or 5XX response that could be valuable for the original client.

There are several things to take into consideration: it is the responsibility of the backend implementation to raise the bo->send_failed flag, if the backend replied something before closing we shouldn't trigger a timeout, the connection must not be recycled, and the object must not be cached.

Fixes #2332 Closes #3686

---

We need to address #3813 before proceeding.

Refs #3761

[nigoroll]

changes.rst which I had written before noticing the revert, dropping here for possible reuse:

* Previously, when sending a backend request failed, varnish would
  always generate a 503 response and disregard any response that the
  backend might have sent.

  Now a backend response is used if available, falling back to the
  varnish-generated response if not.  (2332_, 3686_)

.. _2332: https://github.com/varnishcache/varnish-cache/pull/2332
.. _3686: https://github.com/varnishcache/varnish-cache/pull/3686

[nigoroll]

bugwash: we'd like to get this done during this release cycle

[vvucetic]

Hey @nigoroll or @Dridi, would you be able to recommend some workaround until this is fixed/released?

In our case, header validation or rate limiting down the request line can terminate requests earlier and that wrongly returns 503 instead of 400 or 429. Pipe solves the issue, but it’s not an option for us.