Close idle connections gracefully in GMT

[pwilloughby]

Background

What is the problem/pain point?

Idle connections are currently being terminated by our load balancer with a hard disconnect. FileZilla PRO doesn't handle the hard disconnect well but works fine if the connection is terminated gracefully.

Status: Resolving address of data.gateway.storjshare.io
Error: GnuTLS error -110 in gnutls_record_recv: The TLS connection was non-properly terminated.
Status: Server did not properly shut down TLS connectionError: Could not read from socket: ECONNABORTED - Connection aborted

There are also some traces available here

Requirements

Close idle connections gracefully. One way to do this would be to have gateway-mt close idle connections before the load balancer does.

Acceptance Criteria

FileZilla Pro no longer gets errors on idle timeout.

[pwilloughby]

The idle timeouts on the load balancer were increased here https://github.com/storj/infra/pull/4738 for an unrelated issue.

[nergdron]

if we make the load balancer have a longer idle timeout than gateway, then would the graceful termination behaviour be enforced, or is there an edge condition I'm not thinking of?

[halkyon]

we did increase the timeouts on haproxy with https://github.com/storj/infra/pull/4738, but then it got reverted due to that authservice client bug where it wouldn't re-use connections, but that's fixed now since v1.61.0. I think we can increase the timeouts again now! @nergdron @pwilloughby

[pwilloughby]

then would the graceful termination behaviour be enforced,

I'm not sure gateway has an idle timeout currently, that probably needs a code or config change.

[storj-gerrit[bot]]

Change {cmd,pkg}: support setting IdleTimeout mentions this issue.

[storj-gerrit[bot]]

Change salt/states/haproxy: increase idle timeout mentions this issue.