Configure sane default value for grpc_next_upstream_tries

[mattb18]

I'd like ingress-nginx to set a sane default value for grpc_next_upstream_tries. There is currently no value set for this directive, meaning that the default nginx value is used, which in this case is zero. Ideally, this value should also be configurable via an annotation.

I've recently experienced this issue, which resulted in an infinite retry (every 5 seconds due to timeouts) to an upstream that was removed mid request. To make it worse, this turned into an infinite retry with no timeout once the IP was used by another pod which was not listening on the corresponding port, resulting in millions of requests attempts a minute.

As per the ticket, configuring grpc_next_upstream_tries to a sane value (3 in my case) resolves this issue, as the request is tried only 3 times before giving up, rather than retrying infinitely.

A similar change was made for proxy_next_upstream_tries in https://github.com/kubernetes/ingress-nginx/pull/6553, to resolve https://github.com/kubernetes/ingress-nginx/issues/5425.

Would it make sense to also set grpc_next_upstream_tries to 3 or another similar value?

Alternatively, it would be good to set the default to 0, and allow configuring via an annotation (to prevent any potential breaking changes). This way we can avoid needing to configure server snippets annotations.

I'm happy to create a PR if the consensus for this requests is positive.