Implement ClientSettingsPolicy

[kate-osborn]

trafficstars

Proposed changes

Write a clear and concise description that helps reviewers understand the purpose and impact of your changes. Use the following format:

Problems:

• As a Cluster Operator, I want to set defaults for client settings that will work for most applications so that most Application Developers will not have to tweak these settings.
• As an Application Developer, I want to be able to configure client settings for my application based on its behavior or requirements.
• As an Application Developer, I want to override the defaults for client settings set by the Cluster Operator because the defaults do not satisfy my application's requirements or behavior.

Solution: Implement ClientSettingsPolicy API.

• Cluster operators can create a ClientSettingsPolicy for a Gateway to set defaults for client settings that apply to all routes attached to that Gateway.
• App devs can create ClientSettingsPolicies for their routes and specify client settings that override the defaults set by the cluster operator.

Testing: Manually tested the following cases:

• attaching to Gateway
• attaching to HTTPRoute
• attaching to GRPCRoute
• merging and inheritance behavior
• conflict handling

Closes #1792 #1760

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• [x] I have read the CONTRIBUTING doc
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] I have checked that all unit tests pass after adding my changes
• [x] I have updated necessary documentation
• [x] I have rebased my branch onto main
• [x] I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

Release notes

If this PR introduces a change that affects users and needs to be mentioned in the release notes, please add a brief note that summarizes the change.

Introduces ClientSettingsPolicy CRD. This CRD allows users to configure the behavior of the connection between the client and NGINX.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 95.46248% with 26 lines in your changes are missing coverage. Please review.

Project coverage is 87.66%. Comparing base (9212c4b) to head (2842083).

Files	Patch %	Lines
internal/mode/static/manager.go	4.76%	20 Missing :warning:
internal/framework/helpers/helpers.go	85.71%	1 Missing and 1 partial :warning:
internal/mode/static/state/change_processor.go	92.30%	1 Missing and 1 partial :warning:
internal/mode/static/state/store.go	92.30%	1 Missing and 1 partial :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1940      +/-   ##
==========================================
+ Coverage   87.04%   87.66%   +0.62%     
==========================================
  Files          89       93       +4     
  Lines        6096     6557     +461     
  Branches       50       50              
==========================================
+ Hits         5306     5748     +442     
- Misses        737      753      +16     
- Partials       53       56       +3     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[sjberman]

I wonder if we want a tech-debt issue to rework BackendTLSPolicy at some point to use this new framework.

[sjberman]

Is the PolicyAncestorStatus supposed to apply to Inherited policies? I thought I read that it only applies to direct policies.

[kate-osborn]

Is the PolicyAncestorStatus supposed to apply to Inherited policies? I thought I read that it only applies to direct policies.

Yeah, currently, it is only required for direct policies, but there's been some discussion to include it on inherited policies. Plus, the spec is moving towards collapsing direct and inherited policies into one policy type with a strategy field.

@pleshakov talked me into including PolicyAncestorStatus on the ClientSettingsPolicy here: https://github.com/nginxinc/nginx-gateway-fabric/pull/1793#discussion_r1551878502

[kate-osborn]

I wonder if we want a tech-debt issue to rework BackendTLSPolicy at some point to use this new framework.

This might be tricky. We'd have to create a wrapper class to pass it around like an NGF Policy. Then, we'd have to figure out how to genericize the ConfigMap resolving part. I'm sure it can be done, but it's not an easy lift.