Remove default tls secret for NGINX Ingress deployments

[jasonwilliams14]

Proposed changes

This PR is to remove the default TLS secret and key that are set for the default_server of NGINX Ingress controller. These files are no longer needed and the recommended approach is to provide users own certificate/key for default_server in the form of a kubernetes TLS secret, if desired.

If no kubernetes secret is defined for the default_server, the NGINX directive ssl_reject_handshake will be enabled for default_server.

This is a much better approach to properly configure NGINX Ingress controller.

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• [ X] I have read the CONTRIBUTING doc
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] I have checked that all unit tests pass after adding my changes
• [ X] I have updated necessary documentation
• [ X] I have rebased my branch onto main
• [ X] I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

[codecov-commenter]

Codecov Report

Merging #2901 (2e4ecbd) into main (2f01158) will decrease coverage by 0.74%. The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #2901      +/-   ##
==========================================
- Coverage   53.03%   52.28%   -0.75%     
==========================================
  Files          58       58              
  Lines       15645    15965     +320     
==========================================
+ Hits         8298     8348      +50     
- Misses       7070     7337     +267     
- Partials      277      280       +3     

Impacted Files	Coverage Δ
cmd/nginx-ingress/flags.go	34.42% <ø> (+8.50%)	:arrow_up:
internal/k8s/status.go	33.27% <0.00%> (-0.97%)	:arrow_down:
internal/certmanager/cm_controller.go	18.04% <0.00%> (-0.76%)	:arrow_down:
internal/k8s/controller.go	11.18% <0.00%> (-0.53%)	:arrow_down:
internal/k8s/configuration.go	95.39% <0.00%> (-0.37%)	:arrow_down:
internal/configs/configmaps.go	18.20% <0.00%> (-0.02%)	:arrow_down:
internal/k8s/leader.go	0.00% <0.00%> (ø)
cmd/nginx-ingress/main.go	0.00% <0.00%> (ø)
internal/configs/config_params.go	76.74% <0.00%> (ø)
internal/externaldns/controller.go	0.00% <0.00%> (ø)
... and 4 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[lucacome]

@vepatel we should replace the check for the generated secret with a check for ssl_reject_handshake but it might not be very straightforward

[vepatel]

@lucacome this is part of IC setup not tests so yeah could be a bit tricky

[pleshakov]

the PR introduces a breaking change -- controller.defaultTLS.cert and controller.defaultTLS.key are removed completely. Users might use them to configure the cert and key with non-default values. Is it necessary to remove those parameters?

[jasonwilliams14]

the PR introduces a breaking change -- controller.defaultTLS.cert and controller.defaultTLS.key are removed completely. Users might use them to configure the cert and key with non-default values. Is it necessary to remove those parameters?

@pleshakov Yea, we definitely do not want to introduce any changes that could causes issues for customers. Perhaps the better option is to keep those values if they are currently being used? I will do some more investigating.

[brianehlert]

To speak to the concern. We can't break. If a customer has them defined, we should use them. If not, it should follow the new behavior. This is about allowing the default tls to not be defined and also not filling that void with some common secret. There are folks that will still want the failback behavior, because that is what they know and expect.