Remove Support for REST EDS

[ben-taussig-solo]

Gloo Edge Product

Open Source

Gloo Edge Version

1.16.0-rc3

Is your feature request related to a problem? Please describe.

• The default GRPC EDS solution has better performance than REST EDS.
  • gRPC's support for multiplexed requests and bidirectional streaming offer significant performance improvements over REST EDS
• Our documentation does not appropriately clarify the differences between REST EDS and gRPC EDS, potentially leading to confusion among users.
  • The lack of clarity on this issue can lead users to choose REST EDS without understanding the performance/scalability benefits of gRPC EDS
• By removing support for REST EDS, we can encourage our user base to migrate to the more efficient and scalable gRPC EDS, leading to overall better performance and user satisfaction with our software.

Describe the solution you'd like

• Update comments in code around xds.SetEdsOnCluster util/helper to indicate that RestEds is deprecated
  • Potentially update this helper to prevent REST EDS from being configured altogether
• Make GlooOptions.enableRestEds a no-op
  • https://docs.solo.io/gloo-edge/latest/reference/api/github.com/solo-io/gloo/projects/gloo/api/v1/settings.proto.sk/#gloooptions
  • Consider logging a warning if users attempt to configure this setting
  • Note that this PR updated the language around GlooOptions.enableRestEds to clarify that it is not recommended for use https://github.com/solo-io/gloo/pull/9050
• Remove settings.enableRestEds from the helm values
  • Consider logging some sort of warning if a user attempts to configure this value

Describe alternatives you've considered

No response

Additional Context

• There is significant discussion around the role of REST EDS in Gloo Edge in this issue and its related PRs https://github.com/solo-io/gloo/issues/8968
• Upstream envoy moved REST config subscription to an extension (essentially one step before deprecation) with the release of 1.26.0 in April 2023
  • See changelog here: https://www.envoyproxy.io/docs/envoy/latest/version_history/v1.26/v1.26.0