Helm Chart Parity

[udnay]

This is a meta issue to track helm chart parity. Achieving helm chart parity allows us to migrate customers over to the operator and away from the helm charts. We don't need full parity before beginning the migration from helm charts but should be able to match 75-80% of existing helm deployments before starting migrations.

• [x] #712
• We should be configuring a load balancer since the service endpoint doesn't properly distribute connections. Ideally this LB will also do connection pooling
• [ ] | Overrideable Join Flag / start parameters
• We need to be able to do this to solve the migration use case and the multi-site use case
• [ ] | Service Monitor / Prometheus auto-config
• Customers need to be able to monitor their production cockroachdb clusters.
• [ ] | Locality Flag auto-configure
• We need to be able to statically and dynamically set the locality flag
• [x] #695
• In case we want only CRDB running on our nodes
• [x] | Tolerations
• In case we want only CRDB running on a class of nodes and the nodes are reserved for us
• [x] | Affinities (node and pod)
• In case we want to run CRDB along side another service
  • Resolved by #626
• [x] | Anti-Affinities (node and pod)
• In case we want to guarantee we never run CRDB along side a specific service
  • Resolved by #626
• [ ] | Cert Manager Integration
• This will provide production-grade custom CA support, and will help us with multi-site in the future
• [x] #696
• Customers need to be able to annotate clusters in the CR for 3rd party service integration
• [x] | Custom Labels
• Lots of 3rd party integrations inspect service labels in kubernetes to know what to monitor
• [ ] | Emit Logging to external service (datadog / logstash)
• Customers need to integrate logging with their external logging services
• [ ] | Topology Spread Constraints
• This will ensure we don't over-schedule cockroachdb instances
• [ ] | Define a post-install SQL config script (create db/user/roles)
• An empty database doesn't do much good to anyone, and the initial user has to have database root/admin access to do this so the Operator should handle it by default
• [ ] | Other cockroach start options
• We should identify any additionalArgs that we should be setting by default and make sure they have APIs. Alternatively we could rearchitect additionalArgs to a configMap structure that will be easier to set.
• [ ] Start a client pod
• Currently an admin has to start a client pod using a custom script. While this gets better once ingress is configured, we should give the option in the CR to start a client pod alongside the DB.
• [ ] Service Discovery enhancements
• The DB supports a mode of using SRV-A records for service discovery - this may be required for certain multi-site patterns so we should consider making this configurable
• [ ] Advanced Networking Support
• We have alternative networking flags that may be required in some platforms that have multiple pod networks and/or alternative routing rules set up

[goshlanguage]

I spent some time futzing with the init job in tandem with the operator, so I might be able to chip in here, any thoughts on implementation @udnay ?