Remaining work on resource limits and requests

[jandubois]

In #1258 we implemented memory and cpu limits and requests for all the bosh-release based containers.

There are a couple of omissions though:

• */log (the operator sidecar for logging)
• boshdns/coredns
• cf-apps-dns/apps-dns
• database/database
• database-seeder/seeder
• database-seeder/output-persist

In addition for Eirini:

• bits/bits
• eirini/opi (request, but no limit)
• eirini-dns-aliases/eirini-dns-aliases
• eirini-events/event-reporter (request, but no limit)
• eirini-metrics/metrics-collector (request, but no limit)
• eirini-routing/* (request, but no limit)
• eirini-staging-reporter/staging-reporter (request, but no limit)
• instance-index-env/injector/instance-index-env-injector
• loggregator-bridge/loggregator-bridge
• persi/persi
• persi-broker/persi-broker
• ssh/ssh
• ssh/ssh-proxy

Furthermore, there is no mechanism to set resource limits or requests on the init containers created by the operator. Maybe these are less important because they are not long-running containers. But if the purpose of limits is to protect the nodes from runaway memory consumption in the case of bugs, then they should be limited too, but maybe just with a generous global limit defined by the operator itself.

Action items:

• [ ] Determine if we need parts are actually required.
• [ ] Create a shared data schema to define native containers in our data files (image, resources, ???).
• [ ] Determine appropriate default limits for all the native containers.
• [ ] Implement resource limits as decided above.

[jandubois]

Added "Validation" label because we need some input from @troytop and @viovanov about the requirements:

• do we need limits for init containers (I'm undecided, but maybe not)
• do we need limits for the log sidecar generated by Quarks (I would say yes)

If the answer to either of these questions is "yes" then we need support for them in Quarks first before we can implement them in kubecf.

[troytop]

I think this issue can be closed without these two requirements if issues are created for kubecf and quarks respectively so that these are addressed at some point.

I agree that the log sidecar limit is the more important one, and that a generous global limit for the operator would be sufficient for handling init containers.