Replace minio blobstore with an OCI compliant registry

[Syerram]

Is your feature request related to a problem? Please describe.

CAPI uses the internal minio blobstore to store app source code. CAPI intends to replace the internal minio blobstore with an external OCI compliant registry. We recommend CAPI to expose the registry configuration to downstream cf-for-k8s, so cf-for-k8s can provide an interface for default and advanced config options.

Describe the solution you'd like

Usecase

Default out-of-the-box install

Given that I want to install cf-for-k8s as is When I configure the app registry in cf-for-k8s Then I expect no additional configuration for the app source code registry   And I expect cf-push to push the source code to the app registry   And the built image is also pushed to the same app-registry   And I am able to curl the app

Usecase

Platform teams want to use an external registry that is different than the app registry. Platform teams may want an highly available app registry for their apps since it can impact app downtime. However, this comes at a high cost. For source code app registry, Platform teams don't have the same stringent requirements as the app-registry. They can use a low-cost non-HA registry since the tradeoff is longer cf-push times in case of data loss and cf-push failures in case of registry downtime.

Given that I want to install cf-for-k8s with an external source code registry When I configure the app-registry and source code repository separately Then I expect cf-for-k8s to install successfully   And I expect cf-push to push source to the source code repository   And the built image is pushed to the app-registry   And I am able to curl the app

Additional context

CAPI Epic => https://www.pivotaltracker.com/epic/show/4596046

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/174578939

The labels on this github issue will be updated when the story is started.

[cwlbraa]

for clarity, in the CF API "source code" is usually referred to as "packages," and in the capi backlog we talk about this as a "package registry"

[tcdowney]

@Syerram @paulcwarren

Are you all planning on keeping the Minio (or blobstore configuration in general) around for v1.0.0? Even though an OCI registry will be used for ultimately storing the packages, there's a lot of baked in assumptions in the cloud_controller_ng codebase around blobstores being configured and we're planning on doing the work to make that config optional after the 1.0.0 milestone.

cc @cwlbraa @pivotal-mikegresham @piyalibanerjee