Examine scope of build controller and expectations about namespace where they run

[pmorie]

This issue is to do some initial exploration about enabling the following use-case in build:

• As a user of build, I want to deploy the build controller into my namespace and limit the scope of that controller to the build resources in my namespace, so that i can adopt newer versions of build on my own schedule and limit the scope of outages in the controller to my namespace

I have written up an issue for this in serving: https://github.com/knative/serving/issues/2107

However, after some thought it became clear that build is a better proving ground to explore what this would take, since it is a much smaller, self-contained piece.

Expected Behavior

• I can run knative build in my namespace, scoped only to my namespace

Actual Behavior

• no way to do this today

Additional Info

One thing to note is that the CRD definitions themselves are inherently cluster-scoped in the sense that there is only a single definition of a CRD per cluster that is constant across all namespaces when the custom resource is namespace scoped. We should accept this as a constant, since it is not likely to change (ie, it touches the lack of ACL-filtered list/watch and some fundamental realities of auth). This means that the API must be forward and backward compatible over the long term for people to be able to safely use different controller versions with a single API.

I will do some analysis on build and document findings here, thoughts and opinions welcome on this subject.

[shashwathi]

/assign @jcrossley3

[pmorie]

After some initial R&D, @jcrossley3 and I (mostly Jim) have uncovered a few entanglements that we should sort through:

• We will need to determine what the best way to represent the differences in resources that will be present between the current (cluster-scoped) mode of operation and a mode where the scope of the controllers is one or a set of namespaces
• What is the best development experience we can provide in the presence of different operational modes?
• What is the best installation experience we can provide?
• Auth:
  • What differences are there in terms of RBAC permissions that we may need to account for?
  • Is it sufficient to make a ClusterRoleBinding to a ClusterRole vs a RoleBinding to a ClusterRole or are there larger differences
  • Should we re-examine the use of the kube-system namespace re: certificates?
• Testing:
  • We will now have to test each of the different modes of operation - what is the best way to do that?
  • Should we have a test that is exercised in an arbitrary namespace?
• Releases:
  • What form do release manifests take on?
    • Is there a manifest per mode of operation?
    • Are the namespaces specified in those manifests?
    • Should we consider formats that allow parameterization like helm charts or ______?
• Documentation: what's the best way to communicate to users about:
  • Why are there different ways to run the controllers?
  • How should you choose?
  • What effect does the choice you make have in how you configure/install/use knative?

[pmorie]

Also commented about ns-arity of controllers: https://github.com/knative/serving/issues/2107#issuecomment-430698592

TLDR: There seem to be a lot of questions raised by just looking at scoping a controller to a single namespace that make it seem to me that a single-ns is plenty to think about initially. There are additional complexities for the set-of-ns use-case that will be easier to address once we've figured out the baseline things raised by the single-ns case. My $0.02

[dprotaso]

What form do release manifests take on?

Another tool in addition to helm that might be worth investigating is kustomize https://github.com/knative/serving/issues/1114