Replace Clouddriver's use of kubectl with the Kubernetes Java client library

[maggieneterval]

Issue Summary:

Replace Clouddriver's use of kubectl with the Kubernetes Java client library.

Cloud Provider(s):

Kubernetes V2

Feature Area:

Clouddriver

Description:

When the Kubernetes V2 provider was initially implemented, the Kubernetes Java client lacked feature parity with kubectl. However, that is no longer the case. We should replace the KubectlJobExecutor, which is responsible for composing kubectl commands that are then run in subprocesses, with analogous calls to the client library.

[lukmy]

It seems that supported kubernetes version of Kubernetes Java Client is even lower than kubectl embedded in clouddriver (currently 1.16.0). Could we have an option to specify using kubectl or java client?

[ezimanyi]

@lukmy : I think the table from their readme that you posted above is actually out of date; they haven't updated it with the recent release of 8.0.0 which I believe supports 1.16.

Overall the compatibility guarantee of the library seems stronger than kubectl:

• kubectl only supports a skew of one minor version from the cluster
• the client library has a much better compatibility guarantee---objects in common work for the entire matrix above; the +/- just indicate that there are some objects on the server the client won't know about and some objects in the client that won't work with an older server

One side-goal of this migration is to have the Spinnaker Kubernetes provider come up with a reasonable policy for supporting versions of Kubernetes clusters. Currently we don't have such a policy (and given that we have kubectl 1.16 such a policy would mean we only support Kubernetes 1.15 and higher which is likely not reflective of users' clusters). I think that by using the library we can stay up to date with recent Kubernetes versions and also keep reasonable backwards-compatibility.

To the question of whether we'll keep a flag for this, I don't anticipate doing that. I'd like to have a unified experience for users and avoid more complexity in configuration (and in debugging). During the rollout there will almost certainly be some kind of toggle, but once we've confirmed the new approach works, we'll remove the toggle so that everyone is running the same code---which avoids needing to maintain/test two paths and simplifies the configuration experience for users. What I think is more likely is if there is some way we use kubectl that doesn't translate well to the library, I could imagine us keeping it around for those operations, but that would be something that happens for all users rather than controlled by a flag.

[lukmy]

@ezimanyi It seems reasonable for me. I just go straight to my problem.

I just create another issue #5715 for it. My concern is that, for my situation, our kubernetes version is higher than kubectl embedded in clouddriver. If using kubctl, I might patch a newer kubectl into clouddriver image to solve this problem. But if using kubernetes java client, I couldn't find another way to support 1.17.x in some situation except waiting client to upgrade.

[ezimanyi]

Sounds good, thanks for following up. I think that the client library will actually allow us to be quicker to upgrade to support new Kubernetes versions, though there will always be some lag between when a new version of Kubernetes is released -> the new library is published -> we pull that library into Spinnaker.

[davidxia]

Cool to see this effort. Just curious, are you planning on using server side apply which is beta in 1.16 or implementing the merging logic yourself in code?

[maggieneterval]

@davidxia While we haven't begun to technically scope this project yet, I think it's safe to say that we will not be implementing any merging logic ourselves! Will definitely keep this issue updated with a more detailed plan when we actually begin this work.

[spinnakerbot]

This issue hasn't been updated in 45 days, so we are tagging it as 'stale'. If you want to remove this label, comment:

@spinnakerbot remove-label stale

[ezimanyi]

Unassigning now that this is in our active tracking project.

[maggieneterval]

Assigning this to @mattsanta as he will be tackling this! (Matt, you may need to comment on this issue before it lets me assign you.)

[mattsanta]

Commenting! Currently working on design doc for this :)

[dsexton]

Is progress for this work being tracked anywhere other than here?