[builder] add kaniko in k8s builder support

[cppforlife]

[cppforlife]

@kvaps says that kaniko v0.15 is less buggy. it also saves digest into a file after building.

[joaopapereira]

@cppforlife Do we still want to add kaniko as a builder for kbld?

[cppforlife]

Do we still want to add kaniko as a builder for kbld?

i think it would be good to support it as a builder. what i imagine might be a custom kbld-kaniko binary that kbld calls out to. kbld-kaniko could focus on interacting with k8s to create kaniko pod. it would also have to figure out how to get local source code available to kaniko (hopefully thats something that could be borrowed from other tools).

this would help support folks that are trying to run non-privileged/non-docker-daemon builds for Dockerfiles on top of k8s.

[danielhelfand]

this would help support folks that are trying to run non-privileged/non-docker-daemon builds for Dockerfiles on top of k8s.

@cppforlife Do you mind explaining the use case here a bit further?

Another question is why this should needs a custom binary and why kbld and kaniko can't be used independently to achieve the same goal?

[cppforlife]

Do you mind explaining the use case here a bit further?

folks want to be able to build their Docker image from Dockerfiles on k8s cluster (not on their local machine) without having to expose privileged socket to underling Docker daemon running on the node (also lots of folks migrated to containerd which means that it's not even possible).

Another question is why this should needs a custom binary and why kbld and kaniko can't be used independently to achieve the same goal?

kaniko does not deal with orchestrating kubernetes pods. kbld should not important special libraries (k8s libraries) for one particular builder since it's a CLI that orchestrates builders (not a builder itself). hence i was thinking about having a small CLI wrapper on top of kaniko that can be used by kbld. this wrapper would handle Pod creation/deletion in k8s for kaniko to run. it would also handle transfer of source code into kaniko runtime.

[joaopapereira]

kaniko does not deal with orchestrating kubernetes pods. kbld should not important special libraries (k8s libraries) for one particular builder since it's a CLI that orchestrates builders (not a builder itself). hence i was thinking about having a small CLI wrapper on top of kaniko that can be used by kbld. this wrapper would handle Pod creation/deletion in k8s for kaniko to run. it would also handle transfer of source code into kaniko runtime.

@cppforlife given your description do you think this should be part of kbld repository/work? or it should be created separately and when finished integrate it with kbld?

[cppforlife]

should be created separately and when finished integrate it with kbld?

this is probably easiest and nicest path.

[joaopapereira]

Sounds good. Accepting this but it will be blocked until we develop this tool to do the integration

[kvaps]

It would be nice to have kaniko support for kbld. I already fully switched to building images directly on Kubernetes.

To achieve this I use my own written wrapper for kaniko. Writing tiny as possible plugin for kubectl was preferred task to me so I used standard shell scripting. Thus, you can easily inspect it, check it out:

https://github.com/kvaps/kubectl-build

[cppforlife]

@kvaps that's pretty cool. would you be interested in making a PR to kbld to support your tool as a builder?

[kvaps]

Right now I'm not sure about this. I'm pretty new in kbld. I like the concept but my use case is slight different. Mainly I use qbec.io to render jsonnet and deploy an applications out of it.

Since the last versions I can use external sources, for example I can inject commit hash into annotations and so on. Now I'm looking for some tool which would allow doing me the following:

qbec show default | kbld -f-

I don't need to resolve and substitute the images into yaml, I need to only build and push image from yaml manifests. Of course I can add some dummy deployments to the yaml. But isn't there another way?

Thank you!

[cppforlife]

I don't need to resolve and substitute the images into yaml, I need to only build and push image from yaml manifests.

sounds like you just want kbld to build them but not use the built references in the yaml output. you can throw away yaml output kbld produces, right? (im guessing you are using tags, and do not want to use digests?)

Of course I can add some dummy deployments to the yaml

kbld doesnt know what kubernetes is, so no need to add dummy deployments. just need to have some image references. e.g. this would be valid kbld input (it just have to be some yaml)

image: blah
---
apiVersion: kbld.k14s.io/v1alpha1
kind: Config
sources:
- image: blah
  path: src/