[Proposal] A more k8s-based/rootfs-transparent instructions in Kubefile

[justadogistaken]

Type: feature request

Issue Description

We now have five instructions like

• FROM
• CMD
• COPY
• RUN
• ARG.

Following is a sample for building a mysql cluster image. This is fine for deploying mysql workload. But Maybe Deploy/Apply mysql-manifest.yaml --locally(an immature example)

FROM kubernetes:v1.18.3
COPY mysql-manifest.yaml .
CMD kubectl apply -f mysql-manifest.yaml

Describe what feature you want

Maybe we need to consider designing a set of more k8s-based/rootfs-transparent relevant instrutions in Kubefile. Otherwise it's challenge for users to write a correct Kubefile.

There are several components could be abstract first, like cni, csi.

Additional context

something should be awared

• Compatible with OCI builder SDK: we are to build cloud image based on some build library, which may be incompatible with new instructions. (this could be fixed anyway)

@sealerio/sealer-maintainers @sealerio/sealer-reviewers Do you have any suggestion/idea on this feature?

[allencloud]

Maybe we need to consider designing a set of more k8s-based/rootfs-transparent relevant instrutions in Kubefile. Otherwise it's challenge for users to write a correct Kubefile.

@justadogistaken I totally agree with what you said. For what is truth, sealer's Kubefile should provide more concrete definitions for a k8s-based distribution app(s). Usually, we divide a k8s-based distributed application's declaration towards Kubernetes into the following parts:

• computing resource: CPU, MEM and disk capacity. All these are dynamic runtime information. It should be included in Clusterfile rather than Kubefile;
• storage: in k8s world, the storage declaration should be CSI-related parts and PV parts.
• network: in k8s world, the network declaration should be CNI-related parts and egress part, which could be both included in Kubefile:
  • CNI type and CNI config and so on.
  • egress configuration about network exposing things.

All above is application's declaration towards Kubernetes, while app also have declarations towards app itself, including app's content package, binaries, app configurations and so on. All these should be use COPY instruction in Kubefile to move them into ClusterImage.

[kakaZhou719]

How to define a ClusterImage?

My comprehension of the ClusterImage is it should include cluster descriptor and image descriptor.

For cluster descriptor:

1. builtin container image list. it shows which container image will be used in the cluster.
2. CRI details.it shows which cornerstones will the cluster used, and it includes the name, version configs.
3. cluster startup command which could work with the user application content.
4. cluster runtime information, name and version.

For image descriptor:

1. layer ID list: the index of the image layer data. layer type and layer value is less meaning.
2. image Metadata: platform,ClusterImage version, sealer version and so on.

How to build a ClusterImage?

I thought the Kubefile is an extension of the build command line. Generally, user could build ClusterImage from command line with some simple flags without to write Kubefile. Building is a process of user context storage and ClusterImage registration, we could execute sealer build at non-overlay file system, and no need to care about the OS platform. About CNI and CSI, its upstream things which could be built from another base ClusterImage.

Build command line flag.

 --imageName: required
 --from: scratch is the default value.
 --cluster-runtime: required
 --cluster-runtime-version
 --cri: docker is the default value.
 --cri-version
 --platform
 --startup-cmd: cluster start up command
 --startup-arg: cooperated with startup cmd.
 --container-images : additional container image wants to be saved to the ClusterImage.
 --build-context: user application content, yaml, charts and so on. all container images from the build context will be downloaded.

Build command line examples.

sealer build --imageName myapp:v1 --cluster-runtime k8s --cluster-runtime-version v1.19.8 --build-context myapp.yaml

Build from Kubefile

In some case, user want to do some more precise controls, will use Kubefile:

FROM scratch
COPY myapp.yaml manifests
COPY shell.yaml plugins
COPY kubeadm.yml etc
CMD kubectl apply -f manifests/myapp.yaml

sealer build --imageName myapp:v1 --cluster-runtime k8s --cluster-runtime-version v1.19.8 -f Kubefile

I am very willing to discuss this with you all.

[justadogistaken]

Supporting build command line is an easy way for encouraging users to try building. But I feel is's hard to deliver and poor in customization. I think this discussion is for proposing some specific instructions in Kubefile. To make description of Kubefile more k8s-based or more sealer. following is my example.

FROM scratch
CNI calico.yaml --local
PLUGINS shell.yaml
CMD kubectl apply -f manifests/myapp.yaml

[kakaZhou719]

Currently, We have implemented a lot of features through the COPY instruction. below is some common examples using COPY instruction

COPY ImageList manifests
COPY mysql.yaml manifests
COPY mycharts manifests
COPY myshell.yaml plugins
COPY myscript.sh scripts
COPY kubeadm.yml etc
COPY helm bin
COPY offlineDockerImage.tar images

so, I thought, we can use these input to redesign some new instructions.

[justadogistaken]

@kakaZhou719 Do you have any information about high-frequency operations relevant with sealer rootfs?

[kakaZhou719]

@kakaZhou719 Do you have any information about high-frequency operations relevant with sealer rootfs?

user copy build context to manifests is the most high-frequency operation.

COPY ImageList manifests
COPY mysql.yaml manifests
COPY mycharts manifests

[bxy4543]

@kakaZhou719 Do you have any information about high-frequency operations relevant with sealer rootfs?

user copy build context to manifests is the most high-frequency operation.

COPY ImageList manifests
COPY mysql.yaml manifests
COPY mycharts manifests

These operations will read and cache the container image from the ImageList, mysql.yaml, mycharts files to the cluster image, the user is not visible to the image we cached, I don't think we need to do it through COPY, we only need a A list such as ImageList can be used to obtain the cache image, or a Kubefile instruction specifies the cache image list. For directories such as mysql.yaml and mycharts, we can provide a command to obtain the list of images in the directory, and then write to ImageList through the command.

[justadogistaken]

@bxy4543 "ImageList" is generated from "*.yaml / charts" or written by users?

[bxy4543]

@bxy4543 "ImageList" is generated from "*.yaml / charts" or written by users?

by users