sealer
sealer copied to clipboard
Published
20 hours ago •
sealerio
sealerio
[BUG] sealer run failed
What happen?
I built my clusterImage with latest code which using buildah , and run my cluster failed.
sealer build -f Kubefile -t registry.cn-hangzhou.aliyuncs.com/starcoming/k0s:v1.24.4 .
sealer run registry.cn-hangzhou.aliyuncs.com/starcoming/k0s:v1.24.4 --masters xxxxx --nodes xxxxxx -p xxxxx --cluster-name k8s-test
but got error with
2022-09-13 17:05:03 [ERROR] [root.go:77] sealer-unknown: read .: is a directory
Relevant log output?
Usage:
sealer run [flags]
Examples:
create cluster to your bare metal server, appoint the iplist:
sealer run kubernetes:v1.19.8 --masters 192.168.0.2,192.168.0.3,192.168.0.4 \
--nodes 192.168.0.5,192.168.0.6,192.168.0.7 --passwd xxx
specify server SSH port :
All servers use the same SSH port (default port: 22):
sealer run kubernetes:v1.19.8 --masters 192.168.0.2,192.168.0.3,192.168.0.4 \
--nodes 192.168.0.5,192.168.0.6,192.168.0.7 --port 24 --passwd xxx
Different SSH port numbers exist:
sealer run kubernetes:v1.19.8 --masters 192.168.0.2,192.168.0.3:23,192.168.0.4:24 \
--nodes 192.168.0.5:25,192.168.0.6:25,192.168.0.7:27 --passwd xxx
create a cluster with custom environment variables:
sealer run -e DashBoardPort=8443 mydashboard:latest --masters 192.168.0.2,192.168.0.3,192.168.0.4 \
--nodes 192.168.0.5,192.168.0.6,192.168.0.7 --passwd xxx
Flags:
--cluster-name string set cluster name (default "my-cluster")
--cmd-args strings set args for image cmd instruction
-e, --env strings set custom environment variables
-h, --help help for run
-m, --masters string set count or IPList to masters
-n, --nodes string set count or IPList to nodes
-p, --passwd string set cloud provider or baremetal server password
--pk string set baremetal server private key (default "/root/.ssh/id_rsa")
--pk-passwd string set baremetal server private key password
--port uint16 set the sshd service port number for the server (default port: 22) (default 22)
--provider ALI_CLOUD set infra provider, example ALI_CLOUD, the local server need ignore this
-u, --user string set baremetal server username (default "root")
Global Flags:
--color string set the log color mode, the possible values can be [never always] (default "always")
--config string config file of sealer tool (default is $HOME/.sealer.json)
-d, --debug turn on debug mode
--hide-path hide the log path
--hide-time hide the log time
--log-to-file write log message to disk
-q, --quiet silence the usage when fail
--remote-logger-url string remote logger url, if not empty, will send log to this url
--task-name string task name which will embedded in the remote logger header, only valid when --remote-logger-url is set
2022-09-13 17:05:03 [ERROR] [root.go:77] sealer-unknown: read .: is a directory
What you expected to happen?
run this cluster successfully.
How to reproduce it (as minimally and precisely as possible)?
kubefile is
FROM scratch
COPY rootfs/* .
COPY amd64/* .
COPY imageList manifests
Anything else we need to know?
I used sealer inspect registry.cn-hangzhou.aliyuncs.com/starcoming/k0s:v1.24.4 to show image info.
{
"Type": "buildah 0.0.1",
"FromImage": "registry.cn-hangzhou.aliyuncs.com/starcoming/k0s:v1.24.4",
"FromImageID": "4325fbde85cf7eb9a2be3dc2c40d9c419ce0846d5ce4beb5b32ea433f0f77819",
"FromImageDigest": "sha256:12ff1786efd662e95cca7866b9ca962fffb8dfc56022c472141bb7d525e02f2e",
"Config": "{\"created\":\"2022-09-13T08:55:04.049600631Z\",\"architecture\":\"amd64\",\"os\":\"linux\",\"config\":{\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Labels\":{\"io.buildah.version\":\"1.25.0\",\"io.sealer.version\":\"unknown\"}},\"rootfs\":{\"type\":\"layers\",\"diff_ids\":[\"sha256:65fbe1848c958074b94eab6035064d5a35d632f24090a1690233acd69196282c\",\"sha256:45395bf08b55c88c526c3ca335d137490eaa9dd9179e01182329fbc8120947b5\",\"sha256:149205a2cdf2229401e160c1fa974ec943d0448b1e9f2b622192f49e00342601\",\"sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef\"]},\"history\":[{\"created\":\"2022-09-13T08:54:45.510716939Z\",\"created_by\":\"/bin/sh -c #(nop) COPY multi:828194e5621fb9dce6782a5546612962124068117d103ba0c2b896d2b35b45cf in . \"},{\"created\":\"2022-09-13T08:54:50.711243423Z\",\"created_by\":\"/bin/sh -c #(nop) COPY multi:b92497396053b19ea9b14660a837db1f45a74619f04afc95b0ecb4ea5b9731f3 in . \"},{\"created\":\"2022-09-13T08:55:02.990311465Z\",\"created_by\":\"/bin/sh -c #(nop) COPY file:8a1d3b18a080554c2cb1ef47beee22d0434b029c65f5112d877bc3e99aee520d in manifests \"},{\"created\":\"2022-09-13T08:55:03.092028963Z\",\"created_by\":\"/bin/sh -c #(nop) LABEL \\\"io.sealer.version\\\"=\\\"unknown\\\"\",\"empty_layer\":true},{\"created\":\"2022-09-13T08:55:04.054008308Z\",\"created_by\":\"/bin/sh\",\"comment\":\"FROM registry.cn-hangzhou.aliyuncs.com/starcoming/k0s:v1.24.4\"}]}",
"Manifest": "{\"schemaVersion\":2,\"mediaType\":\"application/vnd.oci.image.manifest.v1+json\",\"config\":{\"mediaType\":\"application/vnd.oci.image.config.v1+json\",\"digest\":\"sha256:4325fbde85cf7eb9a2be3dc2c40d9c419ce0846d5ce4beb5b32ea433f0f77819\",\"size\":1352},\"layers\":[{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:65fbe1848c958074b94eab6035064d5a35d632f24090a1690233acd69196282c\",\"size\":279040},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:45395bf08b55c88c526c3ca335d137490eaa9dd9179e01182329fbc8120947b5\",\"size\":414043136},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:149205a2cdf2229401e160c1fa974ec943d0448b1e9f2b622192f49e00342601\",\"size\":2560},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar+gzip\",\"digest\":\"sha256:bd9ddc54bea929a22b334e73e026d4136e5b73f5cc29942896c72e4ece69b13d\",\"size\":34}],\"annotations\":{\"org.opencontainers.image.base.digest\":\"sha256:06f3591e3005880f1f64b46b375a26c9139f070b1a7ec527d77f21e69ef6c395\",\"org.opencontainers.image.base.name\":\"registry.cn-hangzhou.aliyuncs.com/starcoming/k0s:v1.24.4\",\"sealer.image.extension\":\"{\\\"image_type\\\":\\\"\\\",\\\"cmd_set\\\":null,\\\"arg_set\\\":null}\"}}",
"Container": "",
"ContainerID": "",
"MountPoint": "",
"ProcessLabel": "",
"MountLabel": "",
"ImageAnnotations": {
"org.opencontainers.image.base.digest": "sha256:06f3591e3005880f1f64b46b375a26c9139f070b1a7ec527d77f21e69ef6c395",
"org.opencontainers.image.base.name": "registry.cn-hangzhou.aliyuncs.com/starcoming/k0s:v1.24.4",
"sealer.image.extension": "{\"image_type\":\"\",\"cmd_set\":null,\"arg_set\":null}"
},
"ImageCreatedBy": "",
"OCIv1": {
"created": "2022-09-13T08:55:04.049600631Z",
"architecture": "amd64",
"os": "linux",
"config": {
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Labels": {
"io.buildah.version": "1.25.0",
"io.sealer.version": "unknown"
}
},
"rootfs": {
"type": "layers",
"diff_ids": [
"sha256:65fbe1848c958074b94eab6035064d5a35d632f24090a1690233acd69196282c",
"sha256:45395bf08b55c88c526c3ca335d137490eaa9dd9179e01182329fbc8120947b5",
"sha256:149205a2cdf2229401e160c1fa974ec943d0448b1e9f2b622192f49e00342601",
"sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
]
},
"history": [
{
"created": "2022-09-13T08:54:45.510716939Z",
"created_by": "/bin/sh -c #(nop) COPY multi:828194e5621fb9dce6782a5546612962124068117d103ba0c2b896d2b35b45cf in . "
},
{
"created": "2022-09-13T08:54:50.711243423Z",
"created_by": "/bin/sh -c #(nop) COPY multi:b92497396053b19ea9b14660a837db1f45a74619f04afc95b0ecb4ea5b9731f3 in . "
},
{
"created": "2022-09-13T08:55:02.990311465Z",
"created_by": "/bin/sh -c #(nop) COPY file:8a1d3b18a080554c2cb1ef47beee22d0434b029c65f5112d877bc3e99aee520d in manifests "
},
{
"created": "2022-09-13T08:55:03.092028963Z",
"created_by": "/bin/sh -c #(nop) LABEL \"io.sealer.version\"=\"unknown\"",
"empty_layer": true
},
{
"created": "2022-09-13T08:55:04.054008308Z",
"created_by": "/bin/sh",
"comment": "FROM registry.cn-hangzhou.aliyuncs.com/starcoming/k0s:v1.24.4"
}
]
},
"Docker": {
"created": "2022-09-13T08:55:04.049600631Z",
"container_config": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": null,
"Image": "",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {
"io.buildah.version": "1.25.0",
"io.sealer.version": "unknown"
}
},
"config": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": null,
"Image": "",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {
"io.buildah.version": "1.25.0",
"io.sealer.version": "unknown"
}
},
"architecture": "amd64",
"os": "linux",
"rootfs": {
"type": "layers",
"diff_ids": [
"sha256:65fbe1848c958074b94eab6035064d5a35d632f24090a1690233acd69196282c",
"sha256:45395bf08b55c88c526c3ca335d137490eaa9dd9179e01182329fbc8120947b5",
"sha256:149205a2cdf2229401e160c1fa974ec943d0448b1e9f2b622192f49e00342601",
"sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
]
},
"history": [
{
"created": "2022-09-13T08:54:45.510716939Z",
"created_by": "/bin/sh -c #(nop) COPY multi:828194e5621fb9dce6782a5546612962124068117d103ba0c2b896d2b35b45cf in . "
},
{
"created": "2022-09-13T08:54:50.711243423Z",
"created_by": "/bin/sh -c #(nop) COPY multi:b92497396053b19ea9b14660a837db1f45a74619f04afc95b0ecb4ea5b9731f3 in . "
},
{
"created": "2022-09-13T08:55:02.990311465Z",
"created_by": "/bin/sh -c #(nop) COPY file:8a1d3b18a080554c2cb1ef47beee22d0434b029c65f5112d877bc3e99aee520d in manifests "
},
{
"created": "2022-09-13T08:55:03.092028963Z",
"created_by": "/bin/sh -c #(nop) LABEL \"io.sealer.version\"=\"unknown\"",
"empty_layer": true
},
{
"created": "2022-09-13T08:55:04.054008308Z",
"created_by": "/bin/sh",
"comment": "FROM registry.cn-hangzhou.aliyuncs.com/starcoming/k0s:v1.24.4"
}
]
},
"DefaultMountsFilePath": "",
"Isolation": "oci",
"NamespaceOptions": [
{
"Name": "cgroup",
"Host": true,
"Path": ""
},
{
"Name": "ipc",
"Host": false,
"Path": ""
},
{
"Name": "mount",
"Host": false,
"Path": ""
},
{
"Name": "network",
"Host": false,
"Path": ""
},
{
"Name": "pid",
"Host": false,
"Path": ""
},
{
"Name": "user",
"Host": true,
"Path": ""
},
{
"Name": "uts",
"Host": false,
"Path": ""
}
],
"Capabilities": null,
"ConfigureNetwork": "NetworkDefault",
"CNIPluginPath": "",
"CNIConfigDir": "",
"IDMappingOptions": {
"HostUIDMapping": true,
"HostGIDMapping": true,
"UIDMap": [],
"GIDMap": []
},
"History": [
{
"created": "2022-09-13T08:54:45.510716939Z",
"created_by": "/bin/sh -c #(nop) COPY multi:828194e5621fb9dce6782a5546612962124068117d103ba0c2b896d2b35b45cf in . "
},
{
"created": "2022-09-13T08:54:50.711243423Z",
"created_by": "/bin/sh -c #(nop) COPY multi:b92497396053b19ea9b14660a837db1f45a74619f04afc95b0ecb4ea5b9731f3 in . "
},
{
"created": "2022-09-13T08:55:02.990311465Z",
"created_by": "/bin/sh -c #(nop) COPY file:8a1d3b18a080554c2cb1ef47beee22d0434b029c65f5112d877bc3e99aee520d in manifests "
},
{
"created": "2022-09-13T08:55:03.092028963Z",
"created_by": "/bin/sh -c #(nop) LABEL \"io.sealer.version\"=\"unknown\"",
"empty_layer": true
},
{
"created": "2022-09-13T08:55:04.054008308Z",
"created_by": "/bin/sh",
"comment": "FROM registry.cn-hangzhou.aliyuncs.com/starcoming/k0s:v1.24.4"
}
],
"Devices": null
}
What is the version of Sealer you using?
{"gitVersion":"unknown","gitCommit":"a4872762","buildDate":"2022-09-13 00:04:18","goVersion":"go1.17.13","compiler":"gc","platform":"linux/amd64"}
What is your OS environment?
Ubuntu 20.04.5 LTS
What is the Kernel version?
Linux sealer 5.4.0-125-generic
Other environment you want to tell us?
- Cloud provider or hardware configuration:
- Install tools:
- Others:
I want to know why this happen, If i need to use some external tools? @justadogistaken
@starComingup Is there the rootfs exists in my-cluster? Check if the rootfs is expected.
Usage:
sealer run [flags]
Examples:
create cluster to your bare metal server, appoint the iplist:
sealer run kubernetes:v1.19.8 --masters 192.168.0.2,192.168.0.3,192.168.0.4 \
--nodes 192.168.0.5,192.168.0.6,192.168.0.7 --passwd xxx
specify server SSH port :
All servers use the same SSH port (default port: 22):
sealer run kubernetes:v1.19.8 --masters 192.168.0.2,192.168.0.3,192.168.0.4 \
--nodes 192.168.0.5,192.168.0.6,192.168.0.7 --port 24 --passwd xxx
Different SSH port numbers exist:
sealer run kubernetes:v1.19.8 --masters 192.168.0.2,192.168.0.3:23,192.168.0.4:24 \
--nodes 192.168.0.5:25,192.168.0.6:25,192.168.0.7:27 --passwd xxx
create a cluster with custom environment variables:
sealer run -e DashBoardPort=8443 mydashboard:latest --masters 192.168.0.2,192.168.0.3,192.168.0.4 \
--nodes 192.168.0.5,192.168.0.6,192.168.0.7 --passwd xxx
Flags:
--cluster-name string set cluster name (default "my-cluster")
--cmd-args strings set args for image cmd instruction
-e, --env strings set custom environment variables
-h, --help help for run
-m, --masters string set count or IPList to masters
-n, --nodes string set count or IPList to nodes
-p, --passwd string set cloud provider or baremetal server password
--pk string set baremetal server private key (default "/root/.ssh/id_rsa")
--pk-passwd string set baremetal server private key password
--port uint16 set the sshd service port number for the server (default port: 22) (default 22)
--provider ALI_CLOUD set infra provider, example ALI_CLOUD, the local server need ignore this
-u, --user string set baremetal server username (default "root")
Global Flags:
--color string set the log color mode, the possible values can be [never always] (default "always")
--config string config file of sealer tool (default is $HOME/.sealer.json)
-d, --debug turn on debug mode
--hide-path hide the log path
--hide-time hide the log time
--log-to-file write log message to disk
-q, --quiet silence the usage when fail
--remote-logger-url string remote logger url, if not empty, will send log to this url
--task-name string task name which will embedded in the remote logger header, only valid when --remote-logger-url is set
2022-09-13 17:05:03 [ERROR] [root.go:77] sealer-unknown: read .: is a directory
This is the all log?
@starComingup Is there the rootfs exists in my-cluster? Check if the rootfs is expected.
@justadogistaken where can i found the image layer in my machine? And I make sure about rootfs existed in my build dir.
Usage: sealer run [flags] Examples: create cluster to your bare metal server, appoint the iplist: sealer run kubernetes:v1.19.8 --masters 192.168.0.2,192.168.0.3,192.168.0.4 \ --nodes 192.168.0.5,192.168.0.6,192.168.0.7 --passwd xxx specify server SSH port : All servers use the same SSH port (default port: 22): sealer run kubernetes:v1.19.8 --masters 192.168.0.2,192.168.0.3,192.168.0.4 \ --nodes 192.168.0.5,192.168.0.6,192.168.0.7 --port 24 --passwd xxx Different SSH port numbers exist: sealer run kubernetes:v1.19.8 --masters 192.168.0.2,192.168.0.3:23,192.168.0.4:24 \ --nodes 192.168.0.5:25,192.168.0.6:25,192.168.0.7:27 --passwd xxx create a cluster with custom environment variables: sealer run -e DashBoardPort=8443 mydashboard:latest --masters 192.168.0.2,192.168.0.3,192.168.0.4 \ --nodes 192.168.0.5,192.168.0.6,192.168.0.7 --passwd xxx Flags: --cluster-name string set cluster name (default "my-cluster") --cmd-args strings set args for image cmd instruction -e, --env strings set custom environment variables -h, --help help for run -m, --masters string set count or IPList to masters -n, --nodes string set count or IPList to nodes -p, --passwd string set cloud provider or baremetal server password --pk string set baremetal server private key (default "/root/.ssh/id_rsa") --pk-passwd string set baremetal server private key password --port uint16 set the sshd service port number for the server (default port: 22) (default 22) --provider ALI_CLOUD set infra provider, example ALI_CLOUD, the local server need ignore this -u, --user string set baremetal server username (default "root") Global Flags: --color string set the log color mode, the possible values can be [never always] (default "always") --config string config file of sealer tool (default is $HOME/.sealer.json) -d, --debug turn on debug mode --hide-path hide the log path --hide-time hide the log time --log-to-file write log message to disk -q, --quiet silence the usage when fail --remote-logger-url string remote logger url, if not empty, will send log to this url --task-name string task name which will embedded in the remote logger header, only valid when --remote-logger-url is set 2022-09-13 17:05:03 [ERROR] [root.go:77] sealer-unknown: read .: is a directoryThis is the all log?
All log pasted here 😭
This is my build dir @justadogistaken
.
├── amd64
│ ├── bin
│ │ ├── containerd-rootless-setuptool.sh
│ │ ├── containerd-rootless.sh
│ │ ├── crictl
│ │ ├── k0s
│ │ ├── kubectl
│ │ └── nerdctl
│ ├── cri
│ │ └── containerd.tar.gz
│ ├── images
│ │ └── registry.tar.gz
│ └── Metadata
├── arm64
│ ├── bin
│ │ ├── conntrack
│ │ ├── containerd-rootless-setuptool.sh
│ │ ├── containerd-rootless.sh
│ │ ├── crictl
│ │ └── nerdctl
│ ├── cri
│ └── images
├── download.sh
├── imageList
├── Kubefile
├── rootfs
│ ├── etc
│ │ ├── dump-config.toml
│ │ └── registry.yml
│ ├── lib64
│ │ ├── libseccomp.so.2
│ │ └── libseccomp.so.2.5.1
│ └── scripts
│ ├── clean.sh
│ ├── containerd.sh
│ ├── init-registry.sh
│ └── init.sh
└── version.sh
@starComingup Which branch you are using? Do you push your code?
@justadogistaken https://github.com/sealerio/sealer/pull/1686
+1, I met the same issue. I think there is some bug in the current main branch.
Using clusterfile can successfully exec next step! It seems that the error exists in apply.go https://github.com/sealerio/sealer/blob/e971bc8fa8eb48d1aec4d8e475476a5ec6bf8faa/apply/apply.go#L80 https://github.com/sealerio/sealer/blob/e971bc8fa8eb48d1aec4d8e475476a5ec6bf8faa/apply/apply.go#L111
run cmd cann't set Annotation, so when there is no Annotation, read error occurred. Annotation like:
metadata:
annotations:
ClusterfileName: /root/k0s-test/Clusterfile
It caused GetAnnotaionsByKey read mistake. https://github.com/sealerio/sealer/blob/e971bc8fa8eb48d1aec4d8e475476a5ec6bf8faa/apply/driver/local.go#L67
@kakaZhou719
I feel puzzled about the annotations.
Why would we record the cluster file path into metadata of Cluster?
As for my knowledge, we set it into cluster, and extract it from cluster for further executing(just a temporary value)?
If i'm right about the usage for ClusterfileName , I think we need to remove it from cluster. And do consider what are needed in the clusterfile.
IMO, clusterfile is a file that declares the runtime info. We'd better don't rely on it to store too many information. Extra information could be store in the system(~/.sealer/xxx).