camel-k
camel-k copied to clipboard
apache
Unable to run an integration on Minikube
I created a Minikube cluster with the registry and olm addons and then follow the steps outlined below.
podman machine init --cpus 2 -m 4096
podman machine start
minikube start --driver=podman --container-runtime=containerd --addons=registry,dashboard,olm
kubectl config set-context --current --namespace=$NS
kamel install --global
kamel init Example.java # Just the default, no additions
kubectl describe integration example
No events get added to the integration and no Camel pods are added to the namespace. Versions of the clients are listed below and this is all on MacOS 12.4.
$ kamel version
Camel K Client 1.9.2
$ minikube version
minikube version: v1.26.0
commit: f4b412861bb746be73053c9f6d2895f12cf78565
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.17-dispatcher", GitCommit:"a39a896b5018d0c800124a36757433c660fd0880", GitTreeState:"clean", BuildDate:"2021-01-28T22:06:27Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:18:48Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}
The registry created by Minikube works, as I can push custom images to it as described https://minikube.sigs.k8s.io/docs/handbook/registry/#docker-on-macos and create pods from the images I create.
Is there a step I'm missing? Any other steps I can take to debug what is going on?
Hi, there is one more important step left to run your integration. It is:
kamel run Example.java
Hi, there is one more important step left to run your integration. It is:
kamel run Example.java
My bad, that's a bit of a documentation fail. Yes, I ran kamel run Example.java before describe integrations. The describe command did return the object (which wouldn't have been the case if run had been omitted entirely) but never one that progressed to actually running.
@michaeljmcd Ah ok, then if you chose kamel install --global method, run the following command in your namespace before creating an integration:
kamel install --skip-operator-setup -n $NS
So, starting fresh, I get CRD installation errors.
$ export NS=example
$ minikube delete
$ minikube start --driver=podman --container-runtime=containerd --addons=registry,dashboard,olm
$ kubectl create namespace $NS
$ kubectl config set-context --current --namespace=$NS
$ kamel install --global
OLM is available in the cluster
Error: cannot check CRD installation after 90 seconds
# Re-run due to the message above.
$ kamel install --global
OLM is available in the cluster
OLM resources are already available: skipping installation
Error: cannot check CRD installation after 90 seconds
$ kamel install --skip-operator-setup -n $NS
Error: cannot check CRD installation after 90 seconds
$ kubectl get pods -n $NS
No resources found in example namespace.
I'll try it again here in a bit.
OK, I think my previous error was an error installing OLM which I worked around by manually pulling the image first.
With the following sequence:
$ export NS=example
$ minikube delete
$ minikube start --driver=podman --container-runtime=containerd --addons=registry,dashboard
$ podman pull quay.io/operator-framework/olm # I got image pull errors without manually pulilng the image.
$ operator-sdk olm install
$ kubectl create namespace $NS
$ kamel install --global
$ kubectl config set-context --current --namespace=$NS
$ kamel install --skip-operator-setup -n $NS
$ kamel init Example.java
$ kamel run -n $NS Example.java
I get the following events:
$ k describe integration example
[...]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal IntegrationConditionChanged 2m59s camel-k-integration-controller Condition "IntegrationPlatformAvailable" is "False" for Integration example: example/camel-k
Normal IntegrationPhaseUpdated 2m59s camel-k-integration-controller Integration "example" in phase "Waiting For Platform"
Warning IntegrationError 2m59s (x3 over 2m59s) camel-k-integration-controller Cannot reconcile Integration example: error during trait customization: unable to find catalog matching version requirement: runtime=, provider=quarkus
Normal IntegrationConditionChanged 2m camel-k-integration-controller Condition "IntegrationPlatformAvailable" is "True" for Integration example: example/camel-k
Normal IntegrationPhaseUpdated 2m camel-k-integration-controller Integration "example" in phase "Initialization"
Normal IntegrationPhaseUpdated 2m camel-k-integration-controller Integration "example" in phase "Building Kit"
Normal IntegrationConditionChanged 2m camel-k-integration-controller Condition "IntegrationKitAvailable" is "False" for Integration example: creating a new integration kit
Normal ReasonRelatedObjectChanged 119s camel-k-integration-kit-controller Integration Kit "kit-cb7ek9ssarkc73860meg", created by Integration "example", changed phase to "Build Submitted"
Normal ReasonRelatedObjectChanged 119s camel-k-build-controller Build "kit-cb7ek9ssarkc73860meg", created by Integration "example", changed phase to "Scheduling"
Normal ReasonRelatedObjectChanged 119s camel-k-build-controller Build "kit-cb7ek9ssarkc73860meg", created by Integration "example", changed phase to "Pending"
Normal ReasonRelatedObjectChanged 119s camel-k-build-controller Build "kit-cb7ek9ssarkc73860meg", created by Integration "example", changed phase to "Running"
Normal ReasonRelatedObjectChanged 119s camel-k-integration-kit-controller Integration Kit "kit-cb7ek9ssarkc73860meg", created by Integration "example", changed phase to "Build Running"
Normal IntegrationConditionChanged 57s camel-k-integration-controller Condition "Ready" is "False" for Integration example: 0/1 updated replicas
Normal ReasonRelatedObjectChanged 57s camel-k-build-controller Build "kit-cb7ek9ssarkc73860meg", created by Integration "example", changed phase to "Succeeded"
Normal IntegrationConditionChanged 57s camel-k-integration-controller Condition "IntegrationKitAvailable" is "True" for Integration example: kit-cb7ek9ssarkc73860meg
Normal IntegrationPhaseUpdated 57s camel-k-integration-controller Integration "example" in phase "Deploying"
Normal IntegrationConditionChanged 57s camel-k-integration-controller Condition "CronJobAvailable" is "False" for Integration example: different controller strategy used (deployment)
Normal IntegrationConditionChanged 57s camel-k-integration-controller Condition "DeploymentAvailable" is "True" for Integration example: deployment name is example
Normal IntegrationConditionChanged 57s camel-k-integration-controller Condition "ServiceAvailable" is "False" for Integration example: no http service required
Normal IntegrationConditionChanged 57s camel-k-integration-controller Condition "ExposureAvailable" is "False" for Integration example: no host or service defined
Normal ReasonRelatedObjectChanged 57s camel-k-integration-kit-controller Integration Kit "kit-cb7ek9ssarkc73860meg", created by Integration "example", changed phase to "Ready"
Normal IntegrationConditionChanged 42s (x2 over 57s) camel-k-integration-controller (combined from similar events): Condition "Ready" is "False" for Integration example: Back-off pulling image "10.100.48.193/example/camel-k-kit-cb7ek9ssarkc73860meg@sha256:5761f6d045fe3549834462b619ef73396e84756ce03f2f25cf9cc9dfe4951e11"
Normal IntegrationConditionChanged 42s (x2 over 57s) camel-k-integration-controller (combined from similar events): Condition "Ready" is "False" for Integration example: Back-off pulling image "10.100.48.193/example/camel-k-kit-cb7ek9ssarkc73860meg@sha256:5761f6d045fe3549834462b619ef73396e84756ce03f2f25cf9cc9dfe4951e11"
Normal IntegrationPhaseUpdated 29s (x2 over 57s) camel-k-integration-controller Integration "example" in phase "Running"
Normal IntegrationConditionChanged 29s camel-k-integration-controller Condition "Ready" is "False" for Integration example: 0/1 ready replicas
Normal IntegrationPhaseUpdated 14s (x2 over 42s) camel-k-integration-controller Integration "example" in phase "Error"
Normal IntegrationConditionChanged 14s camel-k-integration-controller Condition "Ready" is "False" for Integration example: Back-off pulling image "10.100.48.193/example/camel-k-kit-cb7ek9ssarkc73860meg@sha256:5761f6d045fe3549834462b619ef73396e84756ce03f2f25cf9cc9dfe4951e11"
What are the events when you see the related k describe ik and k describe ikb?
Events from k describe ik:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal IntegrationKitPhaseUpdated 2m44s camel-k-integration-kit-controller Integration Kit "kit-cb82mq4tgdqc73a5603g" in phase "Build Submitted"
Normal IntegrationKitPhaseUpdated 2m44s camel-k-integration-kit-controller Integration Kit "kit-cb82mq4tgdqc73a5603g" in phase "Build Running"
Normal IntegrationKitPhaseUpdated 63s camel-k-integration-kit-controller Integration Kit "kit-cb82mq4tgdqc73a5603g" in phase "Ready"
Events from k describe ikb:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal BuildPhaseUpdated 3m54s camel-k-build-controller Build "kit-cb82mq4tgdqc73a5603g" in phase "Scheduling"
Normal BuildPhaseUpdated 3m54s camel-k-build-controller Build "kit-cb82mq4tgdqc73a5603g" in phase "Pending"
Normal BuildPhaseUpdated 3m54s camel-k-build-controller Build "kit-cb82mq4tgdqc73a5603g" in phase "Running"
Normal BuildPhaseUpdated 2m14s camel-k-build-controller Build "kit-cb82mq4tgdqc73a5603g" in phase "Succeeded"
After experimenting a little more, I think this is tied to the detection of the registry. If I use Docker Hub, everything works on both Minikube and Kind.
Continuing the experiment, if I run the registry:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2c726839ea18 docker.io/library/registry:2 /etc/docker/regis... About an hour ago Up About an hour ago 127.0.0.1:5001->5000/tcp kind-registry
ae0d4e841510 docker.io/kindest/node@sha256:0866296e693efe1fed79d5e6c7af8df71fc73ae45e3679af05342239cdc5bc8e 16 minutes ago Up 16 minutes ago 127.0.0.1:53017->6443/tcp kind-control-plane
```
With the registry attached to the Kind network and pull the IP:
```
$ podman inspect kind-registry -f='{{json .NetworkSettings.Networks.kind}}'
{"EndpointID":"","Gateway":"10.89.0.1","IPAddress":"10.89.0.4","IPPrefixLen":24,"IPv6Gateway":"fc00:f853:ccd:e793::1","GlobalIPv6Address":"fc00:f853:ccd:e793::4","GlobalIPv6PrefixLen":64,"MacAddress":"c2:52:5a:54:81:d2","NetworkID":"kind","DriverOpts":null,"IPAMConfig":null,"Links":null,"Aliases":["2c726839ea18"]}
```
Then install Camel-K with that address:
```
$ kamel install --olm=false -n $NS --registry 10.89.0.4:5000 --organization myorg --registry-insecure true --wait
```
And run the default integration, I get a successful build:
```
k get builds
NAME PHASE AGE STARTED DURATION ATTEMPTS
kit-cbcq60phte5c73bqjoq0 Succeeded 12m 12m 1m30.171809865s
A successful image push:
$ curl -X GET http://localhost:5001/v2/_catalog
{"repositories":["myorg/camel-k-kit-cbcq60phte5c73bqjoq0"]}
And a failed pod initialization:
$ k describe pod example-59b4bc7758-d8d2d
....
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 12m default-scheduler Successfully assigned example/example-59b4bc7758-d8d2d to kind-control-plane
Normal Pulling 10m (x4 over 12m) kubelet, kind-control-plane Pulling image "10.89.0.4:5000/myorg/camel-k-kit-cbcq60phte5c73bqjoq0@sha256:1bd03e9065dfd97939773f15ad83e87976f05ca3cd99d577bdc236fc36600ae0"
Warning Failed 10m (x4 over 12m) kubelet, kind-control-plane Failed to pull image "10.89.0.4:5000/myorg/camel-k-kit-cbcq60phte5c73bqjoq0@sha256:1bd03e9065dfd97939773f15ad83e87976f05ca3cd99d577bdc236fc36600ae0": rpc error: code = Unknown desc = failed to pull and unpack image "10.89.0.4:5000/myorg/camel-k-kit-cbcq60phte5c73bqjoq0@sha256:1bd03e9065dfd97939773f15ad83e87976f05ca3cd99d577bdc236fc36600ae0": failed to resolve reference "10.89.0.4:5000/myorg/camel-k-kit-cbcq60phte5c73bqjoq0@sha256:1bd03e9065dfd97939773f15ad83e87976f05ca3cd99d577bdc236fc36600ae0": failed to do request: Head "https://10.89.0.4:5000/v2/myorg/camel-k-kit-cbcq60phte5c73bqjoq0/manifests/sha256:1bd03e9065dfd97939773f15ad83e87976f05ca3cd99d577bdc236fc36600ae0": http: server gave HTTP response to HTTPS client
Warning Failed 10m (x4 over 12m) kubelet, kind-control-plane Error: ErrImagePull
Warning Failed 10m (x6 over 12m) kubelet, kind-control-plane Error: ImagePullBackOff
Normal BackOff 2m19s (x42 over 12m) kubelet, kind-control-plane Back-off pulling image "10.89.0.4:5000/myorg/camel-k-kit-cbcq60phte5c73bqjoq0@sha256:1bd03e9065dfd97939773f15ad83e87976f05ca3cd99d577bdc236fc36600ae0"
The error appears to be that the insecure registry setting is being respected by the push but not the pull.
Hi,
Sorry, my bad. When you installed operator with kamel install --global you shouldn't need to run the following. My previous instruction was wrong.
kamel install --skip-operator-setup -n $NS
What happens if you do the same steps above but without kamel install --skip-operator-setup? Does it go back to the original issue?
This issue has been automatically marked as stale due to 90 days of inactivity. It will be closed if no further activity occurs within 15 days. If you think that’s incorrect or the issue should never stale, please simply write any comment. Thanks for your contributions!
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}