eclipse-che
workspace fails and starts several times. not sure why
Summary
chectl chectl/0.0.20220809-next.c9f4ba1 darwin-x64 node-v16.13.2
command used to install eclipse-che
chectl server:deploy --che-operator-cr-patch-yaml=/Users/divine/Documents/office_tasks/TAP-4540/terraformCheJune17/eclipseche_yaml/che-operator-cr-patch.yaml --platform=k8s --installer=operator --debug --k8spoderrorrechecktimeout=1500000 --domain=eclipseche-dchelladurai-chejune15.calatrava.vmware.com --k8spodreadytimeout=1500000
project-clone container in pod "workspace0fbe074d8c144b0a-66b54c49d9-2gqkt" shows below error
kubectl events on user namespace divine-chelladurai-che-pmjg9u
below is my devfile.yaml
schemaVersion: 2.1.0
metadata:
name: cbfsel-repo1
projects:
- name: cbfsel-project1
git:
checkoutFrom:
revision: master
remotes:
origin: https://gitlab.eng.com/dchelladurai/cbf-sel.git
components:
- container:
image: 'quay.io/devfile/universal-developer-image:ubi8-b452131'
memoryLimit: 4G
name: javacontainer
project-clone container logs when workspace starts successfully (log1)
2022/08/30 04:04:30 Using temporary directory /projects/project-clone-2868426929
2022/08/30 04:04:30 Read DevWorkspace at /devworkspace-metadata/flattened.devworkspace.yaml
2022/08/30 04:04:30 Processing project cbfsel-project1
2022/08/30 04:04:30 Cloning project cbfsel-project1 to /projects/project-clone-2868426929/cbfsel-project1
Cloning into '/projects/project-clone-2868426929/cbfsel-project1'...
fatal: unable to get credential storage lock in 1000 ms: Permission denied
Updating files: 41% (1749/4191)
Updating files: 42% (1761/4191)
Updating files: 43% (1803/4191)
Updating files: 44% (1845/4191)
Updating files: 45% (1886/4191)
Updating files: 46% (1928/4191)
Updating files: 47% (1970/4191)
Updating files: 48% (2012/4191)
Updating files: 49% (2054/4191)
Updating files: 50% (2096/4191)
Updating files: 51% (2138/4191)
Updating files: 52% (2180/4191)
Updating files: 53% (2222/4191)
Updating files: 54% (2264/4191)
Updating files: 55% (2306/4191)
Updating files: 56% (2347/4191)
Updating files: 57% (2389/4191)
Updating files: 58% (2431/4191)
Updating files: 59% (2473/4191)
Updating files: 60% (2515/4191)
Updating files: 61% (2557/4191)
Updating files: 62% (2599/4191)
Updating files: 63% (2641/4191)
Updating files: 64% (2683/4191)
Updating files: 65% (2725/4191)
Updating files: 66% (2767/4191)
Updating files: 67% (2808/4191)
Updating files: 68% (2850/4191)
Updating files: 69% (2892/4191)
Updating files: 70% (2934/4191)
Updating files: 71% (2976/4191)
Updating files: 72% (3018/4191)
Updating files: 73% (3060/4191)
Updating files: 74% (3102/4191)
Updating files: 75% (3144/4191)
Updating files: 76% (3186/4191)
Updating files: 77% (3228/4191)
Updating files: 78% (3269/4191)
Updating files: 79% (3311/4191)
Updating files: 80% (3353/4191)
Updating files: 81% (3395/4191)
Updating files: 82% (3437/4191)
Updating files: 83% (3479/4191)
Updating files: 84% (3521/4191)
Updating files: 85% (3563/4191)
Updating files: 86% (3605/4191)
Updating files: 87% (3647/4191)
Updating files: 88% (3689/4191)
Updating files: 89% (3730/4191)
Updating files: 90% (3772/4191)
Updating files: 91% (3814/4191)
Updating files: 92% (3856/4191)
Updating files: 93% (3898/4191)
Updating files: 94% (3940/4191)
Updating files: 95% (3982/4191)
Updating files: 96% (4024/4191)
Updating files: 96% (4033/4191)
Updating files: 97% (4066/4191)
Updating files: 98% (4108/4191)
Updating files: 99% (4150/4191)
Updating files: 100% (4191/4191)
Updating files: 100% (4191/4191), done.
2022/08/30 04:05:02 Cloned project cbfsel-project1 to /projects/project-clone-2868426929/cbfsel-project1
2022/08/30 04:05:02 Setting up remotes for project cbfsel-project1
fatal: unable to get credential storage lock in 1000 ms: Permission denied
2022/08/30 04:05:02 Fetched remote origin at https://gitlab.eng.vmware.com/dchelladurai/cbf-sel.git
2022/08/30 04:05:03 Encountered error while setting up project cbfsel-project1: failed to checkout revision: failed to read remote origin: authentication required
project-clone container logs when workspace starts successfully (log2)
2022/08/30 04:36:03 Using temporary directory /projects/project-clone-42631352
2022/08/30 04:36:03 Read DevWorkspace at /devworkspace-metadata/flattened.devworkspace.yaml
2022/08/30 04:36:03 Processing project cbfsel-project1
2022/08/30 04:36:03 Project 'cbfsel-project1' is already cloned and has all remotes configured
Relevant information
No response
@amisevsk can you look at this issue, please? There seem some problems with the project-clone container.
i used below devfile.yaml
schemaVersion: 2.1.0
metadata:
name: cbfsel-repo4
projects:
- name: cbfsel-project4
git:
checkoutFrom:
revision: master
remotes:
origin: https://gitlab.eng.vmware.com/dchelladurai/cbf-sel.git
components:
- container:
image: 'quay.io/devfile/base-developer-image:ubi8-latest'
memoryLimit: 4G
name: javacontainer
received below error in project-clone
i used below devfile.yaml
schemaVersion: 2.1.0
metadata:
name: cbfsel-repo4
projects:
- name: cbfsel-project4
git:
checkoutFrom:
revision: master
remotes:
origin: https://gitlab.eng.vmware.com/dchelladurai/cbf-sel.git
components:
- container:
image: 'quay.io/devfile/universal-developer-image:ubi8-latest'
memoryLimit: 4G
name: javacontainer
received below error in project-clone
2022/08/31 13:03:19 Using temporary directory /projects/project-clone-2413608989
2022/08/31 13:03:19 Read DevWorkspace at /devworkspace-metadata/flattened.devworkspace.yaml
2022/08/31 13:03:19 Processing project cbfsel-project4
2022/08/31 13:03:19 Cloning project cbfsel-project4 to /projects/project-clone-2413608989/cbfsel-project4
Cloning into '/projects/project-clone-2413608989/cbfsel-project4'...
fatal: unable to get credential storage lock in 1000 ms: Permission denied
Updating files: 41% (1748/4190)
Updating files: 42% (1760/4190)
Updating files: 43% (1802/4190)
Updating files: 44% (1844/4190)
Updating files: 45% (1886/4190)
Updating files: 46% (1928/4190)
Updating files: 47% (1970/4190)
Updating files: 48% (2012/4190)
Updating files: 49% (2054/4190)
Updating files: 50% (2095/4190)
Updating files: 51% (2137/4190)
Updating files: 52% (2179/4190)
Updating files: 53% (2221/4190)
Updating files: 54% (2263/4190)
Updating files: 55% (2305/4190)
Updating files: 56% (2347/4190)
Updating files: 57% (2389/4190)
Updating files: 58% (2431/4190)
Updating files: 59% (2473/4190)
Updating files: 60% (2514/4190)
Updating files: 61% (2556/4190)
Updating files: 62% (2598/4190)
Updating files: 63% (2640/4190)
Updating files: 64% (2682/4190)
Updating files: 65% (2724/4190)
Updating files: 66% (2766/4190)
Updating files: 67% (2808/4190)
Updating files: 68% (2850/4190)
Updating files: 69% (2892/4190)
Updating files: 70% (2933/4190)
Updating files: 71% (2975/4190)
Updating files: 72% (3017/4190)
Updating files: 73% (3059/4190)
Updating files: 74% (3101/4190)
Updating files: 75% (3143/4190)
Updating files: 76% (3185/4190)
Updating files: 77% (3227/4190)
Updating files: 78% (3269/4190)
Updating files: 79% (3311/4190)
Updating files: 80% (3352/4190)
Updating files: 81% (3394/4190)
Updating files: 82% (3436/4190)
Updating files: 83% (3478/4190)
Updating files: 84% (3520/4190)
Updating files: 85% (3562/4190)
Updating files: 86% (3604/4190)
Updating files: 87% (3646/4190)
Updating files: 88% (3688/4190)
Updating files: 88% (3688/4190)
Updating files: 89% (3730/4190)
Updating files: 90% (3771/4190)
Updating files: 91% (3813/4190)
Updating files: 92% (3855/4190)
Updating files: 93% (3897/4190)
Updating files: 94% (3939/4190)
Updating files: 95% (3981/4190)
Updating files: 96% (4023/4190)
Updating files: 96% (4032/4190)
Updating files: 97% (4065/4190)
Updating files: 98% (4107/4190)
Updating files: 99% (4149/4190)
Updating files: 100% (4190/4190)
Updating files: 100% (4190/4190), done.
2022/08/31 13:04:10 Cloned project cbfsel-project4 to /projects/project-clone-2413608989/cbfsel-project4
2022/08/31 13:04:10 Setting up remotes for project cbfsel-project4
fatal: unable to get credential storage lock in 1000 ms: Permission denied
2022/08/31 13:04:10 Fetched remote origin at https://gitlab.eng.vmware.com/dchelladurai/cbf-sel.git
2022/08/31 13:04:11 Encountered error while setting up project cbfsel-project4: failed to checkout revision: failed to read remote origin: authentication required
unable to retrieve container logs for containerd://b23ae9ce04107ea4baa79df8d9849d1273c07530a4e25815a52aab167de9e009
@Divine1 can you confirm how you've configured authentication for your repository? Is it through a SSH key or a personal access token?
@amisevsk
i have installed below secret component in eclipse-che namespace. i have also configured dex with gitlab sso.
does this answer your question?
kind: Secret
apiVersion: v1
metadata:
name: gitlab-oauth-config
namespace: 'eclipse-che'
labels:
app.kubernetes.io/part-of: che.eclipse.org
app.kubernetes.io/component: oauth-scm-configuration
annotations:
che.eclipse.org/oauth-scm-server: gitlab
che.eclipse.org/scm-server-endpoint: 'https://gitlab.eng.vmware.com'
type: Opaque
data:
id: 'ZWrFkNDhjmY2OGQ4NDY5ODZmMTQzZTYyNDllMzJhMDZhZA=='
secret: 'OrGM0MDMwNWQ22YWUwMjhjZGM2NThkMA=='
@amisevsk
may be i'm wrong, but from my observation it looks like in my devfile.yaml when i use quay.io/devfile/universal-developer-image:ubi8-latest image, the workspace starts even though there are few errors in project-clone container logs. But when i use divine6/customopenjdk8:v10 image , my workspace doesn't start and project-clone container shows failure logs.
please share your thoughts on this @amisevsk
below is my devfile.yaml
schemaVersion: 2.1.0
metadata:
name: cbfsel-repo4
projects:
- name: cbfsel-project4
git:
checkoutFrom:
revision: master
remotes:
origin: https://gitlab.eng.vmware.com/dchelladurai/cbf-sel.git
components:
- container:
image: 'quay.io/devfile/universal-developer-image:ubi8-latest'
memoryLimit: 4G
name: javacontainer
The project-clone container's success/failure should not depend on the devfile image used. However, there might be an issue in how your GitLab credentials are being propagated to the workspace (going off of the log message
Encountered error while setting up project cbfsel-project4: failed to checkout revision: failed to read remote origin: authentication required
What Che should be doing is configuring the DevWorkspace Operator to add your git credentials, to allow you to clone the repository. There are two ways for this to happen: either your workspace mounts an SSH key or it mounts a Personal Access Token. Can you check if there is a secret in your user namespace with the label controller.devfile.io/git-credential (do not post the data section of that secret if it exists, as it contains an access token for your GitLab account!)
Some other notes:
- The project-clone container should never cause the workspace to fail to start, so I suspect something else is going wrong there -- what is the message when the workspace fails to start with image
divine6/customopenjdk8:v10? You can get this by looking atdevworkspacecustom resource objects in the user namespace on the cluster:oc get dw -n <your namespace> - If the default branch in your repo (
https://gitlab.eng.vmware.com/dchelladurai/cbf-sel.git) ismasteryou can try omitting thecheckoutFromsection, since from the logs it appears to clone successfully.
@amisevsk
i have installed below secret component in eclipse-che namespace. i have also configured dex with gitlab sso.
kind: Secret
apiVersion: v1
metadata:
name: gitlab-oauth-config
namespace: 'eclipse-che'
labels:
app.kubernetes.io/part-of: che.eclipse.org
app.kubernetes.io/component: oauth-scm-configuration
annotations:
che.eclipse.org/oauth-scm-server: gitlab
che.eclipse.org/scm-server-endpoint: 'https://gitlab.eng.vmware.com'
type: Opaque
data:
id: 'ZWrFkNDhjmY2OGQ4NDY5ODZmMTQzZTYyNDllMzJhMDZhZA=='
secret: 'OrGM0MDMwNWQ22YWUwMjhjZGM2NThkMA=='
below secret is in my namespace divine-chelladurai-che-pmjg9u
apiVersion: v1
kind: Secret
metadata:
name: git-credentials-secret-94a92
namespace: divine-chelladurai-che-pmjg9u
uid: ed457f9e-6e52-448d-98a2-6b87d90a4615
resourceVersion: '34768337'
creationTimestamp: '2022-08-31T18:18:39Z'
labels:
app.kubernetes.io/component: workspace-secret
app.kubernetes.io/part-of: che.eclipse.org
controller.devfile.io/git-credential: 'true'
controller.devfile.io/watch-secret: 'true'
annotations:
che.eclipse.org/automount-workspace-secret: 'true'
che.eclipse.org/che-userid: CgUyMTAwMhIGZ2l0bGFi
che.eclipse.org/git-credential: 'true'
che.eclipse.org/mount-as: file
che.eclipse.org/mount-path: /.git-credentials
che.eclipse.org/scm-url: https://gitlab.eng.vmware.com
che.eclipse.org/scm-username: dchelladurai
controller.devfile.io/mount-path: /.git-credentials
managedFields:
- manager: okhttp
operation: Update
apiVersion: v1
time: '2022-08-31T18:18:39Z'
fieldsType: FieldsV1
fieldsV1:
f:data:
.: {}
f:credentials: {}
f:metadata:
f:annotations:
.: {}
f:che.eclipse.org/automount-workspace-secret: {}
f:che.eclipse.org/che-userid: {}
f:che.eclipse.org/git-credential: {}
f:che.eclipse.org/mount-as: {}
f:che.eclipse.org/mount-path: {}
f:che.eclipse.org/scm-url: {}
f:che.eclipse.org/scm-username: {}
f:controller.devfile.io/mount-path: {}
f:labels:
.: {}
f:app.kubernetes.io/component: {}
f:app.kubernetes.io/part-of: {}
f:controller.devfile.io/git-credential: {}
f:controller.devfile.io/watch-secret: {}
f:type: {}
selfLink: >-
/api/v1/namespaces/divine-chelladurai-che-pmjg9u/secrets/git-credentials-secret-94a92
type: Opaque
data:
credentials: >-
aHR0cHM6Ly9vYXY5YmM3NjkyN2UwMTVkNWY1NjNjYWViYTlmNjg1ZTNlMTNlZkBnaXRsYWIuZW5nLnZtd2FyZS5jb20=
below secret is in my namespace divine-chelladurai-che-pmjg9u
apiVersion: v1
kind: Secret
metadata:
name: personal-access-token-0mria
namespace: divine-chelladurai-che-pmjg9u
uid: 0ff04cbe-54e7-48cc-abd4-9fca6f8f0184
resourceVersion: '34768333'
creationTimestamp: '2022-08-31T18:18:38Z'
labels:
app.kubernetes.io/component: scm-personal-access-token
app.kubernetes.io/part-of: che.eclipse.org
annotations:
che.eclipse.org/che-userid: CgUyMTAwMhIGZ2l0bGFi
che.eclipse.org/scm-personal-access-token-id: id-90kfs
che.eclipse.org/scm-personal-access-token-name: oauth2-jcd9m
che.eclipse.org/scm-url: https://gitlab.eng.vmware.com
che.eclipse.org/scm-userid: '21002'
che.eclipse.org/scm-username: dchelladurai
managedFields:
- manager: okhttp
operation: Update
apiVersion: v1
time: '2022-08-31T18:18:38Z'
fieldsType: FieldsV1
fieldsV1:
f:data:
.: {}
f:token: {}
f:metadata:
f:annotations:
.: {}
f:che.eclipse.org/che-userid: {}
f:che.eclipse.org/scm-personal-access-token-id: {}
f:che.eclipse.org/scm-personal-access-token-name: {}
f:che.eclipse.org/scm-url: {}
f:che.eclipse.org/scm-userid: {}
f:che.eclipse.org/scm-username: {}
f:labels:
.: {}
f:app.kubernetes.io/component: {}
f:app.kubernetes.io/part-of: {}
f:type: {}
selfLink: >-
/api/v1/namespaces/divine-chelladurai-che-pmjg9u/secrets/personal-access-token-0mria
type: Opaque
data:
token: >-
YmU4YzUxZDY5YmM3Njk0MWY1NjNjYWViYTlmNjg1ZTNlMTNlZg==
After looking into the project-clone container, I figured out there's an issue with checkoutFrom in devfiles when auth is required: https://github.com/devfile/devworkspace-operator/issues/913. The workaround is to omit checkoutFrom from the devfile and checkout the desired branch once the workspace starts if necessary. For the devfiles linked above, I believe simply omitting checkoutFrom would resolve the issue, as master is typically a default branch.
Even with the issue above, project-clone is not the reason the workspace is failing; the log
2022/08/31 13:04:11 Encountered error while setting up project cbfsel-project4: failed to checkout revision: failed to read remote origin: authentication required
Does not cause the project-clone container to exit with a nonzero exit code, and workspace start should continue. From the screenshots linked, it looks like there are other issues in the cluster itself which prevent starting workspace pods.
@amisevsk could you let me know how to debug and locate the logs for this issue? i'm available to share the entire cluster information/logs with you. please help
I'm not able to debug the cluster itself, but some useful information about the cluster/images would be
- What is the final status of the workspaces that are failing to start? I.e. when the workspace fails, what is in the
.statusfield of the DevWorkspace CR in your namespace- You can get this information from the kubernetes commandline:
kubectl get devworkspace <workspace-name> -n <namespace> --output yaml <workspace-name>can be found through the Che dashboard (what's the name of the failing workspace) or through the commandline:kubectl get devworkspaces -n <namespace><namespace>is the namespace your workspaces are created in and depends on your username in Che or on Che configuration
- You can get this information from the kubernetes commandline:
- You can get logs from the DevWorkspace Operator by using
kubectl logs -n "$NAMESPACE" -f deploy/devworkspace-controller-manager -c devworkspace-controllerwhere$NAMESPACEis the namespace where you installed Che - How are you building the image
divine6/customopenjdk8:v10? Is it available in the cluster? What is the entrypoint/args for the image? - Try removing the
checkoutFromfrom the devfile as suggested:schemaVersion: 2.1.0 metadata: name: cbfsel-repo4 projects: - name: cbfsel-project4 git: - checkoutFrom: - revision: master remotes: origin: https://gitlab.eng.vmware.com/dchelladurai/cbf-sel.git components: - container: image: 'quay.io/devfile/universal-developer-image:ubi8-latest' memoryLimit: 4G name: javacontainer
You're reporting what seems to be 3-4 disparate issues:
- Project clone is failing to clone your project -- you should be able to work around this by removing
checkoutFromfrom the devfile as above. The underlying issue is https://github.com/devfile/devworkspace-operator/issues/913 - The Devfile is failing to start when you use
divine6/customopenjdk8:v10instead ofquay.io/devfile/universal-developer-image:ubi8-latest-- for this you need to figure out what the issue with your image is (I can't access it to look at it), but I suspect it's a terminating image or something similar. The DevWorkspace CR status as described above would give us tips here - There is/was some issue cluster-wide that is also happening -- not sure if that's resolved but there's the warning event that
0/4 nodes available. That's beyond our scope for fixing.
Does the devfile below work on your cluster?
schemaVersion: 2.1.0
metadata:
name: cbfsel-repo4
projects:
- name: cbfsel-project4
git:
remotes:
origin: https://gitlab.eng.vmware.com/dchelladurai/cbf-sel.git
components:
- container:
image: 'quay.io/devfile/universal-developer-image:ubi8-latest'
memoryLimit: 4G
name: javacontainer
@amisevsk
"Project clone is failing to clone your project -- you should be able to work around this by removing checkoutFrom from the devfile as above."
after removing checkoutFrom from the devfile.yaml , i feel like the frequency of the issue is reduced. but still i can see the issue sometimes. i will test this scenario with multiple user logins and confirm (i need to get hold of some of the developers from my team for this)
if the workspace fails to load at step 3.waiting for workspace to start, then if i refresh the webpage 2 or 3 times (webpage screenshot posted below),then the workspace loads successfully.
"Does the devfile below work on your cluster?"
currently i use below devfile.yaml
the gitlab repository is accessible only within the organization network. the repository contains Java8,Maven project with selenium scripts to run browser based tests.
i built the image divine6/customopenjdk8:v10 using the Dockerfile present in this repository (https://github.com/Divine1/customopenjdk8Version.git) . xvfb tool is required to run my project, so this tool will be present in Dockerfile. The size of this image is 900mb. So, i thought eclipse-che workspace will load much quicker using this image.
schemaVersion: 2.1.0
metadata:
name: cbfsel-repo7
projects:
- name: cbfsel-project7
git:
remotes:
origin: https://gitlab.eng.vmware.com/dchelladurai/cbf-sel.git
components:
- container:
image: 'divine6/customopenjdk8:v10'
memoryLimit: 4G
volumeMounts:
- name: m2volume
path: /home/user/.m2
name: javacontainer
- name: m2volume
volume:
size: 4G
i'm looking forward for your feedback
@amisevsk
after removing below lines from my devfile.yaml, i'm not facing above issues
- checkoutFrom:
- revision: master
below is my new devfile.yaml
schemaVersion: 2.1.0
metadata:
name: cbfsel-repo7
projects:
- name: cbfsel-project7
git:
remotes:
origin: https://gitlab.eng.com/selen.git
components:
- container:
image: 'artfact-prd.com:5001/qedocker/eclipseche/customopenjdk8:v10'
memoryLimit: 4G
volumeMounts:
- name: m2volume
path: /home/user/.m2
name: javacontainer
- name: m2volume
volume:
size: 4G
when the checkoutFrom issue will be fixed? because allowing users to always directly commit to master branch via eclipse-che theia editor is not a good idea. i want to set git remotes to a different branch (dev branch or stage branch) . How can i do it?
projects:
- name: cbfsel-project7
git:
remotes:
origin: https://gitlab.eng.vmware.com/dchelladurai/cbf-sel.git
@Divine1 https://github.com/devfile/devworkspace-operator/issues/913 is the issue to follow, we will work on it as we have time and given where it fits into priority. Since there's a workaround it's not top-urgency at the moment, but I would expect a fix for DWO v0.17.0 or v0.18.0.
It's worth noting that even if you clone and checkout the main/master branch, you can still use a normal git workflow from within the workspace. In the Theia editor, you can open a terminal and create a new branch to work in and submit work for. I agree however that the hassle of adding remotes for each new workspace is not ideal.
Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.
Mark the issue as fresh with /remove-lifecycle stale in a new comment.
If this issue is safe to close now please do so.
Moderators: Add lifecycle/frozen label to avoid stale mode.
