che
che copied to clipboard
Published
20 hours ago •
eclipse-che
eclipse-che
Unable to update che - InstallOrUpdateFailed 'unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request'
Describe the bug
I cannot update my Che server with chectl server:update
Everything looks fine but
status:
cheClusterRunning: Available
cheURL: https://che-che.xxx.xxx.org
cheVersion: 7.39.1
dbProvisioned: true
devfileRegistryURL: https://devfile-registry-che.xxx.xxx.org/
devworkspaceStatus:
message: DevWorkspace CRDs are not installed
phase: Inactive
gitHubOAuthProvisioned: false
keycloakProvisioned: true
keycloakURL: https://keycloak-che.xxx.xxx.org/auth
message: 'unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1:
the server is currently unable to handle the request'
openShiftoAuthProvisioned: false
pluginRegistryURL: https://plugin-registry-che.xxx.xxx.org/v3
reason: InstallOrUpdateFailed
It is stuck to 7.39.1
Che version
7.40@latest
Steps to reproduce
chectl server:update
› Installer type is set to: 'operator'
› Current Kubernetes context: 'gke_eclipse-che-1-288510_europe-west1-b_eclipse-che-288510'
✔ Verify Kubernetes API...OK
✔ 🏃 Running the Eclipse Che operator Update
✔ Checking existing operator deployment before update...done
✔ Detecting existing version... 7.40.0 -> 7.41.1
✔ Check workspace engine compatibility...
Existing Eclipse Che operator: quay.io/eclipse/che-operator:7.40.0
New Eclipse Che operator : quay.io/eclipse/che-operator:7.41.1
You are going to update Eclipse Che 7.40.0 to 7.41.1
If you want to continue - press Y: Y
✔ ↺ Updating...
✔ 🏃 Running the Eclipse Che operator Update
✔ Updating ServiceAccount che-operator in namespace che...updated.
✔ Read Roles and Bindings...done.
✔ Updating Roles and Bindings...done.
✔ Updating Eclipse Che cluster CRD checlusters.org.eclipse.che...updated.
✔ Updating backup and restore CRDs...updated.
✔ Waiting 5 seconds for the new Kubernetes resources to get flushed...done.
✔ Updating deployment che-operator in namespace che...updated.
✔ Waiting newer operator to be run
↓ Patching the Custom Resource of type checlusters.org.eclipse.che [skipped]
Command server:update has completed successfully in 00:37.
Expected behavior
should be using 7.41.1 or 7.40.0 but it is stuck to 7.39.1
But chectl should at least warn me that something went wrong and that the operator couldn't make the update.
Runtime
Kubernetes (vanilla), other (please specify in additional context)
Screenshots
No response
Installation method
chectl/latest
Environment
GCE
Eclipse Che Logs
Using embedded assembly in /home/user/eclipse-che.
NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
03-Jan-2022 09:29:56.660 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/10.0.11
03-Jan-2022 09:29:56.701 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Sep 6 2021 16:22:12 UTC
03-Jan-2022 09:29:56.701 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 10.0.11.0
03-Jan-2022 09:29:56.702 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux
03-Jan-2022 09:29:56.702 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 5.4.129+
03-Jan-2022 09:29:56.702 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
03-Jan-2022 09:29:56.702 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /opt/java/openjdk
03-Jan-2022 09:29:56.702 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 11.0.11+9
03-Jan-2022 09:29:56.702 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: AdoptOpenJDK
03-Jan-2022 09:29:56.703 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /home/user/eclipse-che/tomcat
03-Jan-2022 09:29:56.703 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: /home/user/eclipse-che/tomcat
03-Jan-2022 09:29:56.732 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
03-Jan-2022 09:29:56.732 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
03-Jan-2022 09:29:56.733 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util=ALL-UNNAMED
03-Jan-2022 09:29:56.733 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
03-Jan-2022 09:29:56.733 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
03-Jan-2022 09:29:56.733 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/home/user/eclipse-che/tomcat/conf/logging.properties
03-Jan-2022 09:29:56.733 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
03-Jan-2022 09:29:56.733 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MaxRAMPercentage=85.0
03-Jan-2022 09:29:56.734 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.docker.network=bridge
03-Jan-2022 09:29:56.734 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djavax.net.ssl.trustStore=/home/user/cacerts
03-Jan-2022 09:29:56.734 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djavax.net.ssl.trustStorePassword=changeit
03-Jan-2022 09:29:56.734 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dport.http=8080
03-Jan-2022 09:29:56.734 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.home=/home/user/eclipse-che
03-Jan-2022 09:29:56.734 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.logs.dir=/logs/
03-Jan-2022 09:29:56.735 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.logs.level=INFO
03-Jan-2022 09:29:56.735 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djuli-logback.configurationFile=file:/home/user/eclipse-che/tomcat/conf/tomcat-logger.xml
03-Jan-2022 09:29:56.735 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
03-Jan-2022 09:29:56.735 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
03-Jan-2022 09:29:56.735 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0022
03-Jan-2022 09:29:56.735 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.local.conf.dir=/home/user/eclipse-che/tomcat/conf/
03-Jan-2022 09:29:56.736 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
03-Jan-2022 09:29:56.736 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/home/user/eclipse-che/tomcat
03-Jan-2022 09:29:56.736 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/home/user/eclipse-che/tomcat
03-Jan-2022 09:29:56.736 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/home/user/eclipse-che/tomcat/temp
03-Jan-2022 09:29:57.890 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
03-Jan-2022 09:29:57.936 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [1897] milliseconds
03-Jan-2022 09:29:58.090 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
03-Jan-2022 09:29:58.091 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/10.0.11]
03-Jan-2022 09:29:58.115 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war]
03-Jan-2022 09:29:59.028 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war] has finished in [913] ms
03-Jan-2022 09:29:59.032 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/api.war]
2022-01-03 09:30:11,630[main] [INFO ] [.e.c.c.d.JNDIDataSourceFactory 63] - This=org.eclipse.che.core.db.postgresql.PostgreSQLJndiDataSourceFactory@68547c81 obj=ResourceRef[className=javax.sql.DataSource,factoryClassLocation=null,factoryClassName=org.apache.naming.factory.ResourceFactory,{type=scope,content=Shareable},{type=auth,content=Container},{type=singleton,content=true},{type=factory,content=org.eclipse.che.core.db.postgresql.PostgreSQLJndiDataSourceFactory}] name=che Context=org.apache.naming.NamingContext@7aa5ea12 environment={}
2022-01-03 09:30:14,821[main] [INFO ] [.e.c.a.d.WsMasterServletModule 53] - Running in classic multi-user mode ...
2022-01-03 09:30:21,112[main] [INFO ] [o.e.c.m.k.s.OIDCInfoProvider 72] - Retrieving OpenId configuration from endpoint: http://keycloak.che.svc:8080/auth/realms/che/.well-known/openid-configuration
2022-01-03 09:30:21,614[main] [INFO ] [o.e.c.m.k.s.OIDCInfoProvider 81] - openid configuration = {issuer=https://keycloak-che.chegcp.xxx.org/auth/realms/che, authorization_endpoint=https://keycloak-che.chegcp.xxx.org/auth/realms/che/protocol/openid-connect/auth, token_endpoint=http://keycloak.che.svc:8080/auth/realms/che/protocol/openid-connect/token, introspection_endpoint=http://keycloak.che.svc:8080/auth/realms/che/protocol/openid-connect/token/introspect, userinfo_endpoint=http://keycloak.che.svc:8080/auth/realms/che/protocol/openid-connect/userinfo, end_session_endpoint=https://keycloak-che.chegcp.xxx.org/auth/realms/che/protocol/openid-connect/logout, jwks_uri=http://keycloak.che.svc:8080/auth/realms/che/protocol/openid-connect/certs, check_session_iframe=https://keycloak-che.chegcp.xxx.org/auth/realms/che/protocol/openid-connect/login-status-iframe.html, grant_types_supported=[authorization_code, implicit, refresh_token, password, client_credentials, urn:ietf:params:oauth:grant-type:device_code, urn:openid:params:grant-type:ciba], response_types_supported=[code, none, id_token, token, id_token token, code id_token, code token, code id_token token], subject_types_supported=[public, pairwise], id_token_signing_alg_values_supported=[PS384, ES384, RS384, HS256, HS512, ES256, RS256, HS384, ES512, PS256, PS512, RS512], id_token_encryption_alg_values_supported=[RSA-OAEP, RSA-OAEP-256, RSA1_5], id_token_encryption_enc_values_supported=[A256GCM, A192GCM, A128GCM, A128CBC-HS256, A192CBC-HS384, A256CBC-HS512], userinfo_signing_alg_values_supported=[PS384, ES384, RS384, HS256, HS512, ES256, RS256, HS384, ES512, PS256, PS512, RS512, none], request_object_signing_alg_values_supported=[PS384, ES384, RS384, HS256, HS512, ES256, RS256, HS384, ES512, PS256, PS512, RS512, none], request_object_encryption_alg_values_supported=[RSA-OAEP, RSA-OAEP-256, RSA1_5], request_object_encryption_enc_values_supported=[A256GCM, A192GCM, A128GCM, A128CBC-HS256, A192CBC-HS384, A256CBC-HS512], response_modes_supported=[query, fragment, form_post, query.jwt, fragment.jwt, form_post.jwt, jwt], registration_endpoint=http://keycloak.che.svc:8080/auth/realms/che/clients-registrations/openid-connect, token_endpoint_auth_methods_supported=[private_key_jwt, client_secret_basic, client_secret_post, tls_client_auth, client_secret_jwt], token_endpoint_auth_signing_alg_values_supported=[PS384, ES384, RS384, HS256, HS512, ES256, RS256, HS384, ES512, PS256, PS512, RS512], introspection_endpoint_auth_methods_supported=[private_key_jwt, client_secret_basic, client_secret_post, tls_client_auth, client_secret_jwt], introspection_endpoint_auth_signing_alg_values_supported=[PS384, ES384, RS384, HS256, HS512, ES256, RS256, HS384, ES512, PS256, PS512, RS512], authorization_signing_alg_values_supported=[PS384, ES384, RS384, HS256, HS512, ES256, RS256, HS384, ES512, PS256, PS512, RS512], authorization_encryption_alg_values_supported=[RSA-OAEP, RSA-OAEP-256, RSA1_5], authorization_encryption_enc_values_supported=[A256GCM, A192GCM, A128GCM, A128CBC-HS256, A192CBC-HS384, A256CBC-HS512], claims_supported=[aud, sub, iss, auth_time, name, given_name, family_name, preferred_username, email, acr], claim_types_supported=[normal], claims_parameter_supported=true, scopes_supported=[openid, phone, email, profile, address, offline_access, microprofile-jwt, roles, web-origins], request_parameter_supported=true, request_uri_parameter_supported=true, require_request_uri_registration=true, code_challenge_methods_supported=[plain, S256], tls_client_certificate_bound_access_tokens=true, revocation_endpoint=https://keycloak-che.chegcp.xxx.org/auth/realms/che/protocol/openid-connect/revoke, revocation_endpoint_auth_methods_supported=[private_key_jwt, client_secret_basic, client_secret_post, tls_client_auth, client_secret_jwt], revocation_endpoint_auth_signing_alg_values_supported=[PS384, ES384, RS384, HS256, HS512, ES256, RS256, HS384, ES512, PS256, PS512, RS512], backchannel_logout_supported=true, backchannel_logout_session_supported=true, device_authorization_endpoint=https://keycloak-che.chegcp.xxx.org/auth/realms/che/protocol/openid-connect/auth/device, backchannel_token_delivery_modes_supported=[poll, ping], backchannel_authentication_endpoint=http://keycloak.che.svc:8080/auth/realms/che/protocol/openid-connect/ext/ciba/auth, backchannel_authentication_request_signing_alg_values_supported=[PS384, ES384, RS384, ES256, RS256, ES512, PS256, PS512, RS512], require_pushed_authorization_requests=false, pushed_authorization_request_endpoint=http://keycloak.che.svc:8080/auth/realms/che/protocol/openid-connect/ext/par/request, mtls_endpoint_aliases={token_endpoint=http://keycloak.che.svc:8080/auth/realms/che/protocol/openid-connect/token, revocation_endpoint=https://keycloak-che.chegcp.xxx.org/auth/realms/che/protocol/openid-connect/revoke, introspection_endpoint=http://keycloak.che.svc:8080/auth/realms/che/protocol/openid-connect/token/introspect, device_authorization_endpoint=https://keycloak-che.chegcp.xxx.org/auth/realms/che/protocol/openid-connect/auth/device, registration_endpoint=http://keycloak.che.svc:8080/auth/realms/che/clients-registrations/openid-connect, userinfo_endpoint=http://keycloak.che.svc:8080/auth/realms/che/protocol/openid-connect/userinfo, pushed_authorization_request_endpoint=http://keycloak.che.svc:8080/auth/realms/che/protocol/openid-connect/ext/par/request, backchannel_authentication_endpoint=http://keycloak.che.svc:8080/auth/realms/che/protocol/openid-connect/ext/ciba/auth}}
2022-01-03 09:30:23,090[main] [INFO ] [o.j.p.kubernetes.KUBE_PING 131] - namespace che set; clustering enabled
-------------------------------------------------------------------
GMS: address=che-7dc9bdcb8-zgjdj-56851, cluster=RemoteSubscriptionChannel, physical address=10.0.36.9:7800
-------------------------------------------------------------------
2022-01-03 09:30:26,606[main] [INFO ] [o.jgroups.protocols.pbcast.GMS 125] - che-7dc9bdcb8-zgjdj-56851: no members discovered after 3477 ms: creating cluster as coordinator
2022-01-03 09:30:26,664[main] [INFO ] [o.j.p.kubernetes.KUBE_PING 131] - namespace che set; clustering enabled
-------------------------------------------------------------------
GMS: address=che-7dc9bdcb8-zgjdj-60054, cluster=WorkspaceLocks, physical address=10.0.36.9:7801
-------------------------------------------------------------------
2022-01-03 09:30:29,794[main] [INFO ] [o.jgroups.protocols.pbcast.GMS 125] - che-7dc9bdcb8-zgjdj-60054: no members discovered after 3102 ms: creating cluster as coordinator
2022-01-03 09:30:29,815[main] [INFO ] [o.j.p.kubernetes.KUBE_PING 131] - namespace che set; clustering enabled
-------------------------------------------------------------------
GMS: address=che-7dc9bdcb8-zgjdj-10765, cluster=WorkspaceStateCache, physical address=10.0.36.9:7802
-------------------------------------------------------------------
2022-01-03 09:30:32,958[main] [INFO ] [o.jgroups.protocols.pbcast.GMS 125] - che-7dc9bdcb8-zgjdj-10765: no members discovered after 3117 ms: creating cluster as coordinator
2022-01-03 09:30:33,417[main] [INFO ] [o.f.c.i.d.DbSupportFactory 44] - Database: jdbc:postgresql://postgres:5432/dbche (PostgreSQL 9.6)
2022-01-03 09:30:33,694[main] [INFO ] [o.f.c.i.util.VersionPrinter 44] - Flyway 4.2.0 by Boxfuse
2022-01-03 09:30:33,703[main] [INFO ] [o.f.c.i.d.DbSupportFactory 44] - Database: jdbc:postgresql://postgres:5432/dbche (PostgreSQL 9.6)
2022-01-03 09:30:33,805[main] [INFO ] [i.f.CustomSqlMigrationResolver 158] - Searching for SQL scripts in locations [classpath:che-schema]
2022-01-03 09:30:33,991[main] [INFO ] [o.f.c.i.command.DbValidate 44] - Successfully validated 63 migrations (execution time 00:00.166s)
2022-01-03 09:30:34,018[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Current version of schema "public": 7.26.0.1
2022-01-03 09:30:34,021[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Schema "public" is up to date. No migration necessary.
2022-01-03 09:30:36,433[main] [INFO ] [o.j.p.kubernetes.KUBE_PING 131] - namespace che set; clustering enabled
-------------------------------------------------------------------
GMS: address=che-7dc9bdcb8-zgjdj-45707, cluster=EclipseLinkCommandChannel, physical address=10.0.36.9:7803
-------------------------------------------------------------------
2022-01-03 09:30:39,540[main] [INFO ] [o.jgroups.protocols.pbcast.GMS 125] - che-7dc9bdcb8-zgjdj-45707: no members discovered after 3047 ms: creating cluster as coordinator
2022-01-03 09:30:39,606[main] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 182] - Configured factories for environments: '[kubernetes, no-environment]'
2022-01-03 09:30:39,607[main] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 183] - Registered infrastructure 'kubernetes'
2022-01-03 09:30:39,740[main] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 760] - Infrastructure is tracking 2 active runtimes
2022-01-03 09:30:39,801[aceSharedPool-0] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 981] - Recovering of runtimes is started.
2022-01-03 09:30:39,821[aceSharedPool-0] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 987] - Recovering runtime workspaceb3gnvdsbvb7kbmdb
2022-01-03 09:30:40,039[main] [INFO ] [o.e.c.a.c.u.ApiInfoLogInformer 36] - Eclipse Che Api Core: Build info '7.39.1' scmRevision '68818ce458bc602bbc27c5e74d2bcca8b8abae63' implementationVersion '7.39.1'
2022-01-03 09:30:40,152[main] [INFO ] [o.e.c.c.metrics.MetricsServer 46] - Metrics server started at port 8087 successfully
03-Jan-2022 09:30:41.505 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/api.war] has finished in [42,473] ms
03-Jan-2022 09:30:41.509 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war]
2022-01-03 09:30:45,830[aceSharedPool-0] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 828] - Successfully recovered workspace runtime 'workspaceb3gnvdsbvb7kbmdb'. Its status is 'RUNNING'
2022-01-03 09:30:45,831[aceSharedPool-0] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 987] - Recovering runtime workspace59mwi9k6p74yd49l
2022-01-03 09:30:46,132[aceSharedPool-0] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 828] - Successfully recovered workspace runtime 'workspace59mwi9k6p74yd49l'. Its status is 'RUNNING'
2022-01-03 09:30:46,133[aceSharedPool-0] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 1004] - All runtimes have been recovered in 6 seconds.
03-Jan-2022 09:30:47.539 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war] has finished in [6,030] ms
03-Jan-2022 09:30:47.594 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
03-Jan-2022 09:30:47.608 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [49671] milliseconds
2022-01-03 09:39:33,213[nio-8080-exec-9] [WARN ] [bstractWorkspaceServiceAccount 149] - Unable to add metrics roles due to insufficient permissions. Workspace metrics will be disabled.
2022-01-03 09:51:52,198[nio-8080-exec-8] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 549] - Workspace 'sunix/event-publisher-mhbt' with id 'workspaceb3gnvdsbvb7kbmdb' is stopping by user 'sunix'
2022-01-03 09:51:54,386[aceSharedPool-1] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 1100] - Workspace 'sunix/event-publisher-mhbt' with id 'workspaceb3gnvdsbvb7kbmdb' is stopped by user 'sunix'
2022-01-03 09:52:45,478[io-8080-exec-10] [INFO ] [o.e.c.a.w.s.WorkspaceManager 364] - Workspace 'workspaceb3gnvdsbvb7kbmdb' removed by user 'sunix'
che-operator.log
2022-01-03T10:23:44.121Z INFO Binary info {"Go version": "go1.16.12"}
2022-01-03T10:23:44.122Z INFO Binary info {"OS": "linux", "Arch": "amd64"}
2022-01-03T10:23:44.122Z INFO Address {"Metrics": ":60000"}
2022-01-03T10:23:44.122Z INFO Address {"Probe": ":6789"}
2022-01-03T10:23:44.122Z INFO Operator is running on {"Infrastructure": "Kubernetes"}
I0103 10:23:45.178603 1 request.go:668] Waited for 1.045167021s due to client-side throttling, not priority and fairness, request: GET:https://10.3.240.1:443/apis/cert-manager.io/v1alpha2?timeout=32s
time="2022-01-03T10:23:45Z" level=info msg="Limit cache by selector: app.kubernetes.io/part-of=che.eclipse.org"
2022-01-03T10:23:46.338Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":60000"}
time="2022-01-03T10:23:47Z" level=info msg="Use 'terminationGracePeriodSeconds' 20 sec. from operator deployment."
time="2022-01-03T10:23:47Z" level=info msg="Set up process signal handler"
2022-01-03T10:23:48.561Z INFO setup DevWorkspace support disabled. Will initiate restart when CheCluster with devworkspaces enabled will appear
2022-01-03T10:23:48.561Z INFO setup starting manager
I0103 10:23:48.561589 1 leaderelection.go:243] attempting to acquire leader lease che/e79b08a4.org.eclipse.che...
2022-01-03T10:23:48.561Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
### Additional context
_No response_
@skabashnyuk may that be related to the metrics work that have been done recently ?
@sunix Is it full che-operator log? I don't see any errors there. So, it might be leftovers from the previous deployments (error in the status).
@tolusha I had that recently:
2022-01-13 23:00:09.594 CETtime="2022-01-13T22:00:09Z" level=error msg="Error discovering image puller APIs: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"
2022-01-13 23:00:09.666 CETtime="2022-01-13T22:00:09Z" level=info msg="Custom resource status eclipse-che updated with status: Reason: InstallOrUpdateFailed"
2022-01-13 23:00:09.684 CETtime="2022-01-13T22:00:09Z" level=info msg="Custom resource status eclipse-che updated with status: Message: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"
2021-12-16 23:00:04.615 CETtime="2021-12-16T22:00:04Z" level=info msg="Signal SIGTERM stop"
2021-12-16 23:00:10.703 CETtime="2021-12-16T22:00:10Z" level=error msg="Error discovering image puller APIs: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"
2021-12-16 23:00:10.857 CETtime="2021-12-16T22:00:10Z" level=info msg="Custom resource status eclipse-che updated with status: Reason: InstallOrUpdateFailed"
2021-12-16 23:00:10.902 CETtime="2021-12-16T22:00:10Z" level=info msg="Custom resource status eclipse-che updated with status: Message: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"
2021-12-16 23:00:11.008 CET2021-12-16T22:00:10.869Z ERROR controller-runtime.manager.controller.checluster Reconciler error {"reconciler group": "org.eclipse.che", "reconciler kind": "CheCluster", "name": "eclipse-che", "namespace": "che", "error": "unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"}
unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
Nothing woring with operator. It basically about that metrics server can't proceed requests. Pls check your kubernetes cluster.
I would recommend you to figure out what wrong with metrics server because it blocks che-operator reconciliation. For instance I can find the similar problem here https://github.com/kubernetes-sigs/metrics-server/issues/157
It fails only when I am scaling down to 0 everything, it should work otherwise. Maybe che-operator shouldn't fail if this kind of issue occurs.
@sunix Make sense to me to handle this kind of error. Will try to provide a fix
Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.
Mark the issue as fresh with /remove-lifecycle stale in a new comment.
If this issue is safe to close now please do so.
Moderators: Add lifecycle/frozen label to avoid stale mode.
