uyuni
uyuni copied to clipboard
uyuni-project
ERROR: Request has timed out, please reload the page
Problem description
During some navigation especially on the Server List page when we try to search for a server, the system becomes unreachable, and to recover the Web Portal again is required to restart the service via spacewalk-service restart. This issue started happening after the last Minor Upgrade performed by updating the system to the version Uyuni-Server-release-2022.08-185.2.uyuni2.x86_64.
Version of Uyuni Server and Proxy
Uyuni Server:
Uyuni-Server:~ # zypper info Uyuni-Server-release
Loading repository data...
Reading installed packages...
Information for package Uyuni-Server-release:
---------------------------------------------
Repository : uyuni-server-stable
Name : Uyuni-Server-release
Version : 2022.08-185.2.uyuni2
Arch : x86_64
Vendor : obs://build.opensuse.org/systemsmanagement:Uyuni
Support Level : Level 3
Installed Size : 1.4 KiB
Installed : Yes (automatically)
Status : up-to-date
Source package : Uyuni-Server-release-2022.08-185.2.uyuni2.src
Summary : Uyuni Server
Description :
Uyuni lets you efficiently manage physical, virtual,
and cloud-based Linux systems. It provides automated and cost-effective
configuration and software management, asset management, and system
provisioning.
Proxy Server:
Proxy Server:~ # zypper info Uyuni-Proxy-release
Loading repository data...
Reading installed packages...
Information for package Uyuni-Proxy-release:
--------------------------------------------
Repository : Uyuni Proxy Stable for openSUSE Leap 15.4 (x86_64)
Name : Uyuni-Proxy-release
Version : 2022.08-185.2.uyuni2
Arch : x86_64
Vendor : obs://build.opensuse.org/systemsmanagement:Uyuni
Support Level : Level 3
Installed Size : 1.4 KiB
Installed : Yes
Status : up-to-date
Source package : Uyuni-Proxy-release-2022.08-185.2.uyuni2.src
Summary : Uyuni Proxy
Description :
Uyuni Proxies extend large and/or geographically
dispersed Uyuni environments to reduce load on the Uyuni
Server, lower bandwidth needs, and provide faster local
updates.
Details about the issue
We have 2 Environments, one was upgraded a time ago from openSUSE Leap 15.3 to 15.4, and the other one was created from scratch on openSUSE 15.4, both masters are under the same version of Uyuni Server mentioned above, and the two of them have the same issue. When they fail, we need to restart the master spacewalk service, and this solves the issue temporarily. But the error is very annoying.
Also, from the Home > Overview on our two environments, we have been finding occasionally the Proxy Server inactive when it is on and reachable from the Uyuni Server and in reverse.
We think that this could be a product bug coming from the last release.
Log entries
catalina.2022-08-31.log
31-Aug-2022 12:19:36.668 INFO [main] org.apache.catalina.core.StandardServer.await A valid shutdown command was received via the shutdown port. Stopping the Server instance.
31-Aug-2022 12:19:36.669 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["ajp-nio-127.0.0.1-8009"]
31-Aug-2022 12:19:36.676 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["ajp-nio-0:0:0:0:0:0:0:1-8009"]
31-Aug-2022 12:19:42.972 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/9.0.36
31-Aug-2022 12:19:42.980 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Apr 7 2022 12:00:00 UTC
31-Aug-2022 12:19:42.981 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 9.0.36.0
31-Aug-2022 12:19:42.981 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux
31-Aug-2022 12:19:42.982 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 5.14.21-150400.24.18-default
31-Aug-2022 12:19:42.982 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
31-Aug-2022 12:19:42.982 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /usr/lib64/jvm/java-11-openjdk-11
31-Aug-2022 12:19:42.982 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 11.0.16+8-suse-150000.3.83.1-x8664
31-Aug-2022 12:19:42.982 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Oracle Corporation
31-Aug-2022 12:19:42.983 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /usr/share/tomcat
31-Aug-2022 12:19:42.983 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: /usr/share/tomcat
31-Aug-2022 12:19:43.010 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -ea
31-Aug-2022 12:19:43.010 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms256m
31-Aug-2022 12:19:43.011 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx1G
31-Aug-2022 12:19:43.011 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.awt.headless=true
31-Aug-2022 12:19:43.012 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.xml.sax.driver=com.redhat.rhn.frontend.xmlrpc.util.RhnSAXParser
31-Aug-2022 12:19:43.012 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.tomcat.util.http.Parameters.MAX_COUNT=1024
31-Aug-2022 12:19:43.012 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MaxNewSize=256
31-Aug-2022 12:19:43.013 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:-UseConcMarkSweepGC
31-Aug-2022 12:19:43.013 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dnet.sf.ehcache.skipUpdateCheck=true
31-Aug-2022 12:19:43.013 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-exports=java.annotation/javax.annotation.security=ALL-UNNAMED
31-Aug-2022 12:19:43.013 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.annotation/javax.annotation.security=ALL-UNNAMED
31-Aug-2022 12:19:43.014 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-modules=java.annotation,com.sun.xml.bind
31-Aug-2022 12:19:43.014 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/usr/share/tomcat
31-Aug-2022 12:19:43.014 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/usr/share/tomcat
31-Aug-2022 12:19:43.015 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.endorsed.dirs=
31-Aug-2022 12:19:43.015 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/var/cache/tomcat/temp
31-Aug-2022 12:19:43.015 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties
31-Aug-2022 12:19:43.015 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
31-Aug-2022 12:19:43.016 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded Apache Tomcat Native library [1.2.23] using APR version [1.6.3].
31-Aug-2022 12:19:43.016 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
31-Aug-2022 12:19:43.016 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
31-Aug-2022 12:19:43.024 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.1.1l 24 Aug 2021 SUSE release 150400.7.7.1]
31-Aug-2022 12:19:43.729 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["ajp-nio-127.0.0.1-8009"]
31-Aug-2022 12:19:43.788 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["ajp-nio-0:0:0:0:0:0:0:1-8009"]
31-Aug-2022 12:19:43.806 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-127.0.0.1-8080"]
31-Aug-2022 12:19:43.813 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [1,336] milliseconds
31-Aug-2022 12:19:43.937 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
31-Aug-2022 12:19:43.938 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.36]
31-Aug-2022 12:19:43.954 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/srv/tomcat/webapps/rhn]
31-Aug-2022 12:19:57.121 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
31-Aug-2022 12:20:27.665 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/srv/tomcat/webapps/rhn] has finished in [43,711] ms
31-Aug-2022 12:20:27.719 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-127.0.0.1-8009"]
31-Aug-2022 12:20:27.741 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-0:0:0:0:0:0:0:1-8009"]
31-Aug-2022 12:20:27.775 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-127.0.0.1-8080"]
31-Aug-2022 12:20:27.783 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [43,969] milliseconds
31-Aug-2022 12:20:31.439 INFO [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-2] org.directwebremoting.log.startup.logStartup Starting: DwrServlet v3.0.2-FINAL on Apache Tomcat/9.0.36 / JDK 11.0.16 from Oracle Corporation at /rhn
31-Aug-2022 12:20:31.798 INFO [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-2] org.directwebremoting.log.startup.resolveMultipleImplementations Starting: Using container abstraction org.directwebremoting.server.servlet2.Servlet24ContainerAbstraction
localhost_access_log.2022-08-31.txt
XX.XXX.XXX.XX - - [31/Aug/2022:12:11:42 -0400] "POST /rhn/manager/frontend-log HTTP/1.1" 200 16
127.0.0.1 - - [31/Aug/2022:12:11:42 -0400] "GET /rhn/websocket/notifications HTTP/1.1" 101 -
XX.XXX.XXX.XX - - [31/Aug/2022:12:20:30 -0400] "GET /rhn/systems/Overview.do HTTP/1.1" 200 97147
XX.XXX.XXX.XX - - [31/Aug/2022:12:20:32 -0400] "GET /rhn/dwr/interface/DWRItemSelector.js?cb=20220809112425 HTTP/1.1" 200 411
XX.XXX.XXX.XX - - [31/Aug/2022:12:20:32 -0400] "GET /rhn/dwr/util.js?cb=20220809112425 HTTP/1.1" 200 33731
XX.XXX.XXX.XX - - [31/Aug/2022:12:20:32 -0400] "GET /rhn/dwr/engine.js?cb=20220809112425 HTTP/1.1" 200 53126
XX.XXX.XXX.XX - - [31/Aug/2022:12:20:34 -0400] "POST /rhn/manager/frontend-log HTTP/1.1" 200 16
127.0.0.1 - - [31/Aug/2022:12:20:34 -0400] "GET /rhn/websocket/notifications HTTP/1.1" 101 -
127.0.0.1 - - [31/Aug/2022:12:20:34 -0400] "GET /rhn/websocket/notifications HTTP/1.1" 101 -
YY.YYY.YYY.YY - - [31/Aug/2022:12:20:35 -0400] "POST /rhn/rpc/api HTTP/1.1" 200 128
rhn_web_ui.log
2022-08-31 12:20:07,333 [main] WARN org.hibernate.cfg.AnnotationBinder - HHH000457: Joined inheritance hierarchy [com.redhat.rhn.domain.image.ImageProfile] defined explicit @DiscriminatorColumn. Legacy Hibernate behavior was to ignore the @DiscriminatorColumn. However, as part of issue HHH-6911 we now apply the explicit @DiscriminatorColumn. If you would prefer the legacy behavior, enable the `hibernate.discriminator.ignore_explicit_for_joined` setting (hibernate.discriminator.ignore_explicit_for_joined=true)
Thanks a lot, in advance!
I have the same problem with version 2022.08. Restarting the service tomcat temporarily fixes the problem.
how many systems are registered on your server? There are known issues with the performance of those pages... not sure if this could be related. I am currently working on optimizing these, I hope this could address the issue: https://github.com/uyuni-project/uyuni/pull/5889
At the moment I have about 150 servers registered on server.
OK, that is not the same issue then, the order of magnitude I'm referring to is 10000.
One of the systems that we have, has 158 clients already registered (most of them are registered to a Proxy Server) but will come around 500 more. They are CentOS 7, 2 openSUSE Leap 15.4, and around 4 Amazon Linux 2. But we haven't seen that error in the past, it just came from the last upgrade we did I think.
Today, within 1 hour, we restarted the tomcat service 5 times. It's very hard to use in this version. I am more and more inclined to go back to an older version. I am not able to identify any errors in the tomcat logs.
Unfortunately the same thing here. I was registering more clients to the system today, and when I was grouping them into the appropriate groups, the system became unresponsive around 5 times, and we had to restart the tomcat service each time to move further with the activity planned for today. I was digging into the logs, and apart from those details already provided when I opened this issue, I haven't found anything else.
I have another Lab environment that is working everything fine, but the difference in this lab is that the Uyuni System is still under version 2022-06.
I noticed too, that when there is more than one user logged in to the Web Portal, the error seems to be more frequent.
I can confirm the same issue after Suse Manager update (4.3.0 -> 4.3.1). Scrolling Systems List or searching for systems ends with timeout on all subsequent connections to :443. No errors in log files. Only spacewalk-service restart possible at that moment. There is no memory/CPU/io pressure on the system during that events.
do you guys have long running SQL queries? You can figure out by looking into the result of:
echo "select * from pg_stat_activity;" | spacewalk-sql --select-mode-direct -
HI @cbosdo, I used that command yesterday, and I wonder what are those idle in transaction messages in the output? Please see the details below:
Uyuni-Server:~ # echo "select * from pg_stat_activity;" | spacewalk-sql --select-mode-direct -
Prototype mismatch: sub Spacewalk::Setup::idn_to_ascii: none vs ($;$$) at /usr/lib/perl5/5.26.1/Exporter.pm line 66.
at /usr/lib/perl5/vendor_perl/5.26.1/Spacewalk/Setup.pm line 29.
datid | datname | pid | leader_pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | xact_start | query_start | state_change | wait_event_type | wait_event | state | backend_xid | backend_xmin | query_id | query | backend_type
-------+---------+-------+------------+----------+----------+------------------------+-------------+-----------------+-------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------+----------------------+---------------------+-------------+--------------+----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------
| | 1841 | | | | | | | | 2022-08-19 20:43:52.475282-04 | | | | Activity | AutoVacuumMain | | | | | | autovacuum launcher
| | 1844 | | 10 | postgres | | | | | 2022-08-19 20:43:52.478018-04 | | | | Activity | LogicalLauncherMain | | | | | | logical replication launcher
16384 | uyuni | 29228 | | 16385 | uyuni | | ::1 | | 33272 | 2022-09-09 12:53:31.760003-04 | | 2022-09-14 17:00:01.760455-04 | 2022-09-14 17:00:01.765415-04 | Client | ClientRead | idle | | | | COMMIT | client backend
16384 | uyuni | 50920 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 35114 | 2022-09-14 17:39:13.519132-04 | | 2022-09-14 17:43:00.7189-04 | 2022-09-14 17:43:00.718953-04 | Client | ClientRead | idle | | | | select 'c3p0 ping' from dual | client backend
16384 | uyuni | 50921 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 35120 | 2022-09-14 17:39:13.520336-04 | | 2022-09-14 17:43:00.682471-04 | 2022-09-14 17:43:00.686213-04 | Client | ClientRead | idle | | | | COMMIT | client backend
16384 | uyuni | 51129 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 52614 | 2022-09-14 17:39:31.048355-04 | | 2022-09-14 17:39:31.106981-04 | 2022-09-14 17:39:31.107044-04 | Client | ClientRead | idle | | | | SHOW TRANSACTION ISOLATION LEVEL | client backend
16384 | uyuni | 47193 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 39968 | 2022-09-14 17:34:13.51546-04 | | 2022-09-14 17:39:12.719375-04 | 2022-09-14 17:39:12.719863-04 | Client | ClientRead | idle | | | | select 'c3p0 ping' from dual | client backend
16384 | uyuni | 53941 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 53658 | 2022-09-14 17:43:16.04865-04 | | 2022-09-14 17:43:16.105661-04 | 2022-09-14 17:43:16.105746-04 | Client | ClientRead | idle | | | | SHOW TRANSACTION ISOLATION LEVEL | client backend
16384 | uyuni | 31493 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 34098 | 2022-09-09 12:54:10.198655-04 | | 2022-09-14 09:29:08.935528-04 | 2022-09-14 09:29:08.935546-04 | Client | ClientRead | idle | | | | COMMIT | client backend
16384 | uyuni | 31467 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 34096 | 2022-09-09 12:54:09.244498-04 | | 2022-09-13 00:04:08.933966-04 | 2022-09-13 00:04:08.934059-04 | Client | ClientRead | idle | | | | COMMIT | client backend
16384 | uyuni | 31465 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 34076 | 2022-09-09 12:54:09.242235-04 | | 2022-09-14 09:29:08.935522-04 | 2022-09-14 09:29:08.93563-04 | Client | ClientRead | idle | | | | COMMIT | client backend
16384 | uyuni | 31466 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 34086 | 2022-09-09 12:54:09.243371-04 | | 2022-09-09 12:54:10.534374-04 | 2022-09-09 12:54:10.534387-04 | Client | ClientRead | idle | | | | COMMIT | client backend
16384 | uyuni | 31495 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 34116 | 2022-09-09 12:54:10.200775-04 | | 2022-09-14 17:39:08.948763-04 | 2022-09-14 17:39:08.948781-04 | Client | ClientRead | idle | | | | COMMIT | client backend
16384 | uyuni | 31494 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 34104 | 2022-09-09 12:54:10.199601-04 | | 2022-09-14 17:39:08.946817-04 | 2022-09-14 17:39:08.946853-04 | Client | ClientRead | idle | | | | COMMIT | client backend
16384 | uyuni | 31502 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 34134 | 2022-09-09 12:54:10.417121-04 | | 2022-09-14 17:39:08.935199-04 | 2022-09-14 17:39:08.935214-04 | Client | ClientRead | idle | | | | COMMIT | client backend
16384 | uyuni | 31501 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 34130 | 2022-09-09 12:54:10.417861-04 | | 2022-09-14 09:29:08.935557-04 | 2022-09-14 09:29:08.935655-04 | Client | ClientRead | idle | | | | COMMIT | client backend
16384 | uyuni | 31503 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 34136 | 2022-09-09 12:54:10.428761-04 | | 2022-09-14 09:29:08.934442-04 | 2022-09-14 09:29:08.93458-04 | Client | ClientRead | idle | | | | COMMIT | client backend
16384 | uyuni | 31514 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 34152 | 2022-09-09 12:54:10.536958-04 | | 2022-09-13 00:04:08.933192-04 | 2022-09-13 00:04:08.933207-04 | Client | ClientRead | idle | | | | COMMIT | client backend
16384 | uyuni | 64188 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 59260 | 2022-09-13 19:04:30.904495-04 | 2022-09-13 19:04:52.228231-04 | 2022-09-13 19:04:52.269092-04 | 2022-09-13 19:04:52.269128-04 | Client | ClientRead | idle in transaction | | 1937961 | | select roleimpl0_.id as id1_238_, roleimpl0_.name as name2_238_, roleimpl0_.label as label3_238_, roleimpl0_.created as created4_238_, roleimpl0_.modified as modified5_238_ from RHNUSERGROUPTYPE roleimpl0_ where roleimpl0_.label=$1 | client backend
16384 | uyuni | 31517 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 34164 | 2022-09-09 12:54:10.546913-04 | | 2022-09-14 17:39:08.934872-04 | 2022-09-14 17:39:08.93489-04 | Client | ClientRead | idle | | | | COMMIT | client backend
16384 | uyuni | 31516 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 34162 | 2022-09-09 12:54:10.542871-04 | | 2022-09-14 17:39:08.935714-04 | 2022-09-14 17:39:08.935728-04 | Client | ClientRead | idle | | | | COMMIT | client backend
16384 | uyuni | 31574 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 34166 | 2022-09-09 12:54:10.646266-04 | | 2022-09-09 12:54:10.873855-04 | 2022-09-09 12:54:10.873917-04 | Client | ClientRead | idle | | | | SHOW TRANSACTION ISOLATION LEVEL | client backend
16384 | uyuni | 31648 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 34180 | 2022-09-09 12:54:10.812863-04 | | 2022-09-14 17:39:08.93569-04 | 2022-09-14 17:39:08.935714-04 | Client | ClientRead | idle | | | | COMMIT | client backend
16384 | uyuni | 31655 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 34184 | 2022-09-09 12:54:10.878173-04 | | 2022-09-14 15:34:08.935308-04 | 2022-09-14 15:34:08.935397-04 | Client | ClientRead | idle | | | | COMMIT | client backend
16384 | uyuni | 31663 | | 16385 | uyuni | | | | -1 | 2022-09-09 12:54:11.46872-04 | | 2022-09-14 17:43:22.193927-04 | 2022-09-14 17:43:22.193998-04 | Client | ClientRead | idle | | | | COMMIT | client backend
16384 | uyuni | 52979 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 48292 | 2022-09-14 17:42:01.049739-04 | | 2022-09-14 17:42:01.115235-04 | 2022-09-14 17:42:01.115341-04 | Client | ClientRead | idle | | | | SHOW TRANSACTION ISOLATION LEVEL | client backend
16384 | uyuni | 47194 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 39978 | 2022-09-14 17:34:13.516827-04 | | 2022-09-14 17:39:12.719445-04 | 2022-09-14 17:39:12.719735-04 | Client | ClientRead | idle | | | | select 'c3p0 ping' from dual | client backend
16384 | uyuni | 64200 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 59266 | 2022-09-13 19:04:31.741935-04 | 2022-09-13 19:04:51.308265-04 | 2022-09-14 17:43:29.32665-04 | 2022-09-14 17:43:29.330764-04 | Client | ClientRead | idle in transaction | | 1993231 | | SELECT queue, +| client backend
| | | | | | | | | | | | | | | | | | | | COUNT(*) as count +|
| | | | | | | | | | | | | | | | | | | | FROM suseSaltEvent +|
| | | | | | | | | | | | | | | | | | | | GROUP BY queue |
16384 | uyuni | 40419 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 39384 | 2022-09-10 01:19:12.565541-04 | 2022-09-10 01:30:00.405024-04 | 2022-09-10 01:30:00.406162-04 | 2022-09-10 01:30:00.406181-04 | Client | ClientRead | idle in transaction | | 1720012 | | select mgrserveri0_.server_id as server_i1_289_, mgrserveri0_.mgr_evr_id as mgr_evr_2_289_, mgrserveri0_.report_db_name as report_d3_289_, mgrserveri0_.report_db_host as report_d4_289_, mgrserveri0_.report_db_port as report_d5_289_, mgrserveri0_.report_db_last_synced as report_d6_289_, mgrserveri0_.report_db_cred_id as report_d7_289_ from suseMgrServerInfo mgrserveri0_ | client backend
16384 | uyuni | 64343 | | 16385 | uyuni | pgjdbc-ng | 127.0.0.1 | | 42492 | 2022-09-13 19:04:50.090161-04 | | 2022-09-14 17:43:29.325686-04 | 2022-09-14 17:43:29.326002-04 | Client | ClientRead | idle | | | | SELECT 'salt-event-connection-watchdog' | client backend
16384 | uyuni | 32057 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 39178 | 2022-09-09 12:54:22.974039-04 | | 2022-09-14 17:43:24.742225-04 | 2022-09-14 17:43:24.742267-04 | Client | ClientRead | idle | | | | COMMIT | client backend
16384 | uyuni | 32056 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 39176 | 2022-09-09 12:54:22.972997-04 | | 2022-09-14 17:41:00.698272-04 | 2022-09-14 17:41:00.702556-04 | Client | ClientRead | idle | | | | COMMIT | client backend
16384 | uyuni | 32058 | | 16385 | uyuni | PostgreSQL JDBC Driver | 127.0.0.1 | | 39190 | 2022-09-09 12:54:22.975208-04 | | 2022-09-14 17:43:00.671996-04 | 2022-09-14 17:43:00.676164-04 | Client | ClientRead | idle | | | | COMMIT | client backend
16384 | uyuni | 54161 | | 16385 | uyuni | psql | ::1 | | 41894 | 2022-09-14 17:43:30.681985-04 | 2022-09-14 17:43:30.73171-04 | 2022-09-14 17:43:30.73171-04 | 2022-09-14 17:43:30.731714-04 | | | active | | 1993231 | | select * from pg_stat_activity; | client backend
| | 1839 | | | | | | | | 2022-08-19 20:43:52.473728-04 | | | | Activity | BgWriterMain | | | | | | background writer
| | 1842 | | | | | | | | 2022-08-19 20:43:52.476186-04 | | | | Activity | ArchiverMain | | | | | | archiver
| | 1838 | | | | | | | | 2022-08-19 20:43:52.472985-04 | | | | Timeout | CheckpointWriteDelay | | | | | | checkpointer
| | 1840 | | | | | | | | 2022-08-19 20:43:52.474562-04 | | | | Activity | WalWriterMain | | | | | | walwriter
(38 rows)
Uyuni-Server:~ #
As I mentioned before, I am not sure if the error only appears when there is more than 1 User logged in to the Web Portal, we have been doing other registrations today only with 1 User logged in, and the error didn't appear during the activity. I don't know if this could be a clue for your review.
Also, we have patched the Proxy Servers in the 2 environments, and the activity went just fine. We also tried to patch the Uyuni Servers, but we didn't receive a new Uyuni-Server-release for upgrading the system.
Thanks a lot, for your support!
I can confirm the same issue after Suse Manager update (4.3.0 -> 4.3.1). Scrolling Systems List or searching for systems ends with timeout on all subsequent connections to :443. No errors in log files. Only spacewalk-service restart possible at that moment. There is no memory/CPU/io pressure on the system during that events.
@kurld, Do you mean the SUSE Manager system installed on SLES, or Uyuni on openSUSE Leap instead? Thanks
If anyone wants to help testing the PR linked above I can help you build it. Feedback would be appreciated
@cbosdo, I would like to help, but I don't have a test environment with that version for this purpose. The ones that are failing are Production Environments.
Hi @cbosdo, What fix? Do we need to upgrade to a new release to receive it?
Hi @cbosdo, What fix? Do we need to upgrade to a new release to receive it?
The fix is the PR linked a few days earlier in this issue: it has been reported to work for several customers. It will also be in the next release... but I have no idea when that one will be since there is a severe issue with zypper.
2022.10 will receive it, but to release 2022.10 we need the fix for zypper, as right now --allow-vendor-change is broken, and we need it for the update to 2022.10.
Hopefully we'll be have the release of the zypper fix today/tomorrow, and then the Uyuni release should happen before end of the week.
Thanks a lot, guys for all the hard work and your comments. Hopefully, we could coordinate the patching to the new release for the next week. 😉
Hi @cbosdo @juliogonzalez, guys. We just have upgraded the systems to the version 2022.10, and now they are getting the following error
Both environments are unreachable because of that. We followed the Official documentation guide below:
https://www.uyuni-project.org/uyuni-docs/en/uyuni/installation-and-upgrade/server-minor-upgrade-uyuni.html
But the error is exactly the same on both environments.
rhn_taskomatic_daemon.log:
2022-10-18 17:31:26,722 [Thread-41] FATAL com.redhat.rhn.taskomatic.core.TaskomaticDaemon - javax/persistence/PersistenceException
2022-10-18 17:31:29,416 [Thread-41] FATAL com.redhat.rhn.taskomatic.core.TaskomaticDaemon - javax/persistence/PersistenceException
2022-10-18 17:31:31,842 [Thread-41] FATAL com.redhat.rhn.taskomatic.core.TaskomaticDaemon - javax/persistence/PersistenceException
2022-10-18 17:31:34,505 [Thread-41] FATAL com.redhat.rhn.taskomatic.core.TaskomaticDaemon - javax/persistence/PersistenceException
2022-10-18 17:31:37,375 [Thread-41] FATAL com.redhat.rhn.taskomatic.core.TaskomaticDaemon - javax/persistence/PersistenceException
2022-10-18 17:31:39,957 [Thread-41] FATAL com.redhat.rhn.taskomatic.core.TaskomaticDaemon - javax/persistence/PersistenceException
2022-10-18 17:31:42,585 [Thread-41] FATAL com.redhat.rhn.taskomatic.core.TaskomaticDaemon - javax/persistence/PersistenceException
2022-10-18 17:31:45,216 [Thread-41] FATAL com.redhat.rhn.taskomatic.core.TaskomaticDaemon - javax/persistence/PersistenceException
2022-10-18 17:31:47,780 [Thread-41] FATAL com.redhat.rhn.taskomatic.core.TaskomaticDaemon - javax/persistence/PersistenceException
2022-10-18 17:31:50,330 [Thread-41] FATAL com.redhat.rhn.taskomatic.core.TaskomaticDaemon - javax/persistence/PersistenceException
2022-10-18 17:31:52,880 [Thread-41] FATAL com.redhat.rhn.taskomatic.core.TaskomaticDaemon - javax/persistence/PersistenceException
2022-10-18 17:31:55,344 [Thread-41] FATAL com.redhat.rhn.taskomatic.core.TaskomaticDaemon - javax/persistence/PersistenceException
2022-10-18 17:31:57,837 [Thread-41] FATAL com.redhat.rhn.taskomatic.core.TaskomaticDaemon - javax/persistence/PersistenceException
2022-10-18 17:32:00,307 [Thread-41] FATAL com.redhat.rhn.taskomatic.core.TaskomaticDaemon - javax/persistence/PersistenceException
2022-10-18 17:32:02,968 [Thread-41] FATAL com.redhat.rhn.taskomatic.core.TaskomaticDaemon - javax/persistence/PersistenceException
2022-10-18 17:32:05,259 [Thread-41] FATAL com.redhat.rhn.taskomatic.core.TaskomaticDaemon - javax/persistence/PersistenceException
2022-10-18 17:32:08,201 [Thread-41] FATAL com.redhat.rhn.taskomatic.core.TaskomaticDaemon - javax/persistence/PersistenceException
2022-10-18 17:32:10,944 [Thread-41] FATAL com.redhat.rhn.taskomatic.core.TaskomaticDaemon - javax/persistence/PersistenceException
2022-10-18 17:32:13,434 [Thread-41] FATAL com.redhat.rhn.taskomatic.core.TaskomaticDaemon - javax/persistence/PersistenceException
catalina.2022-10-18.log:
18-Oct-2022 17:26:18.091 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/srv/tomcat/webapps/rhn]
18-Oct-2022 17:26:19.164 SEVERE [main] org.apache.catalina.startup.HostConfig.deployDirectory Error deploying web application directory [/srv/tomcat/webapps/rhn]
java.lang.IllegalStateException: Error starting child
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:720)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:705)
at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1132)
at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1865)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1044)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:429)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1575)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:309)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:936)
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:841)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909)
at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardService.startInternal(StandardService.java:421)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.startup.Catalina.start(Catalina.java:633)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:343)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:474)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/rhn]]
at org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:440)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:198)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:717)
... 37 more
Caused by: java.lang.NullPointerException
at org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:382)
at org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:195)
at org.apache.catalina.startup.ContextConfig.processJarsForWebFragments(ContextConfig.java:1971)
at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1129)
at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:775)
at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:301)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5044)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
... 38 more
18-Oct-2022 17:26:19.174 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/srv/tomcat/webapps/rhn] has finished in [1,082] ms
18-Oct-2022 17:26:19.184 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-127.0.0.1-8009"]
18-Oct-2022 17:26:19.205 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-0:0:0:0:0:0:0:1-8009"]
18-Oct-2022 17:26:19.215 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-127.0.0.1-8080"]
18-Oct-2022 17:26:19.235 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [1,329] milliseconds
This seems to me like a missing jar in the installation, but not related to this bug: the whole tomcat application is failing to start here.
Please, could you re-open the this case again? Thanks
Better open a new bug since this is completely unrelated to the original issue
Both environments are unreachable because of that. We followed the Official documentation guide below:
https://www.uyuni-project.org/uyuni-docs/en/uyuni/installation-and-upgrade/server-minor-upgrade-uyuni.html
I think this is the problem.
This time, the procedure is a bit different. Make sure you read the release notes.
I think the problem was regarding the zypper didn't pull all the required packages for the upgrade. We saw the following on a zypper up
Uyuni-Server:/var/log/tomcat # zypper up
Loading repository data...
Reading installed packages...
The following 27 package updates will NOT be installed:
cobbler dom4j golang-github-QubitProducts-exporter_exporter hibernate-types netty objectweb-asm pgjdbc-ng prometheus-postgres_exporter spacewalk-backend spacewalk-backend-app spacewalk-backend-applet
spacewalk-backend-config-files spacewalk-backend-config-files-common spacewalk-backend-config-files-tool spacewalk-backend-iss spacewalk-backend-iss-export spacewalk-backend-package-push-server
spacewalk-backend-server spacewalk-backend-sql spacewalk-backend-sql-postgresql spacewalk-backend-tools spacewalk-backend-xml-export-libs spacewalk-backend-xmlrpc spacewalk-java spacewalk-java-lib
spacewalk-setup spacewalk-taskomatic
Nothing to do.
But we could figure out the issue by forcing zypper to install all of those packages manually:
Uyuni-Server:~ # zypper in cobbler dom4j golang-github-QubitProducts-exporter_exporter hibernate-types netty objectweb-asm pgjdbc-ng prometheus-postgres_exporter spacewalk-backend spacewalk-backend-app spacewalk-backend-applet \
> spacewalk-backend-config-files spacewalk-backend-config-files-common spacewalk-backend-config-files-tool spacewalk-backend-iss spacewalk-backend-iss-export spacewalk-backend-package-push-server \
> spacewalk-backend-server spacewalk-backend-sql spacewalk-backend-sql-postgresql spacewalk-backend-tools spacewalk-backend-xml-export-libs spacewalk-backend-xmlrpc spacewalk-java spacewalk-java-lib \
> spacewalk-setup spacewalk-taskomatic
After the packages above were installed, the spacewalk-service restarted on both Uyuni Masters, the issue with the Web Portal was resolved, and the Uyuni Servers were migrated to version 2022.10 as was expected.
As I said, this time the procedure is a bit different. As the release notes explain, zypper up is not enough.
Make sure you follow the instructions, just in case. I am not sure zypper in and the list the packages left everything as it should be.
@juliogonzalez, The release notes from version 2022.10 are not clear, even is not mentioned version 2022.08 in them. They are more referred to as older versions on openSUSE Leap 15.3. But because both environments were already patched and we haven't found errors at this time, I don't want to touch them until the next patching cycle.
Let's cruise the fingers do not find any errors on the systems already upgraded.
What do you mean 2022.08 is not mentioned? I see it at https://www.uyuni-project.org/doc/2022.10/release-notes-uyuni-server.html#_version_2022_08
While the instructions for the special steps are at https://www.uyuni-project.org/doc/2022.10/release-notes-uyuni-server.html#_update_notes