uyuni
uyuni copied to clipboard
uyuni-project
After upgrade from 2024.12 to 2025.05 existing groups lose association with configuration channels for recurring highstate
Problem description
- After upgrade from 2024.12 to 2025.05 existing groups lose association with configuration channels for recurring highstate.
- Also no ability to modify configuration channel assignment for existing groups.
Steps to reproduce
- Create group
- Create recurring highstate task for the group
- Assign configuration channels to the group's highstate task
- Upgrade 2024.12 to 2025.05
- Note that existing groups lose association with configuration channels for recurring highstate
- Note no ability to modify configuration channel assignment for existing groups
Uyuni version
Information for pattern uyuni_server:
-------------------------------------
Repository : @System
Name : uyuni_server
Version : 2025.05-241000.1.1.uyuni5
Arch : x86_64
Vendor : obs://build.opensuse.org/systemsmanagement:Uyuni
Installed : Yes (automatically)
Visible to User : Yes
Summary : Uyuni Server
Uyuni proxy version (if used)
N/A
uyuni-tools version used to deploy/manage the server or proxy (if relevant)
$ mgradm -v
mgradm version 5.1.11 for latest image (Master 93b5460) (compilation tag: %{_uyuni_tools_tags})
$ mgrctl -v
mgrctl version 5.1.11 for latest image (Master 93b5460) (compilation tag: %{_uyuni_tools_tags})
Useful logs
Additional information
Pre-upgrade
-
Pre-upgrade (2024.12) recurring tasks for group LAB; note the Highstate
-
Pre-upgrade (2024.12) config channels for group LAB
-
Pre-upgrade (2024.12) highstates for minions displayed at group level for group LAB
-
Pre-upgrade (2024.12) highstate for specific minion in group TPSF-GW
Post-upgrade
-
Post-upgrade (2024.12) recurring tasks for group LAB displayed properly; note the Highstate
-
Post-upgrade (2025.05) group LAB has lost association with configuration channels and now also no ability assign configuration channels
-
Post-upgrade (2025.05) missing highstates for minions displayed at group level for group LAB (hangs on "Retrieving highstate summary...")
-
Post-upgrade (2025.05) missing highstate for specific minion in group TPSF-GW (hangs on "Retrieving highstate summary...")
I think that this could be caused by the issue fixed on the pr: https://github.com/uyuni-project/uyuni/pull/10466 @nadvornik could you check if that is the case? @admd for awareness.
I didn't see this before I submitted https://github.com/uyuni-project/uyuni/issues/10485 This sounds similar to my problems and likely the same root cause.
EDIT - Nottrue: ~~At least for mine, I seem to have have resolved the issue myself with a chmod. ~~
I got machines added since the upgrade being able to manage conf chans and highstate, but existing machines remain problematic with errors like this when highstating;
Error: Could not parse state file. Please check YAML syntax.
- Specified SLS custom.custom_2fb0f5eaef1e066992ed17f76091565a in saltenv base is not available on the salt master or through a configured fileserver
- Specified SLS custom.group_9 in saltenv base is not available on the salt master or through a configured fileserver
- Specified SLS custom.group_25 in saltenv base is not available on the salt master or through a configured fileserver
- Specified SLS custom.org_1 in saltenv base is not available on the salt master or through a configured fileserver
I think that this could be caused by the issue fixed on the pr: #10466 @nadvornik could you check if that is the case? @admd for awareness.
@rjmateus Is there a way to fix this issue after upgrade to 2025.05? Or is the only option to skip 2025.05 and wait until the next Uyuni release?
Thanks. It would be really useful to get some guidance from the devs on how to deal with this - we currently have a broken uyuni.
In 2025.05 this will be fixed automatically. If you cannot wait, the safest option is to re-create the groups and reassign all data.
If you want to experiment with the DB, you can try to run this query on the database, which will update the position (which is the cause of this issue).
UPDATE rhnServerConfigChannel
SET position = (
SELECT rank
from (
SELECT server_id, config_channel_id, position, (row_number() OVER (PARTITION BY server_id ORDER BY position ASC)) rank
FROM rhnServerConfigChannel sscc
) ranking
WHERE config_channel_id = rhnServerConfigChannel.config_channel_id
AND server_id = rhnServerConfigChannel.server_id) - 1;
In 2025.05 this will be fixed automatically. If you cannot wait, the safest option is to re-create the groups and reassign all data.
@rjmateus I am confused. This issue occurred after upgrading to 2025.05. Has the fix been backported into the latest 2025.05 container image?
If you want to experiment with the DB, you can try to run this query on the database, which will update the position (which is the cause of this issue).
UPDATE rhnServerConfigChannel SET position = ( SELECT rank from ( SELECT server_id, config_channel_id, position, (row_number() OVER (PARTITION BY server_id ORDER BY position ASC)) rank FROM rhnServerConfigChannel sscc ) ranking WHERE config_channel_id = rhnServerConfigChannel.config_channel_id AND server_id = rhnServerConfigChannel.server_id) - 1;
@rjmateus What is the syntax to run this query from the command line? I am going to try it in our LAB environment.
@gabjef sorry for the confusion, it will BE fix on the versions next to 2025.05, which was the version that introduced the issue.
To run the script you can use the tool 'mgrctl support sql -i' and then past the SQL I shared (I didn't check the command since I typing from my phone)
I tried this (Small change; it's mgradm support rather than mgrctl
But sadly this SQL did not fix my problem.
I am still unable to get a server to join or leave any configuration channels.
Any idea when the next version is due? I think I'm going to have to restore a very old backup of Uyuni and spend some hours bringing it up to date. :/
Recreating 37 config channels and re-adding 197 files and re-assigning 202 systems to a various mix of those is not a task to take on lightly.
Additional: Creating a new config channel worked, but Uyuni crashed when adding a system to it.
As I can't remove any existing channels from a system without it crashing, I don't know how we could proceed with this.
I tried this (Small change; it's
mgradm supportrather thanmgrctlBut sadly this SQL did not fix my problem.
The SQL query did not work for us either.
I am still unable to get a server to join or leave any configuration channels.
Any idea when the next version is due? I think I'm going to have to restore a very old backup of Uyuni and spend some hours bringing it up to date. :/
Recreating 37 config channels and re-adding 197 files and re-assigning 202 systems to a various mix of those is not a task to take on lightly.
I concur.
We have restored our LAB instance from a VM snapshot and are on-hold pending release of the next version.
@digdilem-work @gabjef Looks like I had provide the update for One table and forgot about another One. Tomorrow I Will have a look and provide another query, if you want to take another shot.
@rjmateus - that would be most gratefully received. We're at the point of rolling back and losing a month's work so yes, I'd be happy to try again.
@digdilem-work can you run the following sql command and report back if your issue is solved:
UPDATE suseStateRevisionConfigChannel
SET position = (
SELECT rank - 1
FROM (
SELECT state_revision_id, config_channel_id, position, (row_number() OVER (PARTITION BY state_revision_id ORDER BY position ASC)) rank
FROM suseStateRevisionConfigChannel scc
) ranking
WHERE config_channel_id = suseStateRevisionConfigChannel.config_channel_id
AND state_revision_id = suseStateRevisionConfigChannel.state_revision_id);
@digdilem-work can you run the following sql command and report back if your issue is solved:
UPDATE suseStateRevisionConfigChannel SET position = ( SELECT rank - 1 FROM ( SELECT state_revision_id, config_channel_id, position, (row_number() OVER (PARTITION BY state_revision_id ORDER BY position ASC)) rank FROM suseStateRevisionConfigChannel scc ) ranking WHERE config_channel_id = suseStateRevisionConfigChannel.config_channel_id AND state_revision_id = suseStateRevisionConfigChannel.state_revision_id);
@rjmateus Thanks for helping out with this; we highly appreciate it.
Question: Do we try only this query? Or this one and the one above?
@gabjef this last One should be enough for this error. Let me know if it fixes the issue. On Next uyuni versions this problem Will fix automatically
Thanks for providing the query, this fixed the issue for our deployment as well.
Unfortunately this fix did not work for us.
Specifically, on 195 machines it does not work. 3 machines it does.
It does work for machines that were onboarded to Uyuni after it was upgraded to 2025.06, but not for the machines that already existed.
I suspect this is because we were unknowingly running the "broken" version of Uyuni in production for 2 or 3 weeks before encountering this issue. Possibly if we reverted to an old version and then upgraded to 2025.06 and immediately applied this fix, as I believe @rjmateus and @MaKraMc have done(?) and reported success with, it would be fine.
(I'm testing this by adding machines via webui to any config channel. At the stage where ranking is selected, Uyuni crashes immediately when clicking "Update channel rankings")
At this stage, it seems our best course is to revert to the previous version in a backup made early May and 7 weeks of config changes and need to re-onboard 3 machines, and either upgrade and apply the fix immediately, or wait for the next release.
Which seems better?
Thank you
We've got a web traceback log in case it's useful
Caused by: java.lang.IllegalArgumentException: Trying to add/subscribe a NON-CENTRALLY-MANAGED Channel to a central channels list.
The following exception occurred while executing this request:
POST /rhn/systems/details/configuration/RankChannels.do
Date:7/1/25 9:48:25 AM BST
Headers:
host: uyuni-server
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:140.0) Gecko/20100101 Firefox/140.0
accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
accept-language: en-US,en;q=0.5
accept-encoding: gzip, deflate, br, zstd
content-type: application/x-www-form-urlencoded
content-length: 218
Origin: https://uyuni-server
connection: keep-alive
referer: https://uyuni-server/rhn/systems/details/configuration/RankChannels.do?wizard_mode=true&sid=1000010341
cookie: rl_anonymous_id=RudderEncrypt%3AU2FsdGVkX1%2Bk1suLUm5H%2BsHfHV3%2FXnF093s9g7oQYpXo9U%2B%2BfvyDxMRRwGd4PFnBAy5EiA27CBbWWo4v1xOzGg%3D%3D; rl_user_id=%22RudderEncrypt%3AU2FsdGVkX1%2BuonfjXAv2dChqzZkDQTaTH3kL2r5g1gKotQoyNtrFj2qiWsVbdzB2%22; rl_trait=RudderEncrypt%3AU2FsdGVkX1%2B61hY4qcsOoceGajnE3VHTnbPOAy9SL5Y%3D; JSESSIONID=716D926BBE3CC76B6E12F4907976C78C; pxt-session-cookie=2613870x66616dc649d6b6e4622d1c6f6890595669c7d4d26792f4bf51d103dbd1e10a87
Upgrade-Insecure-Requests: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Priority: u=0, i
Request:
Local Name = uyuni-server
Server Name = uyuni-server
Requested Session Id came from Cookie
Requested Session Valid = true
Session = org.apache.catalina.session.StandardSessionFacade@4976abca[session=StandardSession[716D926BBE3CC76B6E12F4907976C78C]]
Protocol = https
Request Locale = en_US
Request Character Encoding = UTF-8
Attribute Names = rhnActiveLang, org.apache.struts.action.MESSAGE, org.apache.struts.action.mapping.instance, org.apache.tomcat.util.net.secure_protocol_version, requestedUri, session, javax.servlet.request.key_size, org.apache.struts.action.MODULE, channelRanksForm, javax.servlet.request.cipher_suite, javax.servlet.request.ssl_session_id, __sitemesh__filterapplied, system, javax.servlet.jsp.jstl.fmt.timeZone.request,
Form Variables:
sid: 1000010341
csrf_token: -4061417367327858009
submitted: true
wizard_mode: true
rankedValues: 24,9,40,4,41,43,45,46,48,139,250,252
dispatch: Update Channel Rankings
dispatcher: Update Channel Rankings
User Information:
User simon (id 1, org_id 1)
Exception:
javax.servlet.ServletException: java.lang.IllegalArgumentException: Trying to add/subscribe a NON-CENTRALLY-MANAGED Channel to a central channels list.
at org.apache.struts.action.RequestProcessor.processException(RequestProcessor.java:535)
at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:433)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:237)
at com.redhat.rhn.frontend.struts.RhnRequestProcessor.process(RhnRequestProcessor.java:103)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:555)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:623)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:199)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
at com.redhat.rhn.frontend.servlets.SystemDetailsMessageFilter.doFilter(SystemDetailsMessageFilter.java:57)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
at com.redhat.rhn.frontend.servlets.AuthorizationFilter.doFilter(AuthorizationFilter.java:101)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
at com.redhat.rhn.frontend.servlets.AuthenticationFilter.doFilter(AuthenticationFilter.java:108)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:142)
at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:58)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
at com.redhat.rhn.frontend.servlets.LocalizedEnvironmentFilter.doFilter(LocalizedEnvironmentFilter.java:71)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
at com.redhat.rhn.frontend.servlets.EnvironmentFilter.doFilter(EnvironmentFilter.java:97)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
at com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:59)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
at com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:98)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:656)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346)
at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:424)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:935)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1792)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1189)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:658)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.IllegalArgumentException: Trying to add/subscribe a NON-CENTRALLY-MANAGED Channel to a central channels list.
at com.redhat.rhn.domain.config.ConfigChannelListProcessor.check(ConfigChannelListProcessor.java:40)
at com.redhat.rhn.domain.config.ConfigChannelListProcessor.add(ConfigChannelListProcessor.java:73)
at com.redhat.rhn.domain.config.ConfigChannelListProcessor.add(ConfigChannelListProcessor.java:61)
at com.redhat.rhn.domain.server.Server.lambda$subscribeConfigChannels$0(Server.java:449)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.redhat.rhn.domain.server.Server.subscribeConfigChannels(Server.java:449)
at com.redhat.rhn.domain.server.MinionServer.subscribeConfigChannels(MinionServer.java:114)
at com.redhat.rhn.frontend.action.configuration.sdc.RankChannelsAction.update(RankChannelsAction.java:105)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.apache.struts.actions.DispatchAction.dispatchMethod(DispatchAction.java:270)
at org.apache.struts.actions.DispatchAction.execute(DispatchAction.java:187)
at org.apache.struts.actions.LookupDispatchAction.execute(LookupDispatchAction.java:150)
at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431)
... 53 more
@digdilem-work what are the channel type of the configuration channels you have on your system? Are those channels traditional configuration channels or salt ones?
They're all traditional channels, shown in the webui as "type = normal"
Interestingly, subscribing a machine to the same channel(s) via SSM works fine, even if the SSM set is one machine or many.
Additional notes from some hours of testing this morning:
Deleting a config channel and recreating it does not fix the issue (with or without same name).
Deleting the machine from uyuni and re-onboarding it DOES fix the issue and it can then add itself to the same channels it couldn't before. .
I'm guessing that the database problem is with the machine's config/ranking data rather than the config channel configuration itself.
Thank you for the information. Can you run the following queries and return the result:
select * from suseStateRevisionConfigChannel sc, rhnconfigchannel cc where sc.config_channel_id = cc.id;
And this one
select * from rhnServerConfigChannel sc, rhnconfigchannel cc where sc.config_channel_id = cc.id;
To run it, you can use the command mgradm support SQL <<< "<SQL_QUERY>"
I think the query that I share is not considering one edge case, where the position field is not set.
I've run those, but the resulting output contains 47 thousand lines and are very large
-rw-r--r--. 1 root root 17M Jul 2 12:24 output1.txt
-rw-r--r--. 1 root root 2.1M Jul 2 12:25 output2.txt
Example line from first;
3626 | 42 | 1 | 42 | 1 | 1 | Debian Servers | debian_base | Base channel for debian-only files | 2023-08-04 10:39:20.024377+01 | 2023-08-04 10:39:20.024377+01
They also include company sensitive strings which I'm unwilling to post in a public place. Please advise if there is a more condensed query possible?
@digdilem-work can you please return the result of the following query
SELECT name confchan_type_id
FROM rhnConfigChannel
WHERE confchan_type_id NOT IN (1, 4);
@admd sure.
(Adding a comma to your query after 'name')
uyuni01:/home/simon # mgradm support sql <<< "SELECT name, confchan_type_id
FROM rhnConfigChannel
WHERE confchan_type_id NOT IN (1, 4);" >output3.txt
ata-oxy-uyuni01:/home/simon # vi output3.txt
2:19PM INF Welcome to mgradm
2:19PM INF Executing command: sql
2:19PM INF Running podman exec -i uyuni-server /usr/bin/spacewalk-sql --select-mode -
name confchan_type_id
----------------------------------------------
hostname1.domain 2
hostname2.domain 2
... repeat for a lot of hostnames...
(189 rows)
confchan_type_id is '2' for all rows returned.
And that's where our problem is, we need to set the null for these config channels ranking. These channels types are not not supported anymore.
@rjmateus @nadvornik fyi . I will see if I can come up with a query but my schedule is pretty full. if someone of you can provide a query for this, that would be super cool.
@digdilem-work can you execute the following query? Take the backup 1st if possible :)
UPDATE susestaterevisionconfigchannel s
SET position = NULL
WHERE s.config_channel_id IN (
SELECT r.id
FROM rhnconfigchannel r
WHERE r.confchan_type_id NOT IN (1, 4)
);
@admd - have applied that, but it didn't appear to change anything?
Fault still evident.
Output from pg is UPDATE 0 - meaning no rows updated?
mgradm support sql -i
3:07PM INF Welcome to mgradm
3:07PM INF Executing command: sql
3:07PM INF Running podman exec -i -t uyuni-server env ENV=/etc/sh.shrc.local TERM=xterm-256color PAGER=less LESS=-M -I -R LESSOPEN=lessopen.sh %s LESSKEY=/etc/lesskey.bin LESSCLOSE=lessclose.sh %s %s LESS_ADVANCED_PREPROCESSOR=no MORE=-sl LANG=POSIX LC_CTYPE=en_US.UTF-8 /usr/bin/spacewalk-sql -i
psql (16.9)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off)
Type "help" for help.
uyuni=# UPDATE susestaterevisionconfigchannel s
uyuni-# SET position = NULL
uyuni-# WHERE s.config_channel_id IN (
uyuni(# SELECT r.id
uyuni(# FROM rhnconfigchannel r
uyuni(# WHERE r.confchan_type_id NOT IN (1, 4)
uyuni(# );
UPDATE 0
uyuni=#