qubes-issues icon indicating copy to clipboard operation
qubes-issues copied to clipboard

qubes-updates-proxy: Start request repeated too quickly

Open kaie opened this issue 1 year ago • 9 comments

How to file a helpful issue

Qubes OS release

4.2

Brief summary

qubes-updates-proxy fails to start in sys-net.

systemctl status qubes-updates-proxy reports:

sys-net systemd[1]: qubes-updates-proxy.service: Start request repeated too quickly.
sys-net systemd[1]: qubes-updates-proxy.service: Failed with result 'start-limit-hit'.
sys-net systemd[1]: Failed to start qubes-updates-proxy.service - Qubes updates proxy (tinyproxy).

Steps to reproduce

Shut down all VMs. Start up sys-net, either directly or indirectly. Try to update a qube, which fails. Open a terminal in sys-net and check systemctl, the above service has failed to start.

I can manually work around the issue, after sys-net has started, by running this command: systemctl start qubes-updates-proxy

Additional notes

I originally mentioned my update problems in issue #9025 and claimed it had to with the template that sys-net was based on. That was an incorrect statement, I had incorrectly concluded that, I got misled during my various attempts to resolve that issue on my system.

The new issue I'm reporting happens regardless of the template used for sys-net, I tried both debian-12-xfce and fedora-39-xfce.

kaie avatar Apr 12 '24 14:04 kaie

Can you check earlier logs why it failed initially? You can get it with sudo journalctl b -u qubes-updates-proxy in sys-net.

marmarek avatar Apr 12 '24 21:04 marmarek

I guess you meant "-b -u" (not "b -u")

Apr 12 16:26:23 sys-net systemd[1]: Started qubes-updates-proxy.service - Qubes updates proxy (tinyproxy).
Apr 12 16:26:23 sys-net tinyproxy-wrapper[680]: Found tinyproxy at /usr/bin/tinyproxy
Apr 12 16:26:23 sys-net tinyproxy[680]: Initializing tinyproxy ...
Apr 12 16:26:23 sys-net tinyproxy-wrapper[680]: WARNING: obsolete config item on line 14
Apr 12 16:26:23 sys-net tinyproxy-wrapper[680]: WARNING: obsolete config item on line 15
Apr 12 16:26:23 sys-net tinyproxy-wrapper[680]: WARNING: obsolete config item on line 16
Apr 12 16:26:23 sys-net tinyproxy-wrapper[680]: WARNING: obsolete config item on line 17
Apr 12 16:26:23 sys-net tinyproxy[680]: Reloading config file
Apr 12 16:26:23 sys-net tinyproxy[680]: Reloading config file finished
Apr 12 16:26:25 sys-net tinyproxy[680]: Shutting down.
Apr 12 16:26:25 sys-net tinyproxy-wrapper[885]: Found tinyproxy at /usr/bin/tinyproxy
Apr 12 16:26:25 sys-net tinyproxy-wrapper[885]: WARNING: obsolete config item on line 14
Apr 12 16:26:25 sys-net tinyproxy-wrapper[885]: WARNING: obsolete config item on line 15
Apr 12 16:26:25 sys-net tinyproxy-wrapper[885]: WARNING: obsolete config item on line 16
Apr 12 16:26:25 sys-net tinyproxy-wrapper[885]: WARNING: obsolete config item on line 17
Apr 12 16:26:25 sys-net tinyproxy[885]: Initializing tinyproxy ...
Apr 12 16:26:25 sys-net tinyproxy[885]: Reloading config file
Apr 12 16:26:25 sys-net tinyproxy[885]: Reloading config file finished
Apr 12 16:26:25 sys-net systemd[1]: Stopping qubes-updates-proxy.service - Qubes updates proxy (tinyproxy)...
Apr 12 16:26:25 sys-net systemd[1]: qubes-updates-proxy.service: Deactivated successfully.
Apr 12 16:26:25 sys-net systemd[1]: Stopped qubes-updates-proxy.service - Qubes updates proxy (tinyproxy).
Apr 12 16:26:25 sys-net systemd[1]: Started qubes-updates-proxy.service - Qubes updates proxy (tinyproxy).
Apr 12 16:26:27 sys-net systemd[1]: Stopping qubes-updates-proxy.service - Qubes updates proxy (tinyproxy)...
Apr 12 16:26:27 sys-net tinyproxy[885]: Shutting down.
Apr 12 16:26:27 sys-net systemd[1]: qubes-updates-proxy.service: Deactivated successfully.
Apr 12 16:26:27 sys-net systemd[1]: Stopped qubes-updates-proxy.service - Qubes updates proxy (tinyproxy).
Apr 12 16:26:27 sys-net systemd[1]: Started qubes-updates-proxy.service - Qubes updates proxy (tinyproxy).
Apr 12 16:26:27 sys-net tinyproxy-wrapper[1146]: Found tinyproxy at /usr/bin/tinyproxy
Apr 12 16:26:27 sys-net tinyproxy-wrapper[1146]: WARNING: obsolete config item on line 14
Apr 12 16:26:27 sys-net tinyproxy-wrapper[1146]: WARNING: obsolete config item on line 15
Apr 12 16:26:27 sys-net tinyproxy-wrapper[1146]: WARNING: obsolete config item on line 16
Apr 12 16:26:27 sys-net tinyproxy-wrapper[1146]: WARNING: obsolete config item on line 17
Apr 12 16:26:27 sys-net tinyproxy[1146]: Initializing tinyproxy ...
Apr 12 16:26:27 sys-net tinyproxy[1146]: Reloading config file
Apr 12 16:26:27 sys-net tinyproxy[1146]: Reloading config file finished
Apr 12 16:26:27 sys-net tinyproxy[1146]: Shutting down.
Apr 12 16:26:27 sys-net systemd[1]: Stopping qubes-updates-proxy.service - Qubes updates proxy (tinyproxy)...
Apr 12 16:26:27 sys-net systemd[1]: qubes-updates-proxy.service: Deactivated successfully.
Apr 12 16:26:27 sys-net systemd[1]: Stopped qubes-updates-proxy.service - Qubes updates proxy (tinyproxy).
Apr 12 16:26:27 sys-net systemd[1]: Started qubes-updates-proxy.service - Qubes updates proxy (tinyproxy).
Apr 12 16:26:27 sys-net tinyproxy-wrapper[1222]: Found tinyproxy at /usr/bin/tinyproxy
Apr 12 16:26:27 sys-net tinyproxy-wrapper[1222]: WARNING: obsolete config item on line 14
Apr 12 16:26:27 sys-net tinyproxy-wrapper[1222]: WARNING: obsolete config item on line 15
Apr 12 16:26:27 sys-net tinyproxy-wrapper[1222]: WARNING: obsolete config item on line 16
Apr 12 16:26:27 sys-net tinyproxy-wrapper[1222]: WARNING: obsolete config item on line 17
Apr 12 16:26:27 sys-net tinyproxy[1222]: Initializing tinyproxy ...
Apr 12 16:26:27 sys-net tinyproxy[1222]: Reloading config file
Apr 12 16:26:27 sys-net tinyproxy[1222]: Reloading config file finished
Apr 12 16:26:29 sys-net systemd[1]: Stopping qubes-updates-proxy.service - Qubes updates proxy (tinyproxy)...
Apr 12 16:26:29 sys-net tinyproxy[1222]: Shutting down.
Apr 12 16:26:29 sys-net systemd[1]: qubes-updates-proxy.service: Deactivated successfully.
Apr 12 16:26:29 sys-net systemd[1]: Stopped qubes-updates-proxy.service - Qubes updates proxy (tinyproxy).
Apr 12 16:26:29 sys-net systemd[1]: Started qubes-updates-proxy.service - Qubes updates proxy (tinyproxy).
Apr 12 16:26:29 sys-net tinyproxy-wrapper[1282]: Found tinyproxy at /usr/bin/tinyproxy
Apr 12 16:26:29 sys-net tinyproxy-wrapper[1282]: WARNING: obsolete config item on line 14
Apr 12 16:26:29 sys-net tinyproxy-wrapper[1282]: WARNING: obsolete config item on line 15
Apr 12 16:26:29 sys-net tinyproxy-wrapper[1282]: WARNING: obsolete config item on line 16
Apr 12 16:26:29 sys-net tinyproxy-wrapper[1282]: WARNING: obsolete config item on line 17
Apr 12 16:26:29 sys-net tinyproxy[1282]: Initializing tinyproxy ...
Apr 12 16:26:29 sys-net tinyproxy[1282]: Reloading config file
Apr 12 16:26:29 sys-net tinyproxy[1282]: Reloading config file finished
Apr 12 16:26:29 sys-net tinyproxy[1282]: Shutting down.
Apr 12 16:26:29 sys-net systemd[1]: Stopping qubes-updates-proxy.service - Qubes updates proxy (tinyproxy)...
Apr 12 16:26:29 sys-net systemd[1]: qubes-updates-proxy.service: Deactivated successfully.
Apr 12 16:26:29 sys-net systemd[1]: Stopped qubes-updates-proxy.service - Qubes updates proxy (tinyproxy).
Apr 12 16:26:29 sys-net systemd[1]: qubes-updates-proxy.service: Start request repeated too quickly.
Apr 12 16:26:29 sys-net systemd[1]: qubes-updates-proxy.service: Failed with result 'start-limit-hit'.
Apr 12 16:26:29 sys-net systemd[1]: Failed to start qubes-updates-proxy.service - Qubes updates proxy (tinyproxy).

kaie avatar Apr 13 '24 17:04 kaie

I think the lines 14-17 it complains about are:

MinSpareServers 2 MaxSpareServers 10 StartServers 2 MaxRequestsPerChild 0

kaie avatar Apr 13 '24 17:04 kaie

I think the above log isn't helpful to understand the reason for the restart. Looking at other log items I see:

Apr 12 16:26:24 sys-net systemd[1]: tinyproxy.service - Tinyproxy lightweight HTTP Proxy was skipped because of an unmet condition check (ConditionPathExists=/var/run/qubes-service/tinyproxy).

Should that maybe check for tinyproxy.pid ? I'll try to change file /usr/lib/systemd/system/tinyproxy.service.d/30_not_needed_in_qubes_by_default.conf which contains that condition

kaie avatar Apr 13 '24 17:04 kaie

Hmm, not sure my theory is right.

The existing script looks for something in /var/run/qubes-service/ I don't have any entries named tinyproxy in that directory.

The pid file I had suggested to check for is in /var/run/tinyproxy-updates/tinyproxy.pid (and this was just a wild guess, I'm just reporting what file with a name containing the word "tinyproxy" I can find in /var/run

kaie avatar Apr 13 '24 17:04 kaie

tinyproxy.service is a different thing than qubes-updates-proxy.service. Do you see any other log entries? Maybe NetworkManager is quickly changing settings (updates proxy is reloaded on each IP change)

marmarek avatar Apr 13 '24 20:04 marmarek

Your theory that it has to do with IP addresses and network manager was correct.

My laptop was connected to the network with an ethernet cable, and in addition it was in reach of a wifi network that was configured for automaatic connection. At startup, network manager reports that it connected to both.

With ethernet disconnected, the updates-proxy stay up, no error. With disabled automatic wifi connection and ethernet cable connected, it also works fine.

The problem happens for me if network manager is able to connect using two different methods at the same time.

kaie avatar Apr 14 '24 14:04 kaie

I can confirm @kaie's findings: When my laptop is connected to LAN and Wifi at the same time then the updates proxy is down. Restarting qubes-updates-proxy in sys-net fixes it. Running Qubes 4.2.2

doublehelix24 avatar Aug 03 '24 06:08 doublehelix24

I have too many vlans connected via ethernet. I can confirm that is the reason "start-limit-hit" on qubes-updates-proxy.service . I modified sys-net (where my qubes-updates-proxy.service is running) /lib/systemd/system/qubes-updates-proxy.service file. Added StartLimitBurst=0 under [Service]. It solved for me. Ref: https://www.suse.com/support/kb/doc/?id=000019750

bzcanli avatar Oct 02 '24 16:10 bzcanli

Automated announcement from builder-github

The component core-agent-linux (including package core-agent-linux) has been pushed to the r4.3 testing repository for the Fedora template. To test this update, please install it with the following command:

sudo dnf update --enablerepo=qubes-vm-r4.3-current-testing

Changes included in this update

qubesos-bot avatar Nov 04 '24 19:11 qubesos-bot

Automated announcement from builder-github

The component core-agent-linux (including package core-agent-linux) has been pushed to the r4.3 testing repository for the Fedora template. To test this update, please install it with the following command:

sudo dnf update --enablerepo=qubes-vm-r4.3-current-testing

Changes included in this update

qubesos-bot avatar Nov 04 '24 19:11 qubesos-bot

Automated announcement from builder-github

The component core-agent-linux (including package core-agent-linux) has been pushed to the r4.3 testing repository for the Fedora template. To test this update, please install it with the following command:

sudo dnf update --enablerepo=qubes-vm-r4.3-current-testing

Changes included in this update

qubesos-bot avatar Nov 04 '24 19:11 qubesos-bot

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.3 testing repository for the Debian template. To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing bookworm-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

qubesos-bot avatar Nov 04 '24 19:11 qubesos-bot

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.3 testing repository for the Debian template. To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing trixie-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

qubesos-bot avatar Nov 04 '24 20:11 qubesos-bot

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.2 testing repository for the Debian template. To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing trixie-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

qubesos-bot avatar Nov 05 '24 20:11 qubesos-bot

Automated announcement from builder-github

The component core-agent-linux (including package core-agent-linux) has been pushed to the r4.2 testing repository for the Fedora template. To test this update, please install it with the following command:

sudo dnf update --enablerepo=qubes-vm-r4.2-current-testing

Changes included in this update

qubesos-bot avatar Nov 05 '24 20:11 qubesos-bot

Automated announcement from builder-github

The component core-agent-linux (including package core-agent-linux) has been pushed to the r4.2 testing repository for the Fedora template. To test this update, please install it with the following command:

sudo dnf update --enablerepo=qubes-vm-r4.2-current-testing

Changes included in this update

qubesos-bot avatar Nov 05 '24 20:11 qubesos-bot

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.2 testing repository for the Debian template. To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing bookworm-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

qubesos-bot avatar Nov 05 '24 22:11 qubesos-bot

Automated announcement from builder-github

The component core-agent-linux (including package core-agent-linux) has been pushed to the r4.2 testing repository for the Fedora template. To test this update, please install it with the following command:

sudo dnf update --enablerepo=qubes-vm-r4.2-current-testing

Changes included in this update

qubesos-bot avatar Nov 05 '24 22:11 qubesos-bot

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.2 stable repository for the Debian template. To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

qubesos-bot avatar Nov 12 '24 15:11 qubesos-bot

Automated announcement from builder-github

The component core-agent-linux (including package core-agent-linux) has been pushed to the r4.2 stable repository for the Fedora template. To install this update, please use the standard update command:

sudo dnf update

Changes included in this update

qubesos-bot avatar Nov 12 '24 15:11 qubesos-bot

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.2 stable repository for the Debian template. To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

qubesos-bot avatar Nov 12 '24 15:11 qubesos-bot

Automated announcement from builder-github

The component core-agent-linux (including package core-agent-linux) has been pushed to the r4.2 stable repository for the Fedora template. To install this update, please use the standard update command:

sudo dnf update

Changes included in this update

qubesos-bot avatar Nov 12 '24 15:11 qubesos-bot

Automated announcement from builder-github

The component core-agent-linux (including package core-agent-linux) has been pushed to the r4.2 stable repository for the Fedora template. To install this update, please use the standard update command:

sudo dnf update

Changes included in this update

qubesos-bot avatar Nov 12 '24 15:11 qubesos-bot