qubes-issues
qubes-issues copied to clipboard
QubesOS
Can't update TemplateVMs, errors in UpdateVM
Qubes OS version R4.0
Affected component(s) or functionality Updates proxy
Brief summary No connection between TemplateVMs and UpdateVM. Unable to update TemplateVMs.
To Reproduce [0. Upgrade TemplateVMs]
- Install new qubes-template-debian-9 (or fedora-30, ...), qubes-template-whonix-gw-15.
- Create sys-whonix AppVM and Debian-9 AppVM (based on just installed qubes-template-whonix-gw-15 and qubes-template-debian-9).
- Setup UpdateVM:
qubes-prefs updatevm sys-whonix - Run in Debian-9:
sudo apt-get update
Expected behavior A distributive repository cache synchronization.
Actual behavior
In Debian-9 TemplateVM: sudo apt-get update causes an error 500 Unable to connect.
In Whonix-gw TemplateVM sudo apt-get update causes: Err:4 tor+https://deb.debian.org/debian buster InRelease Invalid response from proxy: HTTP/1.0 500 Unable to connect Server: tinyproxy/1.10.0 Content-Type: text/html Connection: close [IP: 127.0.0.1 8082]
Additional context
I still can update dom0. And can run sudo apt-get update inside UpdateVM (sys-whonix AppVM) without the connection error.
When I run sudo apt-get update in AppVMs or TemplateVMs, my UpdateVM does not start automatically.
I see errors inside UpdateVM qubes-updates-proxy.service:
user@host:~$ sudo systemctl status qubes-updates-proxy.service
● qubes-updates-proxy.service - Qubes updates proxy (tinyproxy)
Loaded: loaded (/lib/systemd/system/qubes-updates-proxy.service; enabled; vendor preset: enabled)
Drop-In: /lib/systemd/system/qubes-updates-proxy.service.d
└─40_qubes-whonix.conf
Active: active (running) since Sat 2019-07-20 04:18:57 UTC; 15min ago
Process: 2577 ExecStartPre=/usr/bin/install -d --owner tinyproxy --group tinyproxy /var/run/tinyproxy (code=exited, status=0/SUCCESS)
Main PID: 2578 (tinyproxy)
Tasks: 3 (limit: 4667)
Memory: 3.9M
CGroup: /system.slice/qubes-updates-proxy.service
├─2578 /usr/bin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
├─2580 /usr/bin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
└─2581 /usr/bin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
Jul 20 04:33:53 host tinyproxy[2580]: opensock: Could not retrieve info for deb.debian.org
Jul 20 04:33:53 host tinyproxy[2580]: opensock: Could not retrieve info for deb.whonix.org
Jul 20 04:33:53 host tinyproxy[2581]: opensock: Could not retrieve info for deb.qubes-os.org
Jul 20 04:33:53 host tinyproxy[2581]: opensock: Could not retrieve info for sdscoq7snqtznauu.onion
Jul 20 04:33:53 host tinyproxy[2580]: opensock: Could not retrieve info for deb.debian.org
Jul 20 04:33:53 host tinyproxy[2581]: opensock: Could not retrieve info for deb.whonix.org
Jul 20 04:33:53 host tinyproxy[2580]: opensock: Could not retrieve info for deb.debian.org
Jul 20 04:33:53 host tinyproxy[2581]: opensock: Could not retrieve info for deb.debian.org
Jul 20 04:33:53 host tinyproxy[2581]: opensock: Could not retrieve info for deb.debian.org
Jul 20 04:33:54 host tinyproxy[2581]: opensock: Could not retrieve info for deb.debian.org
Solutions you've tried
-
I tried to restart machines.
-
to change UpdateVM to non whonix-gw-15 one.
-
to use proxy in Debian-9 AppVM with wget:
user@host:~$ export http_proxy=127.0.0.1:8082
user@host:~$ wget ipecho.net/plain
--2019-07-21 11:31:28-- http://ipecho.net/plain
Connecting to 127.0.0.1:8082... connected.
Proxy request sent, awaiting response... 500 Unable to connect
2019-07-21 11:31:28 ERROR 500: Unable to connect.
- I tried to check proxy config file (I did not change the file):
user@dom0 ~ % sudo qubesctl state.sls qvm.updates-via-whonix
local:
----------
ID: default-update-policy-whonix
Function: file.prepend
Name: /etc/qubes-rpc/policy/qubes.UpdatesProxy
Result: True
Comment: File /etc/qubes-rpc/policy/qubes.UpdatesProxy is in correct state
Started: 08:10:33.373253
Duration: 5.195 ms
Changes:
Summary for local
------------
Succeeded: 1
Failed: 0
------------
Total states run: 1
Total run time: 5.195 ms
Could you provide the output of cat /etc/qubes-rpc/policy/qubes.UpdatesProxy in dom0?
Yes of course.
user@dom0 ~ % cat /etc/qubes-rpc/policy/qubes.UpdatesProxy
$type:TemplateVM $default allow,target=sys-whonix
$tag:whonix-updatevm $default allow,target=sys-whonix
$tag:whonix-updatevm $anyvm deny
## Note that policy parsing stops at the first match,
## so adding anything below "$anyvm $anyvm action" line will have no effect
## Please use a single # to start your custom comments
# Default rule for all TemplateVMs - direct the connection to sys-net
$type:TemplateVM $default allow,target=sys-net
$anyvm $anyvm deny
Thanks, the content of qubes.UpdatesProxy looks normal to me.
One thing to note: I believe the UpdateVM is only for dom0, not TemplateVMs.
Do Fedora TemplateVMs exhibit the same problems as Debian and Whonix?
Yes. Fedora (just installed and my old one template) exhibit the same problem.
[user@fedora-30 ~]$ sudo dnf update
Fedora Modular 30 - x86_64 0.0 B/s | 0 B 00:00
Failed to synchronize cache for repo 'fedora-modular'
Error: Failed to synchronize cache for repo 'fedora-modular'
If you comment out the first line of /etc/qubes-rpc/policy/qubes.UpdatesProxy, can you update TemplateVMs via sys-net?
(Note: Only try this if you're willing to update TemplateVMs over clearnet. For most people, this should be fine, but if you have especially stringent privacy needs, it might not be.)
(Fortunately, I have a VPN profile in network manager at sys-net.)
I try to comment the first line, and run update with sys-net as updateVM, and the problem persists.
I try to comment the first line, and run update with sys-net as updateVM, and the problem persists.
Again, I'm pretty sure the UpdateVM is just for dom0, not TemplateVMs. For example, if $type:TemplateVM $default allow,target=sys-net is the first applicable rule in /etc/qubes-rpc/policy/qubes.UpdatesProxy, then TemplateVMs should use sys-net for updates, even if the UpdateVM is different (e.g., sys-firewall).
In any case, I'm out of troubleshooting ideas, so I'll leave it this one to the other devs.
From man qubes-prefs (emphasis mine):
updatevm Qube used to download dom0 updates
1dnrr, You posted that you got this error in sys-whonix:
Jul 20 04:33:53 host tinyproxy[2580]: opensock: Could not retrieve info for deb.debian.org
It seams to imply that
- the updates-proxy-setup service works fine in the template
- the policy for qubes.UpdatesProxy routs the rpc call to the correct qube
- the qubes-updates-proxy service in sys-whonix is active
- tinyproxy in sys-whonix is unable to connect to dns
I think that apt-get update succeeding in sys-whonix, rules out issues like not having an internet connection at all as well as tor-related issues.
apt-get manages to resolve domain names just fine, but tinyproxy in the same qube does not. Very strange.
With the help of multiple reinstallations and settings changes, I fixed the updates in whonix-ws-15 and whonix-gw-15 TemplateVMs. But updates in debian-9 and fedora-30 are continue continue to be faulty.
It seems that the problem is in an assignment of network addresses of the machines: the ProxyVM works fine, but the TemplateVMs (fedora and debian) are addressed to a different address.
There are no errors in sudo systemctl status qubes-updates-proxy.service output in ProxyVM (sys-whonix) now.
Does anyone know if this was resolved? I just installed today and am having the same issues... I am unable to update fedora-30 or debian-9. I will be trying to update the whonix VMs separately based on the above.
Does anyone know if this was resolved? I just installed today and am having the same issues... I am unable to update fedora-30 or debian-9. I will be trying to update the whonix VMs separately based on the above.
I'm not aware of any resolution, but I also haven't seen any other recent reports of this problem.
Could you provide a bit more detail about how exactly you're trying to update them and the behavior you're observing?
This is a fresh installation of 4.0.3 on a Dell Optiplex 990 SFF, using stock Qubes configurations/templates. I have not changed any settings from the stock installation standard. I am not using a proxy external to the machine.
I have attempted using the QubesUpdate tool as well as starting a terminal within the template VM to run "sudo apt-get update" or "sudo dnf update" for debian or fedora respectively. Principle symptom using the QubesUpdate tool is that it will sit at the first selected template VM and nothing will occur - no progress is achieved. I have let the system sit for >12hrs with no noticable increase in processor, hard drive or network utilization and no progress completed on update. Running the commands from the terminal I get the same "Error 500" referenced above.
I have not dug too deeply into the status of the proxy service or its logs (yet).
On Tue, Apr 21, 2020 at 5:29 AM Andrew David Wong [email protected] wrote:
Does anyone know if this was resolved? I just installed today and am having the same issues... I am unable to update fedora-30 or debian-9. I will be trying to update the whonix VMs separately based on the above.
I'm not aware of any resolution, but I also haven't seen any other recent reports of this problem.
Could you provide a bit more detail about how exactly you're trying to update them and the behavior you're observing?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/QubesOS/qubes-issues/issues/5187#issuecomment-617065414, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANVNSGGIOKHOHJPHL36D2NTRNVRN3ANCNFSM4IFYETKA .
Go figure. I booted back up this evening, and updating via console commands seems to be working for Debian, Fedora, and both iterations of Whonix. Will try QubesUpdate again once I'm patched to the current baseline.
Steps to reproduce:
Install a debian buster template from offical repositories:
[user@dom0 ~]$ sudo qubes-dom0-update qubes-template-debian-10
The debian-10 TemplateVM won't update. AppVMs based on it are working fine.
user@debian-10:~$ sudo apt update
Err:1 https://deb.qubes-os.org/r4.0/vm buster InRelease
Invalid response from proxy: /usr/lib/qubes/qubes-rpc-multiplexer: 14: /etc/profile.d/20_power_savings_disable_in_vms.sh: shopt: not found HTTP/1.0 200 Connection established Proxy-agent: tinyproxy/1.8.3 [IP: 127.0.0.1 8082]
Err:2 https://deb.debian.org/debian buster InRelease
Invalid response from proxy: /usr/lib/qubes/qubes-rpc-multiplexer: 14: /etc/profile.d/20_power_savings_disable_in_vms.sh: shopt: not found HTTP/1.0 200 Connection established Proxy-agent: tinyproxy/1.8.3 [IP: 127.0.0.1 8082]
Err:3 https://deb.debian.org/debian-security buster/updates InRelease
Invalid response from proxy: /usr/lib/qubes/qubes-rpc-multiplexer: 14: /etc/profile.d/20_power_savings_disable_in_vms.sh: shopt: not found HTTP/1.0 200 Connection established Proxy-agent: tinyproxy/1.8.3 [IP: 127.0.0.1 8082]
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
W: Failed to fetch https://deb.debian.org/debian/dists/buster/InRelease Invalid response from proxy: /usr/lib/qubes/qubes-rpc-multiplexer: 14: /etc/profile.d/20_power_savings_disable_in_vms.sh: shopt: not found HTTP/1.0 200 Connection established Proxy-agent: tinyproxy/1.8.3 [IP: 127.0.0.1 8082]
W: Failed to fetch https://deb.debian.org/debian-security/dists/buster/updates/InRelease Invalid response from proxy: /usr/lib/qubes/qubes-rpc-multiplexer: 14: /etc/profile.d/20_power_savings_disable_in_vms.sh: shopt: not found HTTP/1.0 200 Connection established Proxy-agent: tinyproxy/1.8.3 [IP: 127.0.0.1 8082]
W: Failed to fetch https://deb.qubes-os.org/r4.0/vm/dists/buster/InRelease Invalid response from proxy: /usr/lib/qubes/qubes-rpc-multiplexer: 14: /etc/profile.d/20_power_savings_disable_in_vms.sh: shopt: not found HTTP/1.0 200 Connection established Proxy-agent: tinyproxy/1.8.3 [IP: 127.0.0.1 8082]
W: Some index files failed to download. They have been ignored, or old ones used instead.
sys-net and sys-firewall use fedora-32 TemplateVM.
I have commented out the following line from /etc/qubes-rpc/policy/qubes.UpdateProxy at dom0:
$type:TemplateVM $default allow,target=sys-whonix
It seems that my Whonix is broken. Will fix it.
To debug, try these commands in sys-whonix. Then post the output here.
Just to run a connectivity test (doesn't wait for time synchronization to be done):
whonixcheck --verbose --leak-tests --function check_tor_socks_port
Verbose run of whonixcheck including leak tests (just as a connectivity test. I don't think here is any leak issue.):
whonixcheck --verbose --leak-tests
Running APT inside sys-whonix. If even that fails, it's a "lower level" issue, connectivity issue, no need to look into tinyproxy.
sudo apt update
Today I experienced this issue:
W: Failed to fetch tor+https://deb.debian.org/debian-security/dists/buster/updates/InRelease Invalid response from proxy: HTTP/1.0 500 Unable to connect Server: tinyproxy/1.10.0 Content-Type: text/html Connection: close [IP: 127.0.0.1 8082]
opensock: Could not retrieve info for deb.debian.org
I don't know if it was a similar issue as in this Qubes issue.
Reason was an outdated Tor consensus due to the current DoS issues of the Tor network. References:
- https://forums.whonix.org/t/most-onions-down-due-to-a-denial-of-service-attack-on-the-tor-network/10979
- https://gitlab.torproject.org/tpo/core/tor/-/issues/40237
- https://lists.torproject.org/pipermail/network-health/2021-January/000659.html
As a workaround I stopped Tor, deleted Tor consensus and restarted Tor.
sudo systemctl stop tor@default
sudo su
cd /var/lib/tor
rm /var/lib/tor/cached-*
sudo systemctl restart tor@default
That resolved the issue for me for now.
(I might add a script that users could manually run to Whonix to simplify this process as a usability feature.)
Do you know other Qubes tickets showing HTTP/1.0 500 Unable to connect Server: tinyproxy or systemd journal log containing opensock: Could not retrieve info for deb.debian.org? Please post links to these tickets here.
Did you ever have this issue without any of Whonix being involved? If so, please check if there is an existing Qubes issue ticket for this and if not create one. Mention that Whonix isn't involved and share the link here so I can follow there too.
Almost everyone here knows, but for reference, the updated list of debian onion services:
https://onion.debian.org/
I have experienced this behavior on Qubes Release 4.0.4-rc1 and on 4.1.
Looks like I'm not alone: https://github.com/QubesOS/qubes-issues/issues/4977 https://github.com/QubesOS/qubes-issues/issues/6283
I have experienced this behavior on Qubes Release 4.0.4-rc1 and on 4.1.
Please provide debug information as requested in https://github.com/QubesOS/qubes-issues/issues/5187#issuecomment-762190926.
Looks like I'm not alone: #4977 #6283
Thanks for pointing me to these tickets. Replied there just now.
Hi all, I'm experiencing this particular issue as well. Dom0, Whonix ws and gw updates successfully, however both fedora and debian template shows the following error:
Debian-10:
Failed to fetch https://deb.debian.org/debian/dists/buster/InRelease Invalid response from proxy: HTTP/1.0 500 Unable to connect Server: tinyproxy/1.10.0 Content-Type: text/html Connection: close [IP: 127.0.0.1 8082]
Fedora-33:
Failed to download metadata for repo 'updates': Cannot prepare internal mirrorlist: Curl error (56): Failure when receiving data from the peer for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f33&arch=x86_64 [Received HTTP code 500 from proxy after CONNECT]
The only change I did was the Qubes is now connected to internet via my router. Previously connection to internet was through usb modem. Could it be that the router is somehow blocking the templateVM update? How can I troubleshoot the tinyproxy, if this has something to do with it at all?
Note 0: Any VM created based on fedora-33 and debian-10 connects to the internet just fine.
Note 1: everything was working fine previously (before using the router).
Note 2: content of qubes.UpdatesProxy
$tag:whonix-updatevm $default allow,target=sys-whonix
$tag:whonix-updatevm $anyvm deny
$type:TemplateVM $default allow,target=sys-net
$anyvm $anyvm deny
Thanks
I'm also seeing the "invalid response from proxy" lines in some of my update logs, but it seems to be an intermittent or temporary problem. Doesn't happen every time for me.
I'm also seeing the "invalid response from proxy" lines in some of my update logs, but it seems to be an intermittent or temporary problem. Doesn't happen every time for me.
This seems to be temporary Tor network issues which resolve themselves without user action.
What other users reported here however are permanent update issues. Are these still reproducible?