Proxy settings are ignored by xbps-install

[misirca]

trafficstars

In some cases xbps-install seems to totally ignore proxy settings.

Note: this is not a sudo related issue as previously described here.

Note: Trials are made behind a corporate NTLM proxy, therefore outbound connections are made through a (hopefully well configured) CNTLM daemon.

Sanity check

Environment variables are set:

$ sudo env | grep proxy
http_proxy=127.0.0.1:3128
https_proxy=127.0.0.1:3128

wget downloads files, NTLM authentication is nicely handled by CNTLM

$ sudo wget www.free.fr
--2019-07-09 15:42:52--  https://www.free.fr/freebox/index.html
Connexion à 127.0.0.1:3128… connecté.
requête Proxy transmise, en attente de la réponse… 200 OK
Taille : 217424 (212K) [text/html]
Sauvegarde en : « index.html.1 »
index.html.1        100%[===================>] 212,33K  1,31MB/s    ds 0,2s    
2019-07-09 15:42:52 (1,31 MB/s) — « index.html.1 » sauvegardé [217424/217424]

Wireshark log

13	Computer	Parentproxy	TCP	74	60560 → 8080 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=2763940829 TSecr=0 WS=128
14	Parentproxy	Computer	TCP	78	8080 → 60560 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 SACK_PERM=1 TSval=1312957041 TSecr=2763940829 WS=64
15	Computer	Parentproxy	TCP	66	60560 → 8080 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=2763940832 TSecr=1312957041
16	Computer	Parentproxy	HTTP	367	GET http://www.free.fr/ HTTP/1.1 , NTLMSSP_NEGOTIATE

The issue

The following command seem to ignore proxy settings

$ sudo xbps-install -S -d
[DEBUG] XBPS: 0.56 API: 20190621 GIT: UNSET
[DEBUG] Processing configuration directory: /etc/xbps.d
[DEBUG] Processing system configuration directory: /usr/share/xbps.d
[DEBUG] Parsing configuration file: /usr/share/xbps.d/00-repository-main.conf
[DEBUG] [repo] `https://alpha.de.repo.voidlinux.org/current' stored successfully
[DEBUG] /usr/share/xbps.d/00-repository-main.conf: added repository https://alpha.de.repo.voidlinux.org/current
[DEBUG] Parsing configuration file: /usr/share/xbps.d/xbps.conf
[DEBUG] rootdir=/
[DEBUG] metadir=//var/db/xbps
[DEBUG] cachedir=/var/cache/xbps
[DEBUG] confdir=/etc/xbps.d
[DEBUG] sysconfdir=/usr/share/xbps.d
[DEBUG] syslog=true
[DEBUG] bestmatching=false
[DEBUG] Architecture: x86_64
[DEBUG] Target Architecture: (null)
[DEBUG] Repository[0]=https://alpha.de.repo.voidlinux.org/current
[*] Updating `https://alpha.de.repo.voidlinux.org/current/x86_64-repodata' ...
[DEBUG] st.st_size: 1898638
[DEBUG] st.st_atime: 08 Jul 2019 12:43
[DEBUG] st.st_mtime: 08 Jul 2019 09:17
[DEBUG] url_stat.size: -1
[DEBUG] url_stat.atime: 01 Jan 1970 00:00
[DEBUG] url_stat.mtime: 01 Jan 1970 00:00
ERROR: [reposync] failed to fetch file `https://alpha.de.repo.voidlinux.org/current/x86_64-repodata': Connection refused
[DEBUG] [rpool] `https://alpha.de.repo.voidlinux.org/current' failed to fetch repository data: Connection refused
[DEBUG] [rpool] checking `https://alpha.de.repo.voidlinux.org/current' at index 0
[DEBUG] [rpool] `https://alpha.de.repo.voidlinux.org/current' registered.
[DEBUG] [repo] `https://alpha.de.repo.voidlinux.org/current' public key already stored.

Wireshark log, no connection is made to parent proxy, neither to CNTLM daemon. TCP SYN attempt is dropped by the firewall

8	Computer	DNSserver	DNS	87	Standard query 0x0243 A alpha.de.repo.voidlinux.org
9	Computer	DNSServer	DNS	87	Standard query 0x9e51 AAAA alpha.de.repo.voidlinux.org
10	DNSServer	Computer	DNS	128	Standard query response 0x0243 A alpha.de.repo.voidlinux.org CNAME a-hel-fi.m.voidlinux.org A 95.216.76.97
11	DNSServer	Computer	DNS	205	Standard query response 0x9e51 AAAA alpha.de.repo.voidlinux.org CNAME a-hel-fi.m.voidlinux.org SOA ns-cloud-c1.googledomains.com
12	Computer	95.216.76.97	TCP	74	33066 → 443 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=3671962237 TSecr=0 WS=128
13	95.216.76.97	Computer	TCP	54	443 → 33066 [RST, ACK] Seq=1 Ack=1 Win=64240 Len=0

Quick and dirty workaround

I found today that the problem can be temporarily 'fixed' by forcing all TCP connections through the corporate proxy using proxychains ie: App -> Proxychains -> CNTLM proxy -> Corporate Proxy -> Internet

sudo proxychains4 xbps-install -Syu
[proxychains] config file found: /etc/proxychains.conf
[proxychains] preloading /usr/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.14
[*] Updating `https://alpha.de.repo.voidlinux.org/current/x86_64-repodata' ...
[proxychains] Strict chain  ...  127.0.0.1:3128  ...  alpha.de.repo.voidlinux.org:443  ...  OK

Wireshark logs, now connection begins with a proxy negociation.

8	Computer	Parentproxy	TCP	74	60952 → 8080 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=2767636075 TSecr=0 WS=128
9 	Parentproxy	Computer	TCP	78	8080 → 60952 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 SACK_PERM=1 TSval=1316652196 TSecr=2767636075 WS=64
10	Computer	Parentproxy	TCP	66	60952 → 8080 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=2767636076 TSecr=1316652196
11	Computer	Parentproxy	HTTP	302	CONNECT alpha.de.repo.voidlinux.org:443 HTTP/1.0 , NTLMSSP_NEGOTIATE

So finally I can't tell if this is really an unexpected behavior or maybe the consequence of a paranoid corporate proxy?

[Duncaen]

The code 0 where the http_proxy environment variable is used expects fetchParseURL to be successful and handle if there is no scheme:// and set it to http. But fetchParseURL actually stops parsing if it couldn't find a scheme 1 and returns NULL so the url is never parsed and the "default" is unused.

I guess fetchParseURL needs to be changed to allow scheme less urls, but this might require some more changes at other places to error out without a scheme.

[Duncaen]

This is not done, it requires bigger changes to the URL parsing to allow scheme less URLs and needs to be handled correctly in every place where URLs are parsed.

[elpres]

I would be willing to test it with the current installation images, but at the moment we are working remotely and not from behind the corporate proxy. So I will only be able to do any testing once the lockdown is over, which will take a few more weeks.

[elpres]

Good news! The VM I just set up with the current ISO version (void-live-x86_64-musl-20191109.iso) does work behind the proxy just fine. I was able to update the system as well as install new packages. The only intervention on my part was to set the relevant environment variables. This means that the problem originally reported in #75 which lead to the creation of this issue seems to have been resolved. The network situation didn't change since then, the same proxy is still in place.

Are there any more specific tests you would like me to do?

[CMB]

Maybe I should open another ticket for this; xbps-fetch has a similar problem. The only reason I noticed is that I tried building packages in an IPv6-only VM that uses a proxy for IPv4 connectivity. xbps-src was unable to download anything, and when I tried running xbps-fetch manually under strace, it tried to connect directly rather than through the proxy.

[CMB]

Correction: I'm not seeing this on a dual-stacked host, just on an IPv6-only host. If I try to run xbps-fetch on my dual-stacked host here at home, with the *_PROXY environment variables set, it works as advertised in the manpage. I see requests made by xbps-fetch in my squid logs.

The setup for my IPv6-only-host is a bit different. It's on another network. It proxies through tinyproxy on a nearby dual-stacked system, rather than squid.

My test case is the following:

export ALL_PROXY='my-proxy-hostname:8123'
export HTTP_PROXY='my-proxy-hostname:8123'
curl https://github.com/index.html
xbps-fetch https://github.com/index.html

That's a good smoke-test, because github has no IPv6 in 2021. For the IPv6-only host, curl succeeds and xbps-fetch fails.

[leahneukirchen]

I think for a HTTP proxy you need HTTP_PROXY='http://my-proxy-hostname:8123'

[CMB]

I think for a HTTP proxy you need HTTP_PROXY='http://my-proxy-hostname:8123'

That worked. So yeah at least one person (me) is using xbps-fetch and xbps-install behind a proxy.

[leahneukirchen]

Then perhaps we just need better error handling if HTTP_PROXY does not make sense to xbps.

[66Ton99]

It is not a problem of xbps-install. It is nuances of sudo and can be solved by adding Defaults env_keep += "ftp_proxy http_proxy https_proxy no_proxy" into /etc/sudoers - https://wiki.archlinux.org/title/Sudo#Environment_variables

I think it can be closed.