add http/socks5 proxy support for package manager

[binarycraft007]

Not all people have unrestricted network access to package source, for example: github is censored by some countries, it would be good to add proxy support for package manager so that user under restricted network can fetch package successfully.

[nektro]

personally more interested in this so that Zig supports Tor

[Tesseract22]

What is the progress on this? I am also suffering from censorship of github. Besides directly setting a proxy on zig package manager, is there a way to manually download the dependencies and have the package manager skip over the downloading part?

[snoire]

Besides directly setting a proxy on zig package manager, is there a way manually to download the dependencies and have the package manager skips over the downloading part?

You can download the dependency into ~/.cache/zig/p, unzip it, and rename the folder to the hash value. Then zig build will use the local copies instead of trying to download.

[tizee]

Here is my workaround for the proxy issue. After building my patched zig from source, I could use zig build --build-proxy http://127.0.0.1:1080 to fetch pacakages via proxy.

I'm still confused that there are two zig build commands. One is for fetch packages and another is for build runner.

https://github.com/tizee/zig/tree/feature/http-socks5-proxy-support

[alberic89]

zig build should detect and use http_proxy and https_proxy environment variables. My entire network is behind a proxy that doesn't let anything else through, and I assume this is also the case in many universities and work offices. It's a shame not to be able to use the Zig build system simply because it can't read the signposts of the internet route.

[rumisle]

My temporary workaround is to manually download the package, host it locally and change the dependency URL accordingly:

{
  .binned_allocator = .{
    .url = "http://localhost:8080/8372900fcc09e38d7b0b6bbaddad3904-6c3321e0969ff2463f8335da5601986cf2108690.tar.gz",
    .hash = "1220363c7e27b2d3f39de6ff6e90f9537a0634199860fea237a55ddb1e1717f5d6a5",
  },
}

If anyone wants to implement http_proxy, https_proxy and no_proxy properly, it should be relatively easy, since zig's http client already supports proxies, and there is Go's proxy parsing and matching reference implementation: https://cs.opensource.google/go/x/net/+/refs/tags/v0.15.0:http/httpproxy/proxy.go

[konosubakonoakua]

any update?

[Jack-Ji]

A dirty script to workaround the issue.

#!/bin/bash

set -e

forceupdate=false
if [ "$1" = "-f" ]; then
    forceupdate=true
    shift
fi

do_fetch() {
    for d in `grep -o 'url *=.*' $1 | cut -d = -f 2`; do
        d=`echo $d | grep -o 'https://[^"]*'`
        echo -e "\n>>> Deal with $d"
        if echo $d | grep -q '\.tar\.gz$'; then
            url=$d
        elif echo $d | grep -q '#[.0-9a-z]*$'; then
            url_base=`echo $d | awk -F \# '{print $1}'`
            url_base=${url_base%.git}
            url_commit=`echo $d | awk -F \# '{print $2}'`
            url="${url_base}/archive/${url_commit}.tar.gz"
        else
            echo ">>> Ignored $d, unable to resolve it!"
            continue
        fi
        hash=`grep -m 1 -A 1 "$d" $1 | grep hash |  awk -F \" '{print $(NF-1)}'`
        if [ -z "$hash" ]; then
          forceupdate=true
        fi
        if ! $forceupdate && [ -e ~/.cache/zig/p/$hash ]; then
          echo ">>> Found $url in cache, ignored"
          continue
        fi
        wget $url
        tarfile=${url##*/}
        hash=`zig fetch --debug-hash $tarfile | tail -n 1`
        echo ">> hash of $d:"
        echo -e "\t$hash"
        rm $tarfile
        if [ -e ~/.cache/zig/p/$hash/build.zig.zon ]; then
            do_fetch ~/.cache/zig/p/$hash/build.zig.zon
        fi
    done

    for d in `grep -o 'path *=.*' $1 | cut -d = -f 2`; do
        path=`echo $d | awk -F \" '{print $(NF-1)}'`
        if [ -e $path/build.zig.zon ]; then
            do_fetch $path/build.zig.zon
        fi
    done
}

zonfile=$1
if [ -z "$zonfile" ]; then
    zonfile=build.zig.zon
fi

if ! [ -e $zonfile ]; then
    echo "can't find build.zig.zon!"
    exit 1
fi

do_fetch $zonfile

EDIT 1: Eliminate redundant downloading EDIT 2: Force update if hash is empty, output checksum hash for reference

[konosubakonoakua]

or manually manage plugins like this project. https://github.com/Srekel/tides-of-revival/blob/2aec79d644bbf5358ff502952386e138ebbf247f/sync_external.py#L63

[tw4452852]

A dirty script to workaround the issue.

#!/bin/bash

set -e

forceupdate=false
if [ "$1" = "-f" ]; then
    forceupdate=true
    shift
fi

do_fetch() {
    for d in `grep -o 'url *=.*' $1 | cut -d = -f 2`; do
        d=`echo $d | grep -o 'https://[^"]*'`
        echo -e "\n>>> Deal with $d"
        if echo $d | grep -q '\.tar\.gz$'; then
            url=$d
        elif echo $d | grep -q '#[.0-9a-z]*$'; then
            url_base=`echo $d | awk -F \# '{print $1}'`
            url_base=${url_base%.git}
            url_commit=`echo $d | awk -F \# '{print $2}'`
            url="${url_base}/archive/${url_commit}.tar.gz"
        else
            echo ">>> Ignored $d, unable to resolve it!"
            continue
        fi
        hash=`grep -m 1 -A 1 "$d" $1 | grep hash |  awk -F \" '{print $(NF-1)}'`
        if [ -z "$hash" ]; then
          forceupdate=true
        fi
        if ! $forceupdate && [ -e ~/.cache/zig/p/$hash ]; then
          echo ">>> Found $url in cache, ignored"
          continue
        fi
        wget $url
        tarfile=${url##*/}
        hash=`zig fetch --debug-hash $tarfile | tail -n 1`
        echo ">> hash of $d:"
        echo -e "\t$hash"
        rm $tarfile
        if [ -e ~/.cache/zig/p/$hash/build.zig.zon ]; then
            do_fetch ~/.cache/zig/p/$hash/build.zig.zon
        fi
    done

    for d in `grep -o 'path *=.*' $1 | cut -d = -f 2`; do
        path=`echo $d | awk -F \" '{print $(NF-1)}'`
        if [ -e $path/build.zig.zon ]; then
            do_fetch $path/build.zig.zon
        fi
    done
}

zonfile=$1
if [ -z "$zonfile" ]; then
    zonfile=build.zig.zon
fi

if ! [ -e $zonfile ]; then
    echo "can't find build.zig.zon!"
    exit 1
fi

do_fetch $zonfile

EDIT 1: Eliminate redundant downloading EDIT 2: Force update if hash is empty, output checksum hash for reference

For someone using this script, a little fix to find the global caching directory instead of hardcoded ~/.cache/zig: global_cache_dir=$(zig env | jq -r .global_cache_dir)

[jiacai2050]

Before this issue get fixed, try https://zigcli.liujiacai.net/programs/zigfetch/ if you have problems when download deps.

[crosstyan]

Not sure if it's still useful but another python based solution crosstyan/zig-fetch-py

[FH0]

一个临时的解决方案是使用 clash 的 tun 模式。

[yangh]

Before this issue get fixed, try https://zigcli.liujiacai.net/programs/zigfetch/ if you have problems when download deps.

zigfetch is cool, but it's an egg or chicken issue.

How I success-ed to build the zigfetch from source code:

1. Download zigcli, zigcurl manually
2. Download deps of zigcli/zigcurl manually a. in the zigcli source code dir, add deps of zigcurl into build.zig.zon, change the url to file:///path/to/package..gz, so all the required deps will be in the cache, which allow us to build zigcurl and zigcli.
3. zig build
4. zigfetch is here now.

[jiacai2050]

zigfetch is cool, but it's an egg or chicken issue.

You can download pre-built binaries here: https://github.com/jiacai2050/zigcli/releases

[bridgeQiao]

If use 0.15.1, can use the patch: #23365 patch