civitai and wget don't want to play nice together

[disposeablehero]

Files from civitai links seems to be having issues with wget, including the one already in the default.sh in the lora array (though commented out). The command as written works just fine when manually entered on the command line so I'm somewhat at a loss.

--2024-09-07 04:09:55--  https://civitai.com/api/download/models/125843?type=Model&format=PickleTensor
Resolving civitai.com (civitai.com)... 104.22.19.237, 172.67.12.143, 104.22.18.237, ...
Connecting to civitai.com (civitai.com)|104.22.19.237|:443... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: https://civitai-delivery-worker-prod.5ac0637cfd0766c97916cefa3764fbdf.r2.cloudflarestorage.com/642113/model/4xUltrasharp.D8rD.pt?X-Amz-Expires=86400&response-content-disposition=attachment%3B%20filename%3D%224xUltrasharp_4xUltrasharpV10.pt%22&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=e01358d793ad6966166af8b3064953ad/20240907/us-east-1/s3/aws4_request&X-Amz-Date=20240907T040955Z&X-Amz-SignedHeaders=host&X-Amz-Signature=d177496bd45904020f73986f79f553bdb071c383a89df8486938654040327b43 [following]
--2024-09-07 04:09:55--  https://civitai-delivery-worker-prod.5ac0637cfd0766c97916cefa3764fbdf.r2.cloudflarestorage.com/642113/model/4xUltrasharp.D8rD.pt?X-Amz-Expires=86400&response-content-disposition=attachment%3B%20filename%3D%224xUltrasharp_4xUltrasharpV10.pt%22&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=e01358d793ad6966166af8b3064953ad/20240907/us-east-1/s3/aws4_request&X-Amz-Date=20240907T040955Z&X-Amz-SignedHeaders=host&X-Amz-Signature=d177496bd45904020f73986f79f553bdb071c383a89df8486938654040327b43
Resolving civitai-delivery-worker-prod.5ac0637cfd0766c97916cefa3764fbdf.r2.cloudflarestorage.com (civitai-delivery-worker-prod.5ac0637cfd0766c97916cefa3764fbdf.r2.cloudflarestorage.com)... 104.18.8.90, 104.18.9.90, 2606:4700::6812:85a, ...
Connecting to civitai-delivery-worker-prod.5ac0637cfd0766c97916cefa3764fbdf.r2.cloudflarestorage.com (civitai-delivery-worker-prod.5ac0637cfd0766c97916cefa3764fbdf.r2.cloudflarestorage.com)|104.18.8.90|:443... connected.
HTTP request sent, awaiting response... 400 Bad Request
The destination name is too long (384), reducing to 236
Incomplete or invalid multibyte sequence encountered
2024-09-07 04:09:55 ERROR 400: Bad Request.

[disposeablehero]

So it's seems to be content-disposition issues with wget and signed aws redirects after the server accepts the token. Since I'm lazy I just wrote a separate if statement for HF links, but I'm wondering if there's some super simple wget setting I overlooked (curl seems to have the same issues, content-disposition just names the downloaded file "?model-blah-blah"

function provisioning_download() {
    if [[ -n $HF_TOKEN && $1 =~ ^https://([a-zA-Z0-9_-]+\.)?huggingface\.co(/|$|\?) ]]; then
        auth_token="$HF_TOKEN"
    
        if [[ -n $auth_token ]];then
            wget --header="Authorization: Bearer $auth_token" -qnc --content-disposition --show-progress -e dotbytes="${3:-4M}" -P "$2" "$1"
        else
            wget -qnc --content-disposition --show-progress -e dotbytes="${3:-4M}" -P "$2" "$1"
        fi
    elif 
        [[ -n $CIVITAI_TOKEN && $1 =~ ^https://([a-zA-Z0-9_-]+\.)?civitai\.com(/|$|\?) ]]; then
        auth_token="$CIVITAI_TOKEN"
        
        if [[ -n $auth_token ]];then
            wget -qnc --content-disposition --show-progress -e dotbytes="${3:-4M}" -P "$2" "${1}&token=${auth_token}"
        else
            wget -qnc --content-disposition --show-progress -e dotbytes="${3:-4M}" -P "$2" "$1"
        fi
    fi
}

[robballantyne]

I'm switching all scripts to curl which does not have this issue.

[zeng407]

try this

function provisioning_download() {
    local url="$1"
    local dir="$2"
    
    # For Civitai URLs, handle redirect and get filename from final URL
    if [[ "$url" == *"civitai.com"* ]]; then
        printf "Getting filename from Civitai redirect...\n"
        
        # Follow redirect and get the final URL
        local final_url=$(curl -sL -o /dev/null -w '%{url_effective}' "$url")
        
        # Extract filename from response-content-disposition parameter in the final URL
        local filename=$(echo "$final_url" | sed -n 's/.*filename%3D%22\([^%]*\)%22.*/\1/p')
        
        # If we couldn't extract filename, try alternative method
        if [[ -z "$filename" ]]; then
            # Try to get it from the redirect location header
            local redirect_url=$(curl -sI "$url" | grep -i "location:" | cut -d' ' -f2 | tr -d '\r')
            filename=$(echo "$redirect_url" | sed -n 's/.*filename%3D%22\([^%]*\)%22.*/\1/p')
        fi
        
        # If still no filename, use default
        if [[ -z "$filename" ]]; then
            filename="$(basename "$url").safetensors"
        fi
        
        printf "Downloading as: %s\n" "$filename"
        wget -O "${dir}/${filename}" "$url"
    else
        wget --header="Authorization: Bearer $HF_TOKEN" -qnc --content-disposition --show-progress -e dotbytes="${3:-4M}" -P "$dir" "$url"
    fi
}