Unable to download models from Civitai with custom provisioning script

[draxx318]

I'm experiencing an issue when trying to download models using a custom provisioning script. The token has been supplied correctly, but I am unable to download models from Civitai. However, Hugging Face works as expected. I am running this on Runpod. Also, it works fine if no CIVITAI_TOKEN is provided for models that don't require it.

Steps to Reproduce:

Use the custom provisioning script saved as default.sh on GitHub Gist (link to the raw file).

1. Provide provisioning script link as ENV
2. Provide the token as ENV.
3. Attempt to download the Juggernaut XL model.
4. Hugging Face download works.
5. Civitai download fails.

Expected Behavior:

The model should download from Civitai successfully, similar to how it works with Hugging Face.

Actual Behavior:

The download fails when trying to fetch the model from Civitai.

Additional Information:

• Token was supplied correctly.
• The issue only occurs with Civitai (Hugging Face is working fine).
• No error message is returned when using Civitai download.
• The provisioning script is saved as default.sh and is available as a raw file in the provided GitHub Gist link.
• This is running on Runpod.

[draxx318]

Here are the logs, there is a 400 error but cannot understand why.

2025-03-25T12:23:22.615136801Z Downloading: https://civitai.com/api/download/models/782002
2025-03-25T12:23:22.615143995Z --2025-03-25 12:23:22--  https://civitai.com/api/download/models/782002
2025-03-25T12:23:22.615149722Z Resolving civitai.com (civitai.com)... 104.22.19.237, 172.67.12.143, 104.22.18.237, ...
2025-03-25T12:23:22.615156497Z Connecting to civitai.com (civitai.com)|104.22.19.237|:443... connected.
2025-03-25T12:23:25.616524501Z HTTP request sent, awaiting response... 307 Temporary Redirect
2025-03-25T12:23:25.616562216Z Location: https://civitai-delivery-worker-prod.5ac0637cfd0766c97916cefa3764fbdf.r2.cloudflarestorage.com/model/764940/juggernautXI.UXFd.safetensors?X-Amz-Expires=86400&response-content-disposition=attachment%3B%20filename%3D%22juggernautXL_juggXIByRundiffusion.safetensors%22&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=e01358d793ad6966166af8b3064953ad/20250325/us-east-1/s3/aws4_request&X-Amz-Date=20250325T122322Z&X-Amz-SignedHeaders=host&X-Amz-Signature=8838d7daa04f42a4bca40bcad74848be800f4da63e3118ea450aad061f3749cf [following]
2025-03-25T12:23:25.616574579Z --2025-03-25 12:23:24--  https://civitai-delivery-worker-prod.5ac0637cfd0766c97916cefa3764fbdf.r2.cloudflarestorage.com/model/764940/juggernautXI.UXFd.safetensors?X-Amz-Expires=86400&response-content-disposition=attachment%3B%20filename%3D%22juggernautXL_juggXIByRundiffusion.safetensors%22&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=e01358d793ad6966166af8b3064953ad/20250325/us-east-1/s3/aws4_request&X-Amz-Date=20250325T122322Z&X-Amz-SignedHeaders=host&X-Amz-Signature=8838d7daa04f42a4bca40bcad74848be800f4da63e3118ea450aad061f3749cf
2025-03-25T12:23:25.616583100Z Resolving civitai-delivery-worker-prod.5ac0637cfd0766c97916cefa3764fbdf.r2.cloudflarestorage.com (civitai-delivery-worker-prod.5ac0637cfd0766c97916cefa3764fbdf.r2.cloudflarestorage.com)... 162.159.141.50, 172.66.1.46, 2a06:98c1:58::12e, ...
2025-03-25T12:23:25.616591830Z Connecting to civitai-delivery-worker-prod.5ac0637cfd0766c97916cefa3764fbdf.r2.cloudflarestorage.com (civitai-delivery-worker-prod.5ac0637cfd0766c97916cefa3764fbdf.r2.cloudflarestorage.com)|162.159.141.50|:443... connected.
2025-03-25T12:23:25.616600840Z HTTP request sent, awaiting response... 400 Bad Request
2025-03-25T12:23:25.616637996Z The destination name is too long (407), reducing to 236
2025-03-25T12:23:25.616645120Z 2025-03-25 12:23:24 ERROR 400: Bad Request.

[zeng407]

use the updated provisioning_download() function

function provisioning_download() {
    local url="$1"
    local dir="$2"
    
    # For Civitai URLs, handle redirect and get filename from final URL
    if [[ "$url" == *"civitai.com"* ]]; then
        printf "Getting filename from Civitai redirect...\n"
        
        # Follow redirect and get the final URL
        local final_url=$(curl -sL -o /dev/null -w '%{url_effective}' "$url")
        
        # Extract filename from response-content-disposition parameter in the final URL
        local filename=$(echo "$final_url" | sed -n 's/.*filename%3D%22\([^%]*\)%22.*/\1/p')
        
        # If we couldn't extract filename, try alternative method
        if [[ -z "$filename" ]]; then
            # Try to get it from the redirect location header
            local redirect_url=$(curl -sI "$url" | grep -i "location:" | cut -d' ' -f2 | tr -d '\r')
            filename=$(echo "$redirect_url" | sed -n 's/.*filename%3D%22\([^%]*\)%22.*/\1/p')
        fi
        
        # If still no filename, use default
        if [[ -z "$filename" ]]; then
            filename="$(basename "$url").safetensors"
        fi
        
        printf "Downloading as: %s\n" "$filename"
        wget -O "${dir}/${filename}" "$url"
    else
        wget --header="Authorization: Bearer $HF_TOKEN" -qnc --content-disposition --show-progress -e dotbytes="${3:-4M}" -P "$dir" "$url"
    fi
}