Resume Downloads

[George-D-S]

Is your feature request related to a problem? Please describe. I'm having trouble getting it to install because the larger downloads are failing. My internet connection can be flakey at times.

The first issue was with sd-v1-4.ckpt but I could work round this by getting the URL from the script and do the download manually.

I'm now stuck with it having opened the browser window and downloading something it's putting under

.cache\huggingface\transformers

but I'm unclear of what this is or where I'd get it from.

Describe the solution you'd like

I'd like any downloads to have an auto resume-retry option, or for instructions on how to work round it by downloading the files directly.

[JeLuF]

I think this would be useful: curl -C - --retry 20 --retry-all-errors --retry-delay 5 -L -k https://me.cmdr2.org/stable-diffusion-ui/sd-v1-4.ckpt > sd-v1-4.ckpt

Risk: "-C -" will use any existing file and append to it - whether it's from the same source or something completely different. All current calls to curl have an @if not exist statement that ensures that this problem will not occur.

[JeLuF]

curl --retry 5 does not retry the download if the connection gets lost. For this, --retry-all-errors is needed, wich was added to curl two years ago. According to yesterday's experience, there are a lot of older versions of curl still in use, so we can't rely on --retry-all-errors to be available.

1. Download a new version of curl early during the installation and use it for all following downloads.
2. Write a retry loop. Simple in bash, ugly in cmd. Ideally, using curl -o, so that -C can be used.

[JeLuF]

Draft versions

CMD

@CALL :trycurl 20 "https://me.cmdr2.org/stable-diffusion-ui/sd-v1-4.ckpt" "xxx.data"
@echo Download completed.


@GOTO :EOF
REM ####################################################################################
REM ### Helper subroutines
REM ####################################################################################

REM ------ trycurl -------
REM Try %1 times to download %2 to the file %3
REM The file gets deleted before trying the download
REM The download will resume where it failed before
REM ----------------------
:trycurl
@set count=%1
@set url=%2
@set dest=%3
@del %dest%
:DoWhile
    @echo trying to download (%count% attempts left)
    @if %count%==0 goto EndDoWhile
    @set /a count = %count% -1
    @call curl -C - -L %url% -o %dest%
    @if %errorlevel%==0 exit /b
    @timeout /t 5 /nobreak
    @if %count% gtr 0 goto DoWhile
:EndDoWhile
@echo ERROR - Failed to download %url%
@echo ERROR - Giving up.
@pause
@exit 

Bash

#!/bin/bash

trycurl() {
    tries=$1
    url=$2
    dest=$3

    [ -f $dest ] && rm $dest
    until curl -L -C - -o $dest $url; do
       tries=$(( tries - 1 ))
       if [ "$tries" == "0" ]; then
          echo "ERROR: Failed to download $url"
          echo "ERROR: Giving up"
          exit
       fi
       sleep 5
    done
}

trycurl 2 https://asdlfjasldkfjasldkfjme.cmdr2.org/stable-diffusion-ui/sd-v1-4.ckpt xxx.data

[cmdr2]

@JeLuF I actually like option 1 - install curl with the installer. We're using curl for new installations anyway, so it'll be a simple step to add "curl" to the list of packages installed in bootstrap.{sh,bat}. Just like how we force-install conda in bootstrap.

I'm really not keen on any DOS/Batch scripts, if there's an alternative :)

[JeLuF]

I had a look at pycurl. It doesn't support any retry logic. It would have to be implemented on the client side.

[cmdr2]

We resume downloads now