batchlinks-webui

batchlinks-webui copied to clipboard

BatchLinks Downloader

Batch-downloading models and stuff in stable-diffusion-webui colab made simple and fast.

Report Bug · Request Feature

does this repo still maintained?

Installation

Copy this line into your colab installation cell. Or into a new cell if you already launched the webui.

!git clone https://github.com/etherealxx/batchlinks-webui /content/stable-diffusion-webui/extensions/batchlinks-webui

or, you can copy the url of this repo and install it via webui and restart the UI.

(If gradio no interface is running or bad gateway shows up when restarting the UI, that means you need to restart the cell anyway 😅)

Using --gradio-queue on the launch.py argument is highly recommended, as it enables this extension to show download progress bar on the UI and a cancel button. The option itself has no negative effect on the webui. Read more here.

You can also run this extension in SDless mode btw.

While it's not recommended to use this extension on your local installation, you can use this extension on Windows. More here.

About

This extension will streamline your downloads on your stable-diffusion-webui colab session. Paste the links you need to download (or you can upload a txt file containing the links), use the hashstag syntax to choose the download location (see below), and hit the Download All button to download them!

Example

Look at this example

#model
https://huggingface.co/WarriorMama777/OrangeMixs/resolve/main/Models/AbyssOrangeMix3/AOM3_orangemixs.safetensors
#vae
https://huggingface.co/hakurei/waifu-diffusion-v1-4/resolve/main/vae/kl-f8-anime2.ckpt
#embed
https://huggingface.co/datasets/Nerfgun3/bad_prompt/resolve/main/bad_prompt_version2.pt
https://huggingface.co/etherealxx/whoseisthis/resolve/main/bad_artist.pt
#lora
https://civitai.com/api/download/models/8840 ##1-mb-lora-trained-in-5-mins-that-does-the-same-thing-as-25-gb-model-but-better
https://civitai.com/api/download/models/6891 ##sans undertale
https://mega.nz/file/gAxTWBAI#uL7EZay-OND5G6ELJlGfUNG0s7Q4TynZKqFdvs0v0tc ##agent8-finetune
https://mega.nz/file/oU43FSTY#vwAfsAb7RKJ4xtsSP7uzrKpWhh1y8BdpIBFurwsVP2o ##agent8-dreambooth
#hypernet
https://cdn.discordapp.com/attachments/1070489470127841381/1070489471964954684/MomopocoV3.pt

This piece of lines will be read from top to bottom. Every hashtag, it will change the current output directory to said directory (see below). So what this example do is it will download AOM3 model to the model folder, then it will download the vae and put it to the Vae folder. Next it will download two embed, bad prompt and bad artist. Next it will download several LoRAs from CivitAI and MEGA, and put it to the Lora folder. Lastly, it changes the directory to hypernet directory, then

You can also copy that example and paste it to a .txt file to use later. You can load a .txt file containing that piece of lines directly from the UI.

When the items is downloading, you can inspect the running code on the colab cell, or just take a coffee and chill☕. If you activate logging, you can inspect the download progress from the UI, more here

When the download is complete, the downloaded file will be listed

Syntax

Hashtag

• Hashtag means change current output directory to this directory. #model means every links below this hashtag, will be downloaded to /content/stable-diffusion-webui/models/Stable-diffusion, until it hits another hashtag, which will change the output directory again. See below for valid hashtags.

  Note: If you use some colab that purposefully uses sd-webui-additional-networks extension to load Lora, use #addnetlora instead of #lora. It will download the lora to where it supposed to be.

Links

• Links are the main things you wants to be downloaded. Current supported links are from:

  • Huggingface (https://huggingface.co/)
  • MEGA (https://mega.nz/) {using mega-cmd}
  • CivitAI (https://civitai.com/)
  • Discord attachments (https://cdn.discordapp.com/attachments/)
  • catbox (https://files.catbox.moe)
  • Github (https://github.com or https://raw.githubusercontent.com)
  • Google Drive (https://drive.google.com) {using gdown}
  • Pixeldrain (https://pixeldrain.com/u/)
  • Mediafire (https://www.mediafire.com/file) {using mediafire-dl}
  • anonfiles (https://anonfiles.com)

  Every links other than that will be ignored. Keep in mind the only supported links are direct download links (see here). For Huggingface, Civitai (model page link method), Discord attachments, catbox, pixeldrain, and anonfiles, there will be four different method of downloading offered (see below). For MEGA, it will use mega-cmd to download. For Github, if, the link is a raw file, it will download the file. Else, it will use git clone, useful to clone extension repo into the webui extension folder.

  More about CivitAI download method here.

Double Hashtag

• Double hashtag means comment. You can put double hashtag in the same line of the link and it will be ignored (keep in mind to put the link first then the double hashtag)

Others

• Other texts will be ignored.

Valid Hashtags

#model, #models, #checkpoint, or #checkpoints will put the downloaded file to /content/stable-diffusion-webui/models/Stable-diffusion

#embedding, #embeddings, #embed, #embeds, #textualinversion, or #ti will put the downloaded file to /content/stable-diffusion-webui/embeddings

#vae or #vaes will put the downloaded file to /content/stable-diffusion-webui/models/VAE

#hypernetwork, #hypernetworks, #hypernet, #hypernets, #hynet, or #hynets will put the downloaded file to /content/stable-diffusion-webui/models/hypernetworks

#lora or #loras will put the downloaded file to /content/stable-diffusion-webui/models/Lora

#addnetlora, #loraaddnet, #additionalnetworks, or #addnet will put the downloaded file to /content/stable-diffusion-webui/extensions/sd-webui-additional-networks/models/lora

#aestheticembedding or #aestheticembed will put the downloaded file to content/stable-diffusion-webui/extensions/stable-diffusion-webui-aesthetic-gradients/aesthetic_embeddings

#controlnet or #cnet will put the downloaded file to /content/stable-diffusion-webui/extensions/sd-webui-controlnet/models

#upscale or #upscaler will put the downloaded file to /content/stable-diffusion-webui/models/ESRGAN

#lycoris, #locon, or #loha will put the downloaded file to /content/stable-diffusion-webui/extensions/sd-webui-additional-networks/models/lora/lycoris

#altmodel or #altmodels will put the downloaded file to the path you choose when using --ckptdir argument on launch.py line. If you didn't use that argument, this hashtag will points to the same directory as #model

Lycoris/Locon/Loha will works just fine if you use #addnetlora instead, as long as you have both addnet extension and locon extension installed. It's unclear if it's can be loaded by native auto1111 lora. (Tell me if you know more about this.)

Github links (if it does not contain /raw/ or /release/download/ in it) doesn't need hashtag. It will always considered as webui extension, and the repository will be cloned to /content/stable-diffusion-webui/extensions/(reponame)

How to get the direct links (Important!)

See here

Huggingface's download method

So there's four supported method: gdown, wget, curl and aria2. Use whatever, really. The difference between them are actually little. Myself love using gdown since the output is cleaner than the others. aria2 has the fastest download speed though.

Additional Syntax

Rename Downloaded Files

Using > symbol, you can rename files. Take this for example

Mostly catbox file has random name, by using > symbol after the link, you can type the desired name on the right. (Don't forget the extension)

Running Shell Commands

You can run shell commands by using ! in front of the command you want, just like in google colab cells. Then press the Download All! button. (Sure, it doesn't download anything, but, well😅)

You can run many lines at once too!

Extract Everyting (@extract)

You can use @extract to extract every *.7z, *.rar, and *.zip on current directory).

#lora
<example lora .zip link>
@extract
#embed

This will extract everything on Lora folder, since the code runs from top to bottom, and when the @extract executes, the current directory is still on Lora folder.

Custom Hashtag Path (@new)

You can use @new <hashtagname> <directory> to make a new usable hashtag and assign a directory path to it.
For example, you type this and press the Download All! button:

@new #private /content/stable-diffusion-webui/outputs
#private
<example model link>

This will make a new hashtag #private, make it points to /content/stable-diffusion-webui/outputs, set #private as current directory, then download whatever into it. You can check if the new hashtag is assigned successfully by looking at the hashtag table on the right/bottom of the screen.

Custom Direct Link Download with aria2 (@aria2)

If there are other direct links that isn't supported by Batchlinks, you can use @aria or @aria2 custom command to download it. The main syntax is:

@aria2 {link} {path/hashtag} > {filename} 

You can change @aria2 with @aria, both works. Both the path/hashtag and filename is optional. This command was made to avoid hassle writing this:

aria2c --summary-interval=1 --console-log-level=error -c -x 16 -s 16 -k 1M {link} -d {path} -o {filename}

There are several ways to use it. Here i use a 20MB download test file as an example.

@aria2 http://212.183.159.230/20MB.zip

☝️ This will download the file into the current directory (where the hashtags points to). So if before that line you use #vae, it will be downloaded to the vae directory. If there are no hashtags before, it'll be sent to the default #model directory. The filename will be the last part of the link (512MB.zip).

@aria2 http://212.183.159.230/20MB.zip #vae

☝️ This will download the file into where #vae points to (which is the vae directory). You can use custom hashtag here as well.

@aria2 http://212.183.159.230/20MB.zip /content/stable-diffusion-webui 

☝️ This will download the file into the written path, which is /content/stable-diffusion-webui. Make sure it is a folder path, not a file path.

@aria2 http://212.183.159.230/20MB.zip /content/stable-diffusion-webui > 20megabytes.zip

☝️ This will download the file into the written path and rename the file into 20megabytes.zip. Remember to make sure to write the extension too.

Gradio Queue

If you use --gradio-queue argument on launch.py, some feature will be activated.

Logging

Enable logging by pressing the Turn On Logging radio button, and wait till Logging activated shows up and the box is blinking with orange border. Logging will tell you what are you actually downloading right now on the webui. After your download session is completed, it's recommended to turn back off the feature.

Cancel

Pressing cancel button while download in progress will stops the current session. Useful when at one time the download speeds is too slow. If you're currently downloading a single item, that item will be cancelled, but the other downloaded one will remain intact.

Progress Bar

There will be an additional progress bar that tells you the current activities.

Changes if --gradio-queue is off

Feature listed above will dissapear, and your only option for download is just aria2 (speed is priority).
Note that when you pressed the Download All! button, nothing will shows up on the UI. You need to check the colab console.

Another thing to note is your download session will always be cutted every 70 seconds (to prevent hangs/desync).

Don't worry, you can continue your session by pressing the Resume Download button. It will refresh the links with the one you haven't downloaded yet, then pressing Download All! will download the remaining links.

Other Features

Notification

This extension will play sound effect when the download process is completed. It's somewhat buggy at the moment though.

Copy From Pastebin

You can put your links on pastebin (write it the way you wrote on Batchlinks), put the pastebin link (https://pastebin.com/xxxx) on the link input, and press the 'Copy from Pastebin' button. It will copy all the the links on the pastebin to the UI. Useful to avoid hassle copy and pasting a bunch of text while on mobile.

SDless mode

This mode will run the extension without the need of stable-diffusion-webui. Good for my own purpose of debugging it😁.

Copy-paste this on a new colab cell then run it:

from IPython.display import clear_output
!pip3 show virtualenv >/dev/null || pip install -q virtualenv
![ -d gradiovenv ] || virtualenv gradiovenv
!git clone https://github.com/etherealxx/batchlinks-webui \
/content/stable-diffusion-webui/extensions/batchlinks-webui
pip3 show gradio >/dev/null || pip install -q gradio==3.16.2; \
pip3 show tqdm >/dev/null || pip3 install -q tqdm
clear_output(wait=True)
!source gradiovenv/bin/activate; \
python /content/stable-diffusion-webui/extensions/batchlinks-webui/scripts/batchlinks-downloader.py

or here's the quicker version without venv:

!pip install gradio==3.16.2
!git clone https://github.com/etherealxx/batchlinks-webui /content/stable-diffusion-webui/extensions/batchlinks-webui
!python /content/stable-diffusion-webui/extensions/batchlinks-webui/scripts/batchlinks-downloader.py

For Windows, run the sdless-windows.bat. Make sure you have python3 and gradio package version 3.16.2 or above. It won't use venv.

This one for MacOS. Tested on Mojave. Create a shell script and give execute access to it.

#!/usr/bin/env bash
pip3 show virtualenv >/dev/null || pip3 install virtualenv
[ -d gradiovenv ] || virtualenv gradiovenv
git clone https://github.com/etherealxx/batchlinks-webui \
$HOME/Downloads/stable-diffusion-webui/extensions/batchlinks-webui
source gradiovenv/bin/activate; \
pip3 show gradio >/dev/null || pip3 install gradio==3.16.2; \
pip3 show tqdm >/dev/null || pip3 install -q tqdm
source gradiovenv/bin/activate; \
python3 $HOME/Downloads/stable-diffusion-webui/extensions/batchlinks-webui/scripts/batchlinks-downloader.py

Local Installation Support

This extension was tested to work on Windows 11. But ultimately, the number one priority for me is colab use, so there's might be bugs on Windows 11 one. Maybe also works on Debian-based linux (but you better inspect the source code first). On Windows, this extension will install MEGAcmd for MEGA file download, wget-windows for download using wget, aria2-static-builds for download using aria2, 7zr and 7z for extracting aria2 installation, and for @extract command. MacOS is just partially supported.

Also remember, in order for this extension to work properly, you need make sure that you tick that add Python 3.10 to PATH checkbox when you installed Python (Which is already recommended when installing stable-diffusion-webui)

Debug Mode (Developer only)

By manually switching globaldebug = False to True on the source code, or using --debug argument when running the sdless mode, the Debug Mode will be activated. It shows every print calls i use to track variables. It will also tracks every files on the models (etc.) paths into a snapshot.txt file (if there isn't one).

There are some batchlinks syntax features that only available on debug mode (putting it on the textbox and click Download All!):

@debugresetdownload - This command will check every model (etc.) directory, and remove every file that isn't on snapshot.txt. Useful for me to removes file quickly when testing. Be careful using this on your local installation.

@debugresetdownload - This command is (supposed to) download a single link with every method available (gdown, curl, wget, aria2), but now i rarely use this command, and haven't updated since. Might be buggy.

Release Notes

Moved here. Check it out for latest features & bug fixes.

Roadmap

• [ ] Add checker for downloaded models (so that it won't download again after the model is downloaded)
• [ ] Different UI for mobile
• [ ] Gradio progress bar
• [ ] Integrating refresh button (create_refresh_button) to refresh models etc. all at once
• [ ] Logo change
• [ ] Moving most of the content of this Readme.md to Wiki instead
• [ ] Other download sites (s-ul.eu, gitgud, bunkr.ru, icedrive)
• [ ] Separate 'download' and 'run command' function with a subtab
• [ ] Support download MEGA and Google Drive folder
• [ ] Using yield instead of gradio's every
• [ ] Youtube video (Tutorial)
• [ ] (Crawler) Download every sd-webui related file from a huggingface repo
• [ ] (Crawler) Download every model from a civitai user page
  _
• [x] aria2 for huggingface download method
• [x] Cleaning the code from unnecesarry comments
• [x] Completed download will use the webui's notification.mp3
• [x] Support customizable hashtag from the UI
• [x] Supports Windows local installation
• [x] UI overhaul
• [x] Using threading/subprocess instead of os.system to download files
• [x] (Windows) wget & aria2 support

Known Bugs

• Progress bar (the yellow bar) doesn't progress as expected
• Sometimes notification sound doesn't play when downloading same file twice in a row
• Sometimes notification sound shows up when starting download, instead of when the download process is completed.
• There's still a chance that the UI of non --gradio-queue session and/or onedotsix freezes after a download session
• Windows: The delay between file is downloaded and the output shows is pretty long, and even sometimes the notification comes at the wrong time.
  

Known Fixed Bugs

Contributing

I no longer actively maintain this project. if you have a suggestion or code-fixing that would make this better, please notify me in the issue tab, fork the repo and create a pull request.
But still, a star on this project would be nice. Thanks again.

Contact

My Email - [email protected]
My Youtube - MJ Devlog

Acknowledgments

• Camenduru's Webui Huggingface - I use his extension as my base (my gradio skill sucks T.T)
• SD Civitai Browser - Civit download script
• Mega-to-Google-Drive - MEGA download script
• MEGAcmd
• mediafire-dl
• Pixeldrain downloader - Inspiration for Pixeldrain download method
• wget-windows
• aria2-static-builds
• 7zr, 7z