Akegarasu
能支持设置连续训练队列吗
简单改了下train_by_toml.ps1，可以按配置批量炼丹，把下面代码复制到记事本保存，后缀改成ps1，放到根目录 2024.03.17更新适配v1.8.0.fix1版本，再试试看 @cindylaii @dahuzi773
# LoRA train script by @Akegarasu
# 2024.03.17
# 批量配置文件,每行以逗号结尾,最后一行不要逗号
# 若报错先分别执行：A强制更新-国内加速.bat install-cn.ps1
# 若想看图表可执行：tensorboard.ps1
[string[]]$my_batch_tomls=(
"./toml/批次1.toml",
"./toml/批次2.toml",
"./toml/批次3.toml"
)
# 若你执行此脚本还剩N小时才能跑完想继续加任务,
# 下方代码可提供等待功能,等待时间到了才会执行,
# 页面上导出几个新的训练参数toml,复制一个此脚本,修改上方toml名字,
# 删除下一行的#井号,修改等待秒数7200为N乘以3600秒后的值
#Get-Date; Start-Sleep -Seconds 7200; Get-Date
#$config_file = "./config/default.toml" # config_file | 使用toml文件指定训练参数
#$sample_prompts = "./config/sample_prompts.txt" # sample_prompts | 采样prompts文件,留空则不启用采样功能
$sdxl = 0 # for sdxl model | SDXL 训练
$multi_gpu = 0 # multi gpu | 多显卡训练 该参数仅限在显卡数 >= 2 使用
# ============= DO NOT MODIFY CONTENTS BELOW | 请勿修改下方内容 =====================
# Activate python venv
.\venv\Scripts\activate
$Env:HF_HOME = "huggingface"
$Env:PYTHONUTF8 = 1
$ext_args = [System.Collections.ArrayList]::new()
$launch_args = [System.Collections.ArrayList]::new()
if ($multi_gpu) {
[void]$launch_args.Add("--multi_gpu")
}
if ($sdxl) {
[void]$launch_args.Add("--sdxl")
}
for ($i = 0; $i -lt $my_batch_tomls.Length; ++$i) {
Write-Output "$((Get-Date).ToString("yyyy:MM:dd HH:mm:ss")) Train start$($i+1): $($my_batch_tomls[$i])"
# run train
$script_name = if ($sdxl) { "sdxl_train_network.py" } else { "train_network.py" }
python -m accelerate.commands.launch $launch_args --num_cpu_threads_per_process=8 "./sd-scripts/$script_name" `
--config_file=$($my_batch_tomls[$i]) `
#--sample_prompts=$sample_prompts `
$ext_args
Write-Output "$((Get-Date).ToString("yyyy:MM:dd HH:mm:ss")) Train finished$($i+1): $($my_batch_tomls[$i])"
}
Write-Output "Train finished"
Read-Host | Out-Null
简单改了下train_by_toml.ps1，可以按配置批量炼丹，把下面代码复制到记事本保存，后缀改成ps1，放到根目录
# LoRA train script by @Akegarasu # 批量配置文件,每行以逗号结尾,最后一行不要逗号 # 若报错先分别执行：A强制更新-国内加速.bat install-cn.ps1 # 若想看图表可执行：tensorboard.ps1 [string[]]$my_batch_tomls=( "./toml/批次1.toml", "./toml/批次2.toml", "./toml/批次3.toml" ) $multi_gpu = 0 # multi gpu | 多显卡训练 该参数仅限在显卡数 >= 2 使用 #$config_file = "./toml/default.toml" # config_file | 使用toml文件指定训练参数 #$sample_prompts = "./toml/sample_prompts.txt" # sample_prompts | 采样prompts文件,留空则不启用采样功能 $utf8 = 1 # utf8 | 使用utf-8编码读取toml；以utf-8编码编写的、含中文的toml必须开启 # ============= DO NOT MODIFY CONTENTS BELOW | 请勿修改下方内容 ===================== # Activate python venv .\venv\Scripts\activate $Env:HF_HOME = "huggingface" $ext_args = [System.Collections.ArrayList]::new() $launch_args = [System.Collections.ArrayList]::new() if ($multi_gpu) { [void]$launch_args.Add("--multi_gpu") } if ($utf8 -eq 1) { $Env:PYTHONUTF8 = 1 } for ($i = 0; $i -lt $my_batch_tomls.Length; ++$i) { Write-Output "$((Get-Date).ToString("yyyy:MM:dd HH:mm:ss")) Train start$($i+1): $($my_batch_tomls[$i])" # run train python -m accelerate.commands.launch $launch_args --num_cpu_threads_per_process=8 "./sd-scripts/train_network.py" ` --config_file=$($my_batch_tomls[$i]) ` #--sample_prompts=$sample_prompts ` $ext_args Write-Output "$((Get-Date).ToString("yyyy:MM:dd HH:mm:ss")) Train finished$($i+1): $($my_batch_tomls[$i])" } Read-Host | Out-Null ;
请问下这个ps1是直接用powershell运行吗？我运行总是提示.\venv\Scripts\activate不存在，确实根目录里也没这个venv虚拟环境
简单改了下train_by_toml.ps1，可以按配置批量炼丹，把下面代码复制到记事本保存，后缀改成ps1，放到根目录
# LoRA train script by @Akegarasu # 批量配置文件,每行以逗号结尾,最后一行不要逗号 # 若报错先分别执行：A强制更新-国内加速.bat install-cn.ps1 # 若想看图表可执行：tensorboard.ps1 [string[]]$my_batch_tomls=( "./toml/批次1.toml", "./toml/批次2.toml", "./toml/批次3.toml" ) $multi_gpu = 0 # multi gpu | 多显卡训练 该参数仅限在显卡数 >= 2 使用 #$config_file = "./toml/default.toml" # config_file | 使用toml文件指定训练参数 #$sample_prompts = "./toml/sample_prompts.txt" # sample_prompts | 采样prompts文件,留空则不启用采样功能 $utf8 = 1 # utf8 | 使用utf-8编码读取toml；以utf-8编码编写的、含中文的toml必须开启 # ============= DO NOT MODIFY CONTENTS BELOW | 请勿修改下方内容 ===================== # Activate python venv .\venv\Scripts\activate $Env:HF_HOME = "huggingface" $ext_args = [System.Collections.ArrayList]::new() $launch_args = [System.Collections.ArrayList]::new() if ($multi_gpu) { [void]$launch_args.Add("--multi_gpu") } if ($utf8 -eq 1) { $Env:PYTHONUTF8 = 1 } for ($i = 0; $i -lt $my_batch_tomls.Length; ++$i) { Write-Output "$((Get-Date).ToString("yyyy:MM:dd HH:mm:ss")) Train start$($i+1): $($my_batch_tomls[$i])" # run train python -m accelerate.commands.launch $launch_args --num_cpu_threads_per_process=8 "./sd-scripts/train_network.py" ` --config_file=$($my_batch_tomls[$i]) ` #--sample_prompts=$sample_prompts ` $ext_args Write-Output "$((Get-Date).ToString("yyyy:MM:dd HH:mm:ss")) Train finished$($i+1): $($my_batch_tomls[$i])" } Read-Host | Out-Null ;请问下这个ps1是直接用powershell运行吗？我运行总是提示.\venv\Scripts\activate不存在，确实根目录里也没这个venv虚拟环境
請參考原文：若报错先分别执行：A强制更新-国内加速.bat install-cn.ps1 我不在中國，我使用 install.ps1裝完環境，然後再執行上面大大寫的xxx.ps1，可執行但是後面還是有問題
简单改了下train_by_toml.ps1，可以按配置批量炼丹，把下面代码复制到记事本保存，后缀改成ps1，放到根目录
# LoRA train script by @Akegarasu # 批量配置文件,每行以逗号结尾,最后一行不要逗号 # 若报错先分别执行：A强制更新-国内加速.bat install-cn.ps1 # 若想看图表可执行：tensorboard.ps1 [string[]]$my_batch_tomls=( "./toml/批次1.toml", "./toml/批次2.toml", "./toml/批次3.toml" ) $multi_gpu = 0 # multi gpu | 多显卡训练 该参数仅限在显卡数 >= 2 使用 #$config_file = "./toml/default.toml" # config_file | 使用toml文件指定训练参数 #$sample_prompts = "./toml/sample_prompts.txt" # sample_prompts | 采样prompts文件,留空则不启用采样功能 $utf8 = 1 # utf8 | 使用utf-8编码读取toml；以utf-8编码编写的、含中文的toml必须开启 # ============= DO NOT MODIFY CONTENTS BELOW | 请勿修改下方内容 ===================== # Activate python venv .\venv\Scripts\activate $Env:HF_HOME = "huggingface" $ext_args = [System.Collections.ArrayList]::new() $launch_args = [System.Collections.ArrayList]::new() if ($multi_gpu) { [void]$launch_args.Add("--multi_gpu") } if ($utf8 -eq 1) { $Env:PYTHONUTF8 = 1 } for ($i = 0; $i -lt $my_batch_tomls.Length; ++$i) { Write-Output "$((Get-Date).ToString("yyyy:MM:dd HH:mm:ss")) Train start$($i+1): $($my_batch_tomls[$i])" # run train python -m accelerate.commands.launch $launch_args --num_cpu_threads_per_process=8 "./sd-scripts/train_network.py" ` --config_file=$($my_batch_tomls[$i]) ` #--sample_prompts=$sample_prompts ` $ext_args Write-Output "$((Get-Date).ToString("yyyy:MM:dd HH:mm:ss")) Train finished$($i+1): $($my_batch_tomls[$i])" } Read-Host | Out-Null ;请问下这个ps1是直接用powershell运行吗？我运行总是提示.\venv\Scripts\activate不存在，确实根目录里也没这个venv虚拟环境
請參考原文：若报错先分别执行：A强制更新-国内加速.bat install-cn.ps1 我不在中國，我使用 install.ps1裝完環境，然後再執行上面大大寫的xxx.ps1，可執行但是後面還是有問題
已更新适配新版
简单改了下train_by_toml.ps1，可以按配置批量炼丹，把下面代码复制到记事本保存，后缀改成ps1，放到根目录
# LoRA train script by @Akegarasu # 批量配置文件,每行以逗号结尾,最后一行不要逗号 # 若报错先分别执行：A强制更新-国内加速.bat install-cn.ps1 # 若想看图表可执行：tensorboard.ps1 [string[]]$my_batch_tomls=( "./toml/批次1.toml", "./toml/批次2.toml", "./toml/批次3.toml" ) $multi_gpu = 0 # multi gpu | 多显卡训练 该参数仅限在显卡数 >= 2 使用 #$config_file = "./toml/default.toml" # config_file | 使用toml文件指定训练参数 #$sample_prompts = "./toml/sample_prompts.txt" # sample_prompts | 采样prompts文件,留空则不启用采样功能 $utf8 = 1 # utf8 | 使用utf-8编码读取toml；以utf-8编码编写的、含中文的toml必须开启 # ============= DO NOT MODIFY CONTENTS BELOW | 请勿修改下方内容 ===================== # Activate python venv .\venv\Scripts\activate $Env:HF_HOME = "huggingface" $ext_args = [System.Collections.ArrayList]::new() $launch_args = [System.Collections.ArrayList]::new() if ($multi_gpu) { [void]$launch_args.Add("--multi_gpu") } if ($utf8 -eq 1) { $Env:PYTHONUTF8 = 1 } for ($i = 0; $i -lt $my_batch_tomls.Length; ++$i) { Write-Output "$((Get-Date).ToString("yyyy:MM:dd HH:mm:ss")) Train start$($i+1): $($my_batch_tomls[$i])" # run train python -m accelerate.commands.launch $launch_args --num_cpu_threads_per_process=8 "./sd-scripts/train_network.py" ` --config_file=$($my_batch_tomls[$i]) ` #--sample_prompts=$sample_prompts ` $ext_args Write-Output "$((Get-Date).ToString("yyyy:MM:dd HH:mm:ss")) Train finished$($i+1): $($my_batch_tomls[$i])" } Read-Host | Out-Null ;请问下这个ps1是直接用powershell运行吗？我运行总是提示.\venv\Scripts\activate不存在，确实根目录里也没这个venv虚拟环境
請參考原文：若报错先分别执行：A强制更新-国内加速.bat install-cn.ps1 我不在中國，我使用 install.ps1裝完環境，然後再執行上面大大寫的xxx.ps1，可執行但是後面還是有問題
已更新适配新版
非常感謝你快速的回覆，我剛才試仍有錯誤。 我仔細想了一下先試一下內建的程式， 發現"train_by_toml.ps1"都不能用，錯誤如下。所以是我這裡環境的問題。 但是我用GUI一切正常，我這兩週練了5個LORA，今天凌晨也剛用GUI練完一個。 目前打算再開一個資料夾全部重裝試試。
2024-03-17 08:53:02 INFO loading u-net: <All keys matched successfully> model_util.py:1009
2024-03-17 08:53:03 INFO loading vae: <All keys matched successfully> model_util.py:1017
2024-03-17 08:53:04 INFO loading text encoder: <All keys matched successfully> model_util.py:1074
2024-03-17 08:53:05 INFO Enable xformers for U-Net train_util.py:2577
Traceback (most recent call last):
File "D:\StableDiffusion\lora-scripts-gui\sd-scripts\train_network.py", line 1063, in memory_efficient_attention_forward with inputs:
query : shape=(1, 2, 1, 40) (torch.float32)
key : shape=(1, 2, 1, 40) (torch.float32)
value : shape=(1, 2, 1, 40) (torch.float32)
attn_bias : <class 'NoneType'>
p : 0.0
decoderF is not supported because:
xFormers wasn't build with CUDA support
attn_bias type is <class 'NoneType'>
operator wasn't built - see python -m xformers.info for more info
[email protected] is not supported because:
xFormers wasn't build with CUDA support
dtype=torch.float32 (supported: {torch.bfloat16, torch.float16})
operator wasn't built - see python -m xformers.info for more info
cutlassF is not supported because:
xFormers wasn't build with CUDA support
operator wasn't built - see python -m xformers.info for more info
smallkF is not supported because:
max(query.shape[-1] != value.shape[-1]) > 32
xFormers wasn't build with CUDA support
operator wasn't built - see python -m xformers.info for more info
unsupported embed per head: 40
Traceback (most recent call last):
File "C:\Users\User\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\User\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "D:\StableDiffusion\lora-scripts-gui\venv\lib\site-packages\accelerate\commands\launch.py", line 1027, in
放棄了...一早起來重裝環境就是裝不起來， CUDA也裝了，也照網友說的用release 1.8.3，還是不能訓練..暈
create LoRA for Text Encoder:
create LoRA for Text Encoder: 72 modules.
create LoRA for U-Net: 192 modules.
enable LoRA for text encoder
enable LoRA for U-Net
prepare optimizer, data loader etc.
Traceback (most recent call last):
File "D:\StableDiffusion\lora-scripts-1.8.3\sd-scripts\train_network.py", line 996, in
简单改了下train_by_toml.ps1，可以按配置批量炼丹，把下面代码复制到记事本保存，后缀改成ps1，放到根目录
# LoRA train script by @Akegarasu # 批量配置文件,每行以逗号结尾,最后一行不要逗号 # 若报错先分别执行：A强制更新-国内加速.bat install-cn.ps1 # 若想看图表可执行：tensorboard.ps1 [string[]]$my_batch_tomls=( "./toml/批次1.toml", "./toml/批次2.toml", "./toml/批次3.toml" ) $multi_gpu = 0 # multi gpu | 多显卡训练 该参数仅限在显卡数 >= 2 使用 #$config_file = "./toml/default.toml" # config_file | 使用toml文件指定训练参数 #$sample_prompts = "./toml/sample_prompts.txt" # sample_prompts | 采样prompts文件,留空则不启用采样功能 $utf8 = 1 # utf8 | 使用utf-8编码读取toml；以utf-8编码编写的、含中文的toml必须开启 # ============= DO NOT MODIFY CONTENTS BELOW | 请勿修改下方内容 ===================== # Activate python venv .\venv\Scripts\activate $Env:HF_HOME = "huggingface" $ext_args = [System.Collections.ArrayList]::new() $launch_args = [System.Collections.ArrayList]::new() if ($multi_gpu) { [void]$launch_args.Add("--multi_gpu") } if ($utf8 -eq 1) { $Env:PYTHONUTF8 = 1 } for ($i = 0; $i -lt $my_batch_tomls.Length; ++$i) { Write-Output "$((Get-Date).ToString("yyyy:MM:dd HH:mm:ss")) Train start$($i+1): $($my_batch_tomls[$i])" # run train python -m accelerate.commands.launch $launch_args --num_cpu_threads_per_process=8 "./sd-scripts/train_network.py" ` --config_file=$($my_batch_tomls[$i]) ` #--sample_prompts=$sample_prompts ` $ext_args Write-Output "$((Get-Date).ToString("yyyy:MM:dd HH:mm:ss")) Train finished$($i+1): $($my_batch_tomls[$i])" } Read-Host | Out-Null ;请问下这个ps1是直接用powershell运行吗？我运行总是提示.\venv\Scripts\activate不存在，确实根目录里也没这个venv虚拟环境
請參考原文：若报错先分别执行：A强制更新-国内加速.bat install-cn.ps1 我不在中國，我使用 install.ps1裝完環境，然後再執行上面大大寫的xxx.ps1，可執行但是後面還是有問題
已更新适配新版
非常感謝你快速的回覆，我剛才試仍有錯誤。 我仔細想了一下先試一下內建的程式， 發現"train_by_toml.ps1"都不能用，錯誤如下。所以是我這裡環境的問題。 但是我用GUI一切正常，我這兩週練了5個LORA，今天凌晨也剛用GUI練完一個。 目前打算再開一個資料夾全部重裝試試。
2024-03-17 08:53:02 INFO loading u-net: model_util.py:1009 2024-03-17 08:53:03 INFO loading vae: model_util.py:1017 2024-03-17 08:53:04 INFO loading text encoder: model_util.py:1074 2024-03-17 08:53:05 INFO Enable xformers for U-Net train_util.py:2577 Traceback (most recent call last): File "D:\StableDiffusion\lora-scripts-gui\sd-scripts\train_network.py", line 1063, in trainer.train(args) File "D:\StableDiffusion\lora-scripts-gui\sd-scripts\train_network.py", line 243, in train vae.set_use_memory_efficient_attention_xformers(args.xformers) File "D:\StableDiffusion\lora-scripts-gui\venv\lib\site-packages\diffusers\models\modeling_utils.py", line 262, in set_use_memory_efficient_attention_xformers fn_recursive_set_mem_eff(module) File "D:\StableDiffusion\lora-scripts-gui\venv\lib\site-packages\diffusers\models\modeling_utils.py", line 258, in fn_recursive_set_mem_eff fn_recursive_set_mem_eff(child) File "D:\StableDiffusion\lora-scripts-gui\venv\lib\site-packages\diffusers\models\modeling_utils.py", line 258, in fn_recursive_set_mem_eff fn_recursive_set_mem_eff(child) File "D:\StableDiffusion\lora-scripts-gui\venv\lib\site-packages\diffusers\models\modeling_utils.py", line 258, in fn_recursive_set_mem_eff fn_recursive_set_mem_eff(child) File "D:\StableDiffusion\lora-scripts-gui\venv\lib\site-packages\diffusers\models\modeling_utils.py", line 255, in fn_recursive_set_mem_eff module.set_use_memory_efficient_attention_xformers(valid, attention_op) File "D:\StableDiffusion\lora-scripts-gui\venv\lib\site-packages\diffusers\models\attention_processor.py", line 273, in set_use_memory_efficient_attention_xformers raise e File "D:\StableDiffusion\lora-scripts-gui\venv\lib\site-packages\diffusers\models\attention_processor.py", line 267, in set_use_memory_efficient_attention_xformers _ = xformers.ops.memory_efficient_attention( File "D:\StableDiffusion\lora-scripts-gui\venv\lib\site-packages\xformers\ops\fmha__init__.py", line 247, in memory_efficient_attention return memory_efficient_attention( File "D:\StableDiffusion\lora-scripts-gui\venv\lib\site-packages\xformers\ops\fmha__init_.py", line 365, in memory_efficient_attention return memory_efficient_attention_forward( File "D:\StableDiffusion\lora-scripts-gui\venv\lib\site-packages\xformers\ops\fmha__init.py", line 381, in _memory_efficient_attention_forward op = _dispatch_fw(inp, False) File "D:\StableDiffusion\lora-scripts-gui\venv\lib\site-packages\xformers\ops\fmha\dispatch.py", line 125, in _dispatch_fw return _run_priority_list( File "D:\StableDiffusion\lora-scripts-gui\venv\lib\site-packages\xformers\ops\fmha\dispatch.py", line 65, in _run_priority_list raise NotImplementedError(msg) NotImplementedError: No operator found for
memory_efficient_attention_forwardwith inputs: query : shape=(1, 2, 1, 40) (torch.float32) key : shape=(1, 2, 1, 40) (torch.float32) value : shape=(1, 2, 1, 40) (torch.float32) attn_bias : <class 'NoneType'> p : 0.0decoderFis not supported because: xFormers wasn't build with CUDA support attn_bias type is <class 'NoneType'> operator wasn't built - seepython -m xformers.infofor more info[email protected]is not supported because: xFormers wasn't build with CUDA support dtype=torch.float32 (supported: {torch.bfloat16, torch.float16}) operator wasn't built - seepython -m xformers.infofor more infocutlassFis not supported because: xFormers wasn't build with CUDA support operator wasn't built - seepython -m xformers.infofor more infosmallkFis not supported because: max(query.shape[-1] != value.shape[-1]) > 32 xFormers wasn't build with CUDA support operator wasn't built - seepython -m xformers.infofor more info unsupported embed per head: 40 Traceback (most recent call last): File "C:\Users\User\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\User\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "D:\StableDiffusion\lora-scripts-gui\venv\lib\site-packages\accelerate\commands\launch.py", line 1027, in main() File "D:\StableDiffusion\lora-scripts-gui\venv\lib\site-packages\accelerate\commands\launch.py", line 1023, in main launch_command(args) File "D:\StableDiffusion\lora-scripts-gui\venv\lib\site-packages\accelerate\commands\launch.py", line 1017, in launch_command simple_launcher(args) File "D:\StableDiffusion\lora-scripts-gui\venv\lib\site-packages\accelerate\commands\launch.py", line 637, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['D:\StableDiffusion\lora-scripts-gui\venv\Scripts\python.exe', './sd-scripts/train_network.py', '--config_file=./toml/batch_2.toml', '--sample_prompts=./config/sample_prompts.txt']' returned non-zero exit status 1. Train finished
此版本为基础，先确认已执行 A强制更新.bat 和 install.ps1； 看日志是xFormers有问题，在页面 专家 -速度优化选项下关闭xFormers； 再重新跑一遍，提供启动到报错的完整日志和toml
此版本为基础，先确认已执行 A强制更新.bat 和 install.ps1； 看日志是xFormers有问题，在页面 专家 -速度优化选项下关闭xFormers； 再重新跑一遍，提供启动到报错的完整日志和toml
成功了，您的程式很好。很感謝您的幫忙。這樣我以後可以一次排程練3個。放一個晚上就好了。
是我對安裝不太熟。 寫一下我的流程：
我的環境Windows 11, RTX4080 Laptop
- 安裝CUDA 11.8
- 下載秋葉訓練器(lora-scripts) release版1.8.3
- 在Windows裡要用管理員權限打開Powershell 輸入Set-ExecutionPolicy RemoteSigned 執行.\install.ps1
安裝好後，再試script .\train_by_toml_new_cindy_2_lora.ps1 (即您的程式..我放2個照片集訓練)
若不能用，試一下.\run_gui.ps1 訓練，我用GUI訓練是正常的。 再回到Powershell試.\train_by_toml_new_cindy_2_lora.ps1 (即您的程式..我放2個照片集訓練)
就可以用了，可掛上多個toml訓練 看起來是正常的
剛才的大發現，train.ps1我能用 但是train_by_toml.ps1不能用。 如果我在train_by_toml.ps1裡加一行 $Env:XFORMERS_FORCE_DISABLE_TRITON = "1" 就可以用了。我不知道這有什麼影響。