Simple Installer for Unified Directory Structure, Initial Implementation

[lstein]

A Smooth Install Experience (Maybe)

This PR implements the runtime directory layout scheme described in Discussion item #1789 and performs the install in such a way that everything, including the virtual environment and init file, are located in the invokeai runtime directory. It is dependent on Python 3.9 or 3.10 being installed on the target machine, but is not dependent on conda, git or any other package. (At startup the install script checks that the right version of Python is installed)

Pros

• The install script logic is much simpler, and handles being stopped and restarted halfway through.
• Everything gets installed in one place, including the launch script, the init file and the .venv directory.
• The source code is never left around to confuse the user.
• Works on Linux CUDA and AMD
• Readline autocomplete support is preserved in CLI

Cons

• User has to install Python first.
• Not tested on Mac (please test)
• Windows .bat files still need to be written
• If user moves the invokeai directory everything breaks.

To test:

1. Enter the directory simple_installer

2. Run create_installers.sh to create the three zip files. Or use the ones that are attached InvokeAI-simple-installer-2.2.4-linux.zip InvokeAI-simple-installer-2.2.4-mac.zip

3. Move ~/invokeai out of the way if you plan to reuse this path.

4. Make sure to deactivate any existing conda/venv environment (but I'd be interested in knowing what happens if you don't deactivate them. The install script tries to inactivate any environments hanging around at startup, but I haven't tested if this works)

5. Go to your downloads/build directory.

6. unzip the .zip file for your platform

7. Execute install.sh

8. Go to your invokeai runtime directory and launch using invoke.sh

[hipsterusername]

how are we handling updates, given the folder shift? will this be backward compatible?

[mauwii]

Well, I know it is not the premise of this pull request, but:

🎉 THIS FINALY FIXES THE DOCKERFILE 🥳

[mauwii]

Also verified manual pip installation steps now (on MacBook Air M1):

1. My env has:
   1. INVOKEAI_ROOT=~/invokeai
   2. HUGGINGFACE_TOKEN=***
   3. PYTORCH_ENABLE_MPS_FALLBACK=1
2. in existing venv did a pip install --upgrade requirements.txt (requirements.txt -> environments-and-requirements/requirements-mac-mps-cpu.txt)
3. python scripts/configure_invokeai.py
4. generated some images in python scripts/invoke.py --web

[hussein8686]

حسنًا ، أعلم أنه ليس منطلق طلب السحب هذا ، ولكن:

🎉يصلح هذا الملف النهائي DOCKERFILE🥳

[hussein8686]

شكرا لك على مساعده

[ebr]

Looks awesome so far from eyeballing the PR! I'll give it some thorough testing it in a few hours.

Please note that in #1798 I'm aiming to remove Globals in favour of a more robust path manager, but I can rebase on top of this code as it's likely to be merged first.

[lstein]

I'm glad the response has been positive, so far. I've turned this into a draft pending the windows install .bat file.

[lstein]

Warning: there is a bug in the args parsing currently. The parser doesn’t honour the —root argument when looking for the init file. I will commit a fix later today.

UPDATE: This is now fixed. In addition, if an old ~/.invokeai file is found and no new ~/invokeai/invokeai.init file is present, then the CLI and web UIs will read from this initfile and issue a deprecation warning.

[lstein]

it's

Sorry I keep moving the goal posts on you! It may take some time for me to write the windows bat script so you’re welcome to finalize the globals handling and I’ll adapt my code changes.

[ebr]

Sorry I keep moving the goal posts on you! It may take some time for me to write the windows bat script so you’re welcome to finalize the globals handling and I’ll adapt my code changes.

No problem at all. It's taking me a while to do that refactor right, and it isn't as urgent as this issue.

[ebr]

Linux install went smooth as glass for me on Pop!_OS (Ubuntu) 22.04. Technically works great, though potential stumbling blocks exist in the UX.... I will add those as inline comments shortly.

Will test the Windows .bat installer next. In a VM unfortunately, as I don't have suitable hardware to run SD :(. Should be sufficient for a test though.

[ebr]

Once again, given that Python is a dependency anyway, perhaps we should consider a pure python install script (using only the standard library) rather than suffering through the .bat syntax. I am very tempted to take a stab at it unless the idea is vetoed :)

[lstein]

I've done the toil and written a Windows .bat script that seems to work. It even does some error checking. This was not fun... at all. A pure python install would have been so much better! InvokeAI-simple-installer-2.2.4-windows.zip InvokeAI-simple-installer-2.2.4-mac.zip InvokeAI-simple-installer-2.2.4-linux.zip

The only problem now is that the Windows install puts the web backend's frontend assets in a different place in .venv than it does on Linux. I don't know whether this is a python3.9 vs python3.10 thing, a pip version thing, or I've just got a bug in my code. I've committed a temporary (untested) fix for this, but it shakes my confidence that there won't be another version of python that places the frontend in yet a fourth place.

1. The placeholder URLs for troubleshooting and the install help need to finalized.
2. The installation guide has to be rewritten again.
3. We need to tag release 2.2.4 and have the install script download that tag and not just HEAD.
4. I need to test whether the changes have broken the binary and/or source installers.

[lstein]

I've made a series of changes that may break the installer. I'll post a comment when I've completed Linux and Windows test installs.

EDIT: Indeed, I need to go back to using the invoke-ai version of clipseg.

[lstein]

InvokeAI-installer-2.2.4-windows.zip InvokeAI-installer-2.2.4-mac.zip InvokeAI-installer-2.2.4-linux.zip Now working on Linux again. Windows test is pending.

[lstein]

@mauwii, @ebr My latest commit to this PR broke the "Build and push cloud image" action because I have removed the binary installer directory. Is it possible for this action to use the environments-and-requirements/requirements-lin-cuda.txt to install instead?

[lstein]

I fixed the dockerfile build. A fix for some Windows bat file cosmetic issues is on its way.

[lstein]

Ready for testing. Please download the zip file for your platform and follow the directions at: https://github.com/invoke-ai/InvokeAI/blob/lstein-simple-install/docs/installation/INSTALL_AUTOMATED.md

I'm pretty happy with how this works with one caveat. Currently, I have provided an update.sh script that will reinstall requirements and pull the latest InvokeAI changes from main. The InvokeAI code changes will be picked up, but there is no mechanism to update the requirements, because they are copied from the install directory into the runtime directory at install time. A future enhancement to consider is whether the update script shouldn't connect to the repo and download a new requirements file each time?

InvokeAI-installer-2.2.4-linux.zip InvokeAI-installer-2.2.4-mac.zip InvokeAI-installer-2.2.4-windows.zip

[lstein]

how are we handling updates, given the folder shift? will this be backward compatible?

This appears to be backward compatible. There are two potential gotchas. First, you can no longer set the location of the runtime directory by putting the --root argument in ~/.invokeai. Instead, you need to point invoke.py at the runtime directory by setting the INVOKEAI_ROOT environment variable, or use the --root argument.

The second is that the ~/.invokeai file is deprecated, and the script prefers invokeai.init in the runtime directory. If it can't find the latter it will use the legacy file and warn the user that it is deprecated.

[ebr]

Linux installer worked without a hitch. Windows installer unfortunately fails while trying to install the basicsr module. I was unable to fix it or get the full stack trace due to running it in a VM, but this was a clean Windows 10 VM, so at least that's a data point.

[lstein]

New version of windows installer using the binary installer's :err_exit function. InvokeAI-installer-2.2.4-windows.zip

[Skullfurious]

Out of curiosity, what exactly forces the install directory to be immutable?

[lstein]

Latest versions InvokeAI-installer-2.2.4-linux.zip InvokeAI-installer-2.2.4-mac.zip InvokeAI-installer-2.2.4-windows.zip for testing.

[tildebyte]

FWIW, my Windows test install of https://github.com/invoke-ai/InvokeAI/pull/1819#issuecomment-1340503828 can not only be moved (to a different drive, even), but even works when renaming the encompassing folder - all of which makes perfect sense to me, given that everything is now inside the install folder, and all paths (in scripts etc.) point "down" into subdirs of the install folder. I think people who are moving things around and seeing breakage must be using Conda or source installs

[lstein]

FWIW, my Windows test install of https://github.com/invoke-ai/InvokeAI/pull/1819#issuecomment-1340503828 can not only be moved (to a different drive, even), but even works when renaming the encompassing folder - all of which makes perfect sense to me, given that everything is now inside the install folder, and all paths (in scripts etc.) point "down" into subdirs of the install folder. I think people who are moving things around and seeing breakage must be using Conda or source installs

That's good news. Let me try on my system.... No, breaks. I get the dreaded "no module named ldm found" error.

For testing, here is the most recent set of zip files containing a number of fixes that I copied from the binary installer. InvokeAI-installer-2.2.4-linux.zip InvokeAI-installer-2.2.4-mac.zip InvokeAI-installer-2.2.4-windows.zip

[lstein]

Out of curiosity, what exactly forces the install directory to be immutable?

On the systems I've tested there seem to be absolute paths in the virtual environment. However @tildebyte says that he can move the runtime directory around without breaking anything, so I'm not sure what the difference is. Worth investigating, because it would be nice to move the invokeai directory around without breakage.

[lstein]

Updated zip files for testing... 10 December 2022, 6 pm EST InvokeAI-installer-2.2.4-linux.zip InvokeAI-installer-2.2.4-windows.zip InvokeAI-installer-2.2.4-mac.zip