UnivAICharGen icon indicating copy to clipboard operation
UnivAICharGen copied to clipboard

Use Git LFS or similar for embeddings

Open Omegastick opened this issue 2 years ago • 3 comments

With the embeddings recently added to the repo, it's now a 619.84 MB download to clone, the vast majority of which are embeddings. I use Umi-AI for the (fantastic) wildcard system. The embeddings are not something that I (and presumably a fair amount of other users) am that interested in using, but they are slowing down my automated builds and bloating my Docker images pretty significantly.

Is it possible to use Git LFS to store them, or perhaps move them to a separate repository?

Omegastick avatar Feb 06 '23 07:02 Omegastick

Embeddings themselves aren't even big. The entire Embeddings folder is, at most, like 10 MBs. It's the preview images that gobble up space.

But the fancy new browser inside Auto's Repo pulls directly from the folder. Unless that changes, we don't have a way to make the previews linkable properly...

I am planning to host offsite previews at full quality, but it will be an extra click for most people, and not nearly as navigable as I'd like. Frankly, there's no easy solution for this problem.

Klokinator avatar Feb 06 '23 17:02 Klokinator

@Omegastick sorry for responding so late, Klok didn't ping me about this one.

There's one kinda simple solution:

  1. Fork this repo
  2. Remove all preview images if you don't want them
  3. In your automatic build make sure to install extensions with git clone --depth 1 ... to prevent it from fetching the embedding commits
  4. If there's an update, pull it from upstream to your fork. I don't think we'll do another batch of embeddings so it should not conflict on deleted files. We also don't update this extension that often so keeping up to date shouldn't be a hassle.

Alternatively tell me how you resolved this :P. Keeping the issue open because I'm curious.

ctwrs avatar Jul 22 '23 21:07 ctwrs

I will likely be removing all the preview images in the Umi 2.0 update. That will take a while, but not TOO long. Less than a month from now I bet.

It should also fix all the existing issues Umi AI has.

Klokinator avatar Jul 24 '23 14:07 Klokinator