InvokeAI icon indicating copy to clipboard operation
InvokeAI copied to clipboard

feat(ui): graph builder

Open psychedelicious opened this issue 1 year ago • 0 comments

Summary

This PR adds internal graph-building utilities to the UI and uses them for the generation tab graph builders.

Python changes

  • To support the type hints, an InvocationOutputMap object is added to the OpenAPI schema. This maps invocation types to their output classes.
  • To simplify LoRA loading, 3 new nodes are added. These allow LoRAs to be added via collect nodes, instead of requiring chaining. The old chaining nodes haven't been removed. Current chaining method: image New collect method: image
    • LoRASelectorInvocation: Select a LoRA model and provide its weight.
    • LoRACollectionLoader/SDXLLoRACollectionLoader: Given a collection of LoRAs, the UNet model and the CLIP model(s), adds all the LoRAs to the models.

Frontend Changes

A stateful Graph class abstracts adding nodes and edges to graphs. Its methods have pretty solid type safety and hinting. It's also far more compact and readable. It is fully tested via vitest coverage - which was added in this PR.

https://github.com/invoke-ai/InvokeAI/assets/4822129/08eb48c0-61d0-4887-a966-0aade2db3dea

Some handy new types (finally) allow for inspection of a node's output fields:

https://github.com/invoke-ai/InvokeAI/assets/4822129/be346a5b-c73f-4010-9694-74a8d714dc8e

A MetadataUtil class manages metadata for a graph using the new graph methods. Its API is basically the same as the old metadata helpers.

The Generation tab graphs have been rewritten using the new utilities. Canvas graphs are untouched.

The new utilities require the passing of the nodes themselves to graph building helpers - as opposed to node IDs. This simplifies the graph building logic and allows for type hints when connecting edges and such. For example, compare the new seamless helper with the old one. It's half the size and is type safe!

Because the Graph helper provides a generic abstraction for building graphs, its API could have a separate implementation that builds workflows. This is one step closer to the intuitive internal API where we send workflows instead of graphs to the backend.

TODO

  • [ ] Revise docstrings for the Graph class - I changed the API at some point and haven't fixed the docstrings
  • [ ] More thorough testing. As far as I can tell, the generation tab is fully working - all features work in all combinations I have tested. But it's very possible I've missed some combination.
  • [ ] Explore adding larger-scale tests for the actual graph outputs. vitest does support a snapshot mode, there's probably some way to grab different redux states and test the full graph building logic. TBH I'm not sure how useful this would be, because when we update the graph, we'll need to redo the test snapshot anyways...

Related Issues / Discussions

n/a

QA Instructions

Once out of draft - test all combinations of settings and features on the generation tab. I mostly expect loud failures for things that aren't correct.

Merge Plan

n/a

Checklist

  • [x] The PR has a short but descriptive title, suitable for a changelog
  • [x] Tests added / updated (if applicable)
  • [x] Documentation added / updated (if applicable)

psychedelicious avatar May 13 '24 12:05 psychedelicious