invoke-ai
feat(ui): improve progress image UX
What type of PR is this? (check all applicable)
- [x] Refactor
- [x] Feature
- [x] Bug Fix
- [x] Optimization
- [ ] Documentation Update
- [ ] Community Node Submission
Description
There is a gap between when the last progress image is displayed, and when the output image is displayed. The last generated image (or the blank bg) is displayed during this gap.
https://github.com/invoke-ai/InvokeAI/assets/4822129/b057ad1d-04fd-4534-9290-1ac2ccc41108
Why does this happen
Pre-nodes, every generation had the same artifacts - a series of progress images, then a single output image. This guaranteed sequence meant we could easily present the progress images as "becoming" the output image.
With nodes, a graph can have any combination and sequence of denoise progress and image outputs. It can have 5 denoise nodes and zero image outputs, or the other way round.
The app displays progress for the currently executing node, which may or may not be a denoising node. When the node finishes executing, its progress (if any) is cleared. This is accurate in that it presents progress as being associated with a node, not the whole graph.
What you see:
- Denoising starts, progress images are displayed
- Denoising stops, progress images are cleared and we fall back to displaying the last image - this is the gap
- An output image is received and displayed
While it is accurate, it is not what users want to see. Users want to see the progress images "become" the output image, without the gap.
Fixing it
We can create the illustion of a the progress image "becoming" the output image by waiting to clear the progress image until the output is ready to be displayed. This is a bit trickier than you might imagine:
- Display progress images as they are received.
- When an image is received, we check if it is from the same session as the last progress image. If the session id matches, we can assume that this image is the one the progress should "become".
- Download the image (if we switch to this image before it is downloaded, we get a flash of blank bg)
- Swap the progress image out for the new image.
We should stop showing the progress image in some other situations, too - like if the current queue item is canceled.
Canvas needs additional logic to only show the progress for canvas graphs.
Implementation
There's a new slice progress that manages all the progress state needed to create the illusion. It tracks just enough data about the currently executing node to determine whether to show the progress image or output image.
To make this work, I split image viewer area into 3 modes - "image", "info" and "progress". This is actually very similar to the current UX.
The progress mode only shows the progress and the latest image received. When you click a gallery image, it switches to "image" mode. "info" mode shows the metadata for the currently selected image. I'm not totally happy with how this works.
https://github.com/invoke-ai/InvokeAI/assets/4822129/b2a6ac9f-3cc4-4990-8767-4ca019db5667
Canvas and workflows still work exactly like they did before. Canvas needs to have the same strategy applied but I ran out of steam.
Workflow linear views
We are working on getting the workflow linear view to be a first-class citizen and present it like the linear tabs.
It won't be feasible to maintain the illusion perfectly for workflow linear views, because we have no guarantees about a user's graph. But, the general concept in this PR can be applied to get the illusion correct most of the time.
I'm not happy with this PR yet but I've spent enough time on it for now.
After thinking about this problem more today, I think the app should just work like the workflow editor. For each session, it stores every node's execution state and progress.
Then we could use this simpler logic: If the currently executing node has progress images coming in, display them. When that node completes, we continue displaying the last progress image received until A) an image output is received or B) more progress images are received from another node.
This could probably use/share the code from the workflow editor.
This “bug” is kinda useful when you are quickly iterating and you can see the difference from the last generation for a second.
