feat: Add "Sort Text" Node

[thayol]

Adds a "Sort Text" node.

Node details

Name: Text Sort

Inputs:

• text: the text to sort
• split_separator: the token delimiter used in the prompt
• join_separator: the token delimiter to use in the new sorted prompt
• parse_token_groups: whether to handle weighted token groups or not

Outputs:

• STRING: the sorted prompt text

Why

Animagine XL recommends sorting the prompt.

My setup:

Token parsing

parse_token_groups = False

Not parsing tokens is a simple split operation followed by a custom sort that ignores the (token:weight) wrapping. Not parsing token groups results in the best effort for preserving malformed or unusual prompts.

There is limited support for tokens: only individually wrapped tokens are sorted correctly. Otherwise unrecognizable tokens are sorted without unwrapping, which in most cases means first because of a non-letter ASCII character.

Input

(masterpiece, best quality:1.15), anime, chromatic aberration, 
outdoors, field, (meadow:2), 
sidelighting, (slight smile:0.8), 
(broken,
medium hair, silver hair, (nested, (green:0.1) eyes:1.5), 
1girl, solo, 

Output

(broken, (masterpiece, (nested, 1girl, anime, best quality:1.15), chromatic aberration, field, (green:0.1) eyes:1.5), (meadow:2), medium hair, outdoors, sidelighting, silver hair, (slight smile:0.8), solo

---

parse_token_groups = True

Opting for token parsing may discard some of the prompt in exchange for proper (weighted) per-token sorting.

Known pitfalls:

• Nested weights (anything beyond the first level is discarded)
• Space (the character) as the split separator: the RegEx provided respects some whitespaces in a weird way
  • Input: e (g (d:2.0) f:3) a (c:1.1) b .
  • Output: . a b (c:1.1) (d:2.0) e (f:3) g (Note that g should have a weight of 3 and nesting is not recognized.)
• Broken encapsulation (stray parentheses and unrecognized weights are dropped)

Input

(masterpiece, best quality:1.15), anime, chromatic aberration, 
outdoors, field, (meadow:2), 
sidelighting, (slight smile:0.8), 
(broken,
medium hair, silver hair, (nested, (green:0.1) eyes:1.5), 
1girl, solo, 

Output

1girl, anime, (best quality:1.15), broken, chromatic aberration, (eyes:1.5), field, (masterpiece:1.15), (meadow:2), medium hair, nested, outdoors, sidelighting, silver hair, (slight smile:0.8), solo

[iiOxygen]

this has issues with loras and embeddings within the prompt. I believe one way to go on about it is by separating tokens that start with "embedding:" or "<lora" in lines #10212-10214 with something like this:

normal_tokens = [token for token in tokens if not (token.startswith("embedding:") or token.startswith("lora:"))]
special_tokens = [token for token in tokens if token.startswith("embedding:") or token.startswith("<lora")]

sorted_normal_tokens = sorted(normal_tokens, key=WAS_Text_Sort.token_without_weight)

sorted_tokens = sorted_normal_tokens + special_tokens
    return (join_separator.join(sorted_tokens), )

It has similar issues with prompt control. It'd be perfect if there is a way to combine both.

[WAS-PlaiLabs]

@thayol Do you think you can implement the changes to account for embedding?

[thayol]

I've been personally using it since and I have issues around the tokens being half-weighted or dropped. I should probably revisit the whole thing.

Any other pitfalls I should know of? Currently I know of: no weights, (full weight:2), half (weight:0.5), (default weight), embedding:file.pt, (:3:3) (colon in weight name)

[WAS-PlaiLabs]

Probably wildcards __some-wilcard__ and dynamic prompts {a|b|c}

[thayol]

If wildcards are evaluated after sorting (as in during CLIP encoding) then I believe that wouldn't be an issue. If dynamic prompts can have nested weights, it's probably something I'll have to look at separately.

Do we have unit tests? I feel like I could be more confident in the changes if there was something backing the WAS_Text_Sort class that isn't manually tested.

[WAS-PlaiLabs]

I haven't set up any unit tests -- actually, I don't know how. Other people have always set them up on projects and really wasn't involved beyond running the pipelines on new changes. 😬

[thayol]

After hours of thinking I decided to remove the token separating thing. It gets rid of the unsightly RegEx and it fixes half-weighted tokens. The only downside is that if you have multiple tokens in a weight group, they are kept together.

Also, ComfyUI claims that weights multiply with nesting but in my limited testing it only used the innermost group's weight with no attention to anything else.

[WASasquatch]

Thank you!