Does it support Pixart-delta controlnet in ComfyUI?

[carvychen]

[city96]

Not yet!

To be fair, I totally missed that this got released. I'll see how hard it is to add. It seems to use hed edge as the input, which seems to be available here already, so I think I only need to add the model?

[carvychen]

I think so, yes! Both the preprocessor and the Pixart-delta version of ControlNet are already available. Perhaps what remains is to implement the inference code for ComfyUI.

[city96]

I did a quick test and it kinda works but I think it expects the CN input in a different format than what the controlnet aux repo outputs. I'll have to ask the pixart dev about it.

Uploaded the code in the meantime in case you wanna mess around with it. I'm using the model files from here.

ControlNetPixArt.json (just replace the load cond nodes with T5, I just use those for testing so I don't have to load T5 every time I make a change).

[nikolaiusa]

Am I doing something wrong?)

[city96]

@nikolaiusa The output of the default HED preprocessor is too "bright" for PixArt as an input. Check out my example above, I am combining it with an empty (black) image at 0.01% to make it work.

[kno10]

I tried reproducing this (to see if i can get QR codes to work with PixArt-alpha). However, I always get the error

  File "ComfyUI/custom_nodes/ComfyUI_ExtraModels/PixArt/models/pixart_controlnet.py", line 80, in forward_c
    pos_embed = torch.from_numpy(get_2d_sincos_pos_embed(self.pos_embed.shape[-1], (self.h, self.w), lewei_scale=self.lewei_scale, base_size=self.base_size)).unsqueeze(0).to(c.device).to(self.dtype)
...
AttributeError: 'PixArtMS' object has no attribute 'lewei_scale'

hence, I removed lewei_scale=self.lewei_scale as this variable does not appear to be mentioned anywhere. Now it runs, but I get garbage output.

The workflow for PixArt-XL-2-1024-MS.pth is working, now I switch to the PixArt-XL-2-1024-ControlNet.pth checkpoint, to ControlPixArtMSHalf and connect the PixArtControlNetConditioning instead of the text only conditioning, but it breaks.

[city96]

@kno10 lewei_scale is now called pe_interpolation in the new code that came with Sigma. I didn't uprade the controlnet parts since they were in a half-working state anyway, and sigma didn't seem to come with updated files for that.

You can send a quick PR if just replacing those gets it working again, I'd be happy to merge it.

[kno10]

replacing lewei_scale with pe_interpolation helped (no longer looks completely corrupted, but you can recognize the outlines from the input image), but the results are still far from useful.