Closed
Description
We recently integrated PAG into diffusers! See this PR [here] (#7944) we added PAG to SDXL
we also want to add PAG support to SD1.5 pipelines! we will need:
- StableDiffusionPAGPipeline (assigned to @shauray8, PR add PAG support for SD architecture #8725)StableDiffusionPAGImg2ImgPipeline add PAG support for SD Img2Img #9463StableDiffusionPAGInpaintPipelineStableDiffusionControlNetPAGInpaintPipeline (Add PAG support to StableDiffusionControlNetPAGInpaintPipeline #8875)StableDiffusionControlNetPAGPipeline (assigned to @tuanh123789 )StableDiffusionControlNetPAGImg2ImgPipeline (assigned to @Bhavay-2001 add PAG support for SD Controlnet Img2Img #8864)
- You should put it under the pag folder
- you can use the implementation of SDXL PAG pipelines as a reference (see this PRhttps://github.com/add PAG support #7944 and you can find all the sdxl pag pipelines here https://github.com/huggingface/diffusers/tree/main/src/diffusers/pipelines/pag)
- you need to add AutoPipeline so that you can use this API to create it
AutoPipelineForImage2Image.from_pretrained(repo_id, controlnet=controlnet, enable_pag=True ...)
- tests and docs
If you are interested in working on this, Let me know which pipeline(s) you want to work on:)
Metadata
Metadata
Assignees
Type
Projects
Milestone
Relationships
Development
Participants
Activity
StableDiffusionXLControlNetPAGImg2ImgPipeline#8700blaisedmello commentedon Jun 26, 2024
Hello, I would like to have a go at it. Can I work on the StableDiffusionPAGInpaintPipeline?
yiyixuxu commentedon Jun 26, 2024
@blaisedmello sure!
shauray8 commentedon Jun 26, 2024
Looks rather interesting, adding support for StableDiffusionPAGPipeline.
sajadn commentedon Jun 26, 2024
Hey, I'll work on
StableDiffusionPAGImg2ImgPipelinetuanh123789 commentedon Jun 27, 2024
Hi, can i take
StableDiffusionControlNetPAGPipelineyiyixuxu commentedon Jun 27, 2024
@tuanh123789 sure!
Bhavay-2001 commentedon Jun 27, 2024
Hi @yiyixuxu, I think I can work on
StableDiffusionControlNetPAGImg2ImgPipeline. Pls assign this to me.yiyixuxu commentedon Jun 27, 2024
@Bhavay-2001 sure!
AbhinavJangra29 commentedon Jun 29, 2024
@yiyixuxu can i try ?
3 remaining items
darshil0805 commentedon Aug 17, 2024
Hi, I'd like to work on StableDiffusionPAGInpaintPipeline if it's still open
yiyixuxu commentedon Aug 17, 2024
@darshil0805 sure!
darshil0805 commentedon Sep 7, 2024
Hi @yiyixuxu , @a-r-r-o-w. I have opened a PR for SD Inpainting PAG Pipeline, can you review and help me to test this pipeline?
crapthings commentedon Sep 17, 2024
how do i know the layer names?
only
up mid down?
yiyixuxu commentedon Sep 17, 2024
cc @a-r-r-o-w, maybe we should update doc? (did we?)
a-r-r-o-w commentedon Sep 17, 2024
The correct usage is to look at either the modeling code, or state dict keys, to see what the layer naming looks like. Then you can specify either the full layer names or partial layer names as valid regular expressions when initializing the pipeline (via
pag_applied_layers). For more advanced usage, you can take a look at helper methods in the PAGMixin.See the docs and examples usages here.
TLDR; Our naming convention follows the pattern of
something_blocks.whateverorsomething_blocks.[layer_index}orblocks.{layer_index}ortransformer_blocks.{layer_index}, so the easiest way to enable them without looking into the state dict or modeling code is to specify something likepag_applied_layers="blocks.(10|20|42)"orpag_applied_layers="mid"I think we did, but I noticed that the introduction docs are out of sync with the current usage, and I'm not sure if the current example works. I'll test and open a PR soon
saikomals commentedon Sep 19, 2024
HI @yiyixuxu I'd like to work on
StableDiffusionPAGImg2ImgPipelineif it's still open. Thanksyiyixuxu commentedon Sep 20, 2024
@airookie17
I think I'm about to merge in this #9463