Add StableDiffusion3InstructPix2PixPipeline

[xduzhangjiayu]

What does this PR do?

Add StableDiffusion3InstructPix2PixPipeline
Would you like give a review? Many thanks~ @yiyixuxu @asomoza

[asomoza]

Hi and thanks for your contribution. Is there a model for this pipeline so I can test it?

[xduzhangjiayu]

Hi, thanks for the reply!
You can use model trained by myself from https://huggingface.co/CaptainZZZ/sd3-instructpix2pix/tree/main, you only need to replace the original transformer from official SD3. I have already tested the result and the result is reasonable, Or, for better performance, you can refer to another powerful model from https://huggingface.co/BleachNick/SD3_UltraEdit_freeform/tree/main/transformer
@asomoza

[asomoza]

I did a test but I get a bad result:

import torch

from diffusers.pipelines.stable_diffusion_3.pipeline_stable_diffusion_3_instruct_pix2pix import (
    StableDiffusion3InstructPix2PixPipeline,
)
from diffusers.utils import load_image


resolution = 1024
image = load_image("https://hf.co/datasets/diffusers/diffusers-images-docs/resolve/main/mountain.png").resize(
    (resolution, resolution)
)
edit_instruction = "Turn sky into a cloudy one"

pipe = StableDiffusion3InstructPix2PixPipeline.from_pretrained(
    "BleachNick/SD3_UltraEdit_freeform", torch_dtype=torch.float16
)

pipe.enable_model_cpu_offload()

edited_image = pipe(
    prompt=edit_instruction,
    image=image,
    height=resolution,
    width=resolution,
    guidance_scale=7.5,
    image_guidance_scale=1.5,
    num_inference_steps=30,
).images[0]

edited_image.save("edited_image.png")

and I get this image:

I also tried with changing the transformer model with yours but got the same result.

I don't have the time to look into this right now, can you solve the issue? Also ideally if this works, you will need to add the corresponding doc page and to be able to load it like the example in the dosctring from diffusers import StableDiffusion3InstructPix2PixPipeline you will need to add it to the __init__.py of the pipelines and the main diffusers one.

But still, the priority here should be to make it work and to demo an example with it.

[xduzhangjiayu]

Hi @asomoza
Sorry I didn't mention before that the model was trained on 512×512 images, so it is better to use 512×512 as the input.
I changed the following code and using my transformer model
resolution = 512 image = load_image("https://hf.co/datasets/diffusers/diffusers-images-docs/resolve/main/mountain.png").resize( (resolution, resolution) ) edit_instruction = "Turn sky into a sunny one"
And I got the image below:

And could you please tell me where I should add the doc page? Thanks~

[asomoza]

@xduzhangjiayu the code I used was the one that's in the docstring of the pipeline, so probably it's better to change it there.

And could you please tell me where I should add the doc page?

For the docs, it's inside the docs/source/en, you can learn from other implementations like this one .

But looking at the quality of the model and the work involved, I would recommend to move this pipeline to the community examples, it would be easier to do and also we can move it to core if it gets popular later, this way you don't have to write docs and we can merge it faster.

[xduzhangjiayu]

Hi @asomoza
I agree, and I already moved the pipeline to community, what else do I need to write for this PR? Thanks~

[asomoza]

@xduzhangjiayu thanks, can you please add a small description and a functional snipped of code to run it in the README file.

[xduzhangjiayu]

Hi @asomoza
Done! Please check~

[asomoza]

we don't use the same directory for hosting images, I took the liberty to upload your images to the hub here:

https://huggingface.co/datasets/diffusers/docs-images/resolve/main/StableDiffusion3InstructPix2Pix/mountain.png
https://huggingface.co/datasets/diffusers/docs-images/resolve/main/StableDiffusion3InstructPix2Pix/edited.png

can you delete the images from this PR and use these links instead please?

[xduzhangjiayu]

@asomoza Done

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[asomoza]

thanks, just some minor suggestions

[asomoza]

Suggested change

| Stable Diffusion 3 InstructPix2Pix Pipeline | Implementation of Stable Diffusion 3 InstructPix2Pix Pipeline | [Stable Diffusion 3 InstructPix2Pix Pipeline](#stable-diffusion-3-instructpix2pix-pipeline) | [![Hugging Face Models]()](https://huggingface.co/BleachNick/SD3_UltraEdit_freeform) [![Hugging Face Models]()](https://huggingface.co/CaptainZZZ/sd3-instructpix2pix) | [Jiayu Zhang](https://github.com/xduzhangjiayu) and [Haozhe Zhao](https://github.com/HaozheZhao)|

| Stable Diffusion 3 InstructPix2Pix Pipeline | Implementation of Stable Diffusion 3 InstructPix2Pix Pipeline | [Stable Diffusion 3 InstructPix2Pix Pipeline](#stable-diffusion-3-instructpix2pix-pipeline) | [](https://huggingface.co/BleachNick/SD3_UltraEdit_freeform) [](https://huggingface.co/CaptainZZZ/sd3-instructpix2pix) | [Jiayu Zhang](https://github.com/xduzhangjiayu) and [Haozhe Zhao](https://github.com/HaozheZhao)|

[asomoza]

Suggested change

	### Result
	
	
	|Original|Edited|
	|---|---|
	||

[xduzhangjiayu]

@asomoza OK, done

[asomoza]

@bot /style

[github-actions]

Style fixes have been applied. View the workflow run here.

[asomoza]

thanks!