Can inference be done at FP8? for both 1K and 2K models

[FurkanGozukara]

People asking me to further reduce VRAM usage.

Currently 1K model uses 8.7 GB minimum with VAE offloading.

If we could do inference at FP8 that would reduce VRAM usage significantly

I am using official SANA pipeline shared here

[lawrence-cj]

There will be int4 version of Sana released soon.

[FurkanGozukara]

There will be int4 version of Sana released soon.

Awesome I am following ❤️

[xieenze]

for fp8 you can use torchao toolbox

[FurkanGozukara]

for fp8 you can use torchao toolbox

how to use with current pipeline here?

[lawrence-cj]

https://github.com/huggingface/diffusers/blob/main/docs/source/en/api/pipelines/sana.md#quantization

@FurkanGozukara This is what you are looking for.

[FurkanGozukara]

@lawrence-cj thanks

is diffusers ready right now with all pull requests you made?

[lawrence-cj]

What pull request are we talking about?

[FurkanGozukara]

@lawrence-cj i see none of these are closed they are not merged yet i assume?

huggingface/diffusers#10523

huggingface/diffusers#10510

[lawrence-cj]

They are just opened within 24 hours. No hurry.

[Shiven-saini]

waiting for int4 version, will be a game changer! Any way i can contribute to the quantization process ?