Any plan to support Qwen2.5-Omni for AWQ quantization?

[ykkk1]

Is AWQ quantization officially supported for Omni models?
If not, is there any plan or timeline for releasing the AWQ-compatible quantization code for Qwen2.5-Omni?Really appreciate your help and looking forward to your updates!

[brian-dellabetta]

Hi @ykkk1 , it looks like Qwen/Qwen2.5-Omni-7B has the same architecture as the other Qwen models that have run successfully with AWQ, so it should work fine with the default mappings. You just need to make sure to exclude the audio and vision components from quantization, as they are highly sensitive to quantization loss. Just the textual part will be quantized in this case:

recipe = [
    AWQModifier(
        targets="Linear",
        scheme="W4A16", # or "W4A16_ASYM" for asymmetric, which adds complexity but usually performs better
        ignore=["lm_head", "re:.*audio_tower.*", "re:.*visual.*"],
    ),
]

[ykkk1]

Hi again, thanks for the previous response!

I tried quantizing Qwen2.5-Omni-7B with AWQ using the recipe you suggested (ignoring the audio_tower and visual modules), but I'm running into the following error during tracing:
I don't know why, could it be that Omni uses generate() rather than forward()

[brian-dellabetta]

Hi @ykkk1 , I tried and hit the same error with

model_id = "Qwen/Qwen2.5-Omni-7B"
model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
    model_id, torch_dtype="auto"
)
processor = Qwen2_5OmniProcessor.from_pretrained(model_id, trust_remote_code=True)

though i had to add to llm-compressor/src/llmcompressor/utils/helpers.pyL1001 this conditional:

+        # QwenOmni
+        elif (
+            hasattr(model.config, "thinker_config")
+            and hasattr(model.config.thinker_config, "text_config")
+            and hasattr(model.config.thinker_config.text_config, "use_cache")
+        ):
+            self.config = model.config.thinker_config.text_config

I'm also hitting a tracing error with Qwen2_5OmniThinkerForConditionalGeneration.

This is a pretty complicated model -- audio, vision, text, and talking components. It would fail out for any sequential algorithm, not specific to AWQ. We have some other priorities at the moment, but I will ask around internally to see if there is an easy fix

[brian-dellabetta]

Hi @ykkk1 , I had some time to look into this again today. VLLM only supports running the thinker portion of the model, see docs here. With that in mind, I am able to run AWQ on the thinker model (which excludes the talker component of Qwen 2.5 Omni). My script is attached below. Hope this helps!

AWQ on Qwen Omni 2.5 Thinker only model:

import base64
from io import BytesIO

import torch
from datasets import load_dataset
from qwen_vl_utils import process_vision_info
from transformers import (
    Qwen2_5OmniProcessor,
    Qwen2_5OmniThinkerForConditionalGeneration,
)

from llmcompressor import oneshot
from llmcompressor.modifiers.awq import AWQModifier
from llmcompressor.utils import dispatch_for_generation

MODEL_ID = "Qwen/Qwen2.5-Omni-7B"
OUTPUT_DIR = MODEL_ID + "-w4a16"

model = Qwen2_5OmniThinkerForConditionalGeneration.from_pretrained(
    MODEL_ID,
    device_map="auto",
    torch_dtype="auto",
)
processor = Qwen2_5OmniProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)

# Oneshot arguments
DATASET_ID = "lmms-lab/flickr30k"
NUM_CALIBRATION_SAMPLES = 256
DATASET_SPLIT = f"test[:{NUM_CALIBRATION_SAMPLES}]"
MAX_SEQUENCE_LENGTH = 512

# Load dataset and preprocess.
ds = load_dataset(DATASET_ID, split=DATASET_SPLIT)
ds = ds.shuffle(seed=42)


# Apply chat template and tokenize inputs.
def preprocess_and_tokenize(example):
    # preprocess
    buffered = BytesIO()
    example["image"].save(buffered, format="PNG")
    encoded_image = base64.b64encode(buffered.getvalue())
    encoded_image_text = encoded_image.decode("utf-8")
    base64_qwen = f"data:image;base64,{encoded_image_text}"
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image", "image": base64_qwen},
                {"type": "text", "text": "What does the image show?"},
            ],
        }
    ]
    text = processor.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )
    image_inputs, video_inputs = process_vision_info(messages)

    # tokenize
    return processor(
        text=[text],
        images=image_inputs,
        videos=video_inputs,
        padding=False,
        max_length=MAX_SEQUENCE_LENGTH,
        truncation=True,
    )


ds = ds.map(preprocess_and_tokenize, remove_columns=ds.column_names)


# Define a oneshot data collator for multimodal inputs.
def data_collator(batch):
    assert len(batch) == 1
    return {key: torch.tensor(value) for key, value in batch[0].items()}


recipe = AWQModifier(
    targets="Linear",
    scheme="W4A16",
    ignore=[
        "re:.*lm_head",
        "re:.*audio_tower.*",
        "re:.*visual.*",
    ],
)


oneshot(
    model=model,
    processor=processor,
    recipe=recipe,
    dataset=ds,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
    data_collator=data_collator,
)


# Confirm generations of the quantized model look sane.
print("========== SAMPLE GENERATION ==============")
dispatch_for_generation(model)
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "http://images.cocodataset.org/train2017/000000231895.jpg",
            },
            {"type": "text", "text": "Please describe the animal in this image\n"},
        ],
    }
]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[prompt],
    images=image_inputs,
    videos=video_inputs,
    padding=False,
    max_length=MAX_SEQUENCE_LENGTH,
    truncation=True,
    return_tensors="pt",
).to("cuda")
output = model.generate(**inputs, max_new_tokens=100)
print(processor.decode(output[0], skip_special_tokens=True))
print("==========================================")

model.save_pretrained(OUTPUT_DIR)
processor.save_pretrained(OUTPUT_DIR)