-
Notifications
You must be signed in to change notification settings - Fork 308
Allow weights_only=True
load for gemlite layout
#2081
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Summary: This PR adds a few imports from gemlite so that the gemlite checkpoint can be loaded with weights_only = True in huggingface (which is the default) `torch.load(gemlite_checkpoint, weights_only=True` Note: we need to remove `getattr` in the future Test Plan: ``` import torch from transformers import AutoModelForCausalLM, AutoTokenizer, TorchAoConfig model_id = "jerryzh168/phi4-mini-int4wo-gemlite" quantized_model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", torch_dtype="auto") tokenizer = AutoTokenizer.from_pretrained(model_id) prompt = "Hey, are you conscious? Can you talk to me?" inputs = tokenizer(prompt, return_tensors="pt").to("cuda") generated_ids = quantized_model.generate(**inputs, max_new_tokens=128) output_text = tokenizer.batch_decode( generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False ) print(output_text) ``` Reviewers: Subscribers: Tasks: Tags:
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/2081
Note: Links to docs will display an error until the docs builds have been completed. ❗ 1 Active SEVsThere are 1 currently active SEVs. If your PR is affected, please view them below: ✅ You can merge normally! (1 Unrelated Failure)As of commit e91f88d with merge base 34421b1 ( FLAKY - The following job failed but was likely due to flakiness present on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
from gemlite.core import DType, GemLiteLinearTriton | ||
|
||
# TODO: we need to remove `getattr` since it's unsafe (by picklescan) | ||
torch.serialization.add_safe_globals([DType, GemLiteLinearTriton, getattr]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since hf flags this, would it make sense to wait until we see whether gemlite can remove the need for getattr
on this one
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can remove this part, I fixed this issue in #2096
no longer needed since the serialization issue is fixed in #2096 |
Summary:
This PR adds a few imports from gemlite so that the gemlite checkpoint can be loaded with weights_only = True in huggingface (which is the default)
torch.load(gemlite_checkpoint, weights_only=True
Note: we need to remove
getattr
in the futureTest Plan:
Reviewers:
Subscribers:
Tasks:
Tags: