Enable xpu device #1736
Enable xpu device #1736
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @jiqing-feng, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request introduces support for XPU devices, allowing models to be dispatched to Intel's XPU if available, alongside existing CUDA support. It also updates an example script to dynamically assign tensors to the model's active device rather than hardcoding 'cuda'. The author notes that calibration on XPU is currently slower than CUDA, and the PR is a draft pending resolution of this performance issue.
Highlights
- XPU Device Compatibility: Added support for dispatching models to XPU devices, enhancing hardware compatibility.
- Flexible Device Assignment: Modified llama3_example.py to use model.device for tensor placement, improving adaptability across different hardware setups.
- Enhanced Device Dispatch Logic: Updated the dispatch_for_sequential helper to prioritize CUDA, then XPU, before falling back to CPU, providing broader device support.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
|
👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review. Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed. |
There was a problem hiding this comment.
Code Review
This pull request introduces support for Intel XPU devices, which is a valuable addition for broader hardware compatibility. The changes correctly implement the detection and dispatching to XPU devices, and the example file is updated to be device-agnostic. I have one suggestion to refactor the device selection logic to improve code clarity and reduce repetition. Otherwise, the changes look good.
There was a problem hiding this comment.
Thanks for the contribution! It does seem like that's the only place in src where we explicitly call torch.cuda.is_available, so this looks right to me. We don't have access to intel xpu devices at the moment, can you confirm this is working for you?
One suggestion about changing the hardcoded "cuda" string in all the examples, I can do that as well since it appears in a number of places
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Yes, I've verified it on XPU. |
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
| @@ -22,7 +22,7 @@ | |||
| compressed_model = AutoModelForCausalLM.from_pretrained( | |||
| MODEL_STUB, | |||
| torch_dtype="auto", | |||
| device_map="cuda:0", | |||
| device_map="auto", | |||
There was a problem hiding this comment.
will confirm with team that this is what we want here.
This PR enables gptq example on Intel XPU