Description
Prerequisites
Please answer the following questions for yourself before submitting an issue.
- I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- I carefully followed the README.md.
- I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
- I reviewed the Discussions, and have a new bug or useful enhancement to share.
Feature Description
Add Ascend NPU as a new backend.
Motivation
Ascend is a full-stack AI computing infrastructure for industry applications and services based on Huawei Ascend processors and software. For more information about Ascend, see Ascend Community.
CANN (Compute Architecture of Neural Networks), developped by Huawei, is a heterogeneous computing architecture for AI.
Pytorch has officially announced support for Ascend NPU (through key PrivateUse1), please see the PrivateUse1 tutorial here.
Provide new backend support for llama.cpp, allowing users who are using Ascend NPU to inference model with llama.cpp.
Possible Implementation
Currently, the community has provided a convenient backend access mechanism. Ascend NPU is a CUDA-LIKE device, and I plan to reference CUDA's implementation to complete the Ascend NPU backend.
Due to the large workload, I plan to complete this feature in multiple stages. First, I will focus on compiling, backend registration, and device runtime functionalities. Additionally, I will add a new test file to validate backend registration, memory allocation, tensor operations, and other functionalities.
Next, I will proceed to implement tensor operators and validate them.
Afterward, do performance implementation, including split tensor support.
See also: very first commit #6035.