[Feature request] Any plans for AMD XDNA AI Engine support on Ryzen 7x40 processors?

[KarmaMonk]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

---

Enhancement

Are there any plans to support the AMD XDNA AI Engine (in AMD Ryzen 7x40 (x = 6,8,9) processors)?

[AlphaAtlas]

Is there any API or example code of how to use it out in the wild?

I can't even find a whitepaper or anything like that.

[KarmaMonk]

What I found so far are is:

• XDNA is in the Ryzen AI 7040 series and in the Alveo V70 AI inference acceleration card.
• CDX Driver and CDX Controller for Linux to push code to the XDNA AI Engine
• Older infos from XILINX about their AI Engine, which probably will be similar/same to the Ryzen XDNA AI Engine
• https://twitter.com/ogawa_tter/status/1655131300058267648

[SlyEcho]

If there is a BLAS or even a matrix multiplication API available, it should be easy to add.

[Azeirah]

Do the non-mobile ryzen 7x00 cpus also have this feature?

[KarmaMonk]

Do the non-mobile ryzen 7x00 cpus also have this feature?

As far as i know: no, only the mobile Phoenix 7040 Series, see wikipedia

[GreyXor]

https://github.com/amd/RyzenAI-cloud-to-client-demo

[Dampfinchen]

I second this. With the rise of ML accelerators in PCs starting with Ryzen AI and Meteor Lake VPU's, using them might result in big efficiency gains and speedups.

I'm also sure once memory bottlenecks are reduced, more can be done by using tensor cores and the new RDNA3 AI accelerators more efficiently.

Then, you also have NPUs in modern smartphone that can be leveraged. Hardware acceleration is the way forward.

[AlphaAtlas]

An issue is that inference either has to totally be on the XPU (excluding the possibility of partial OpenCL/CUDA acceleration), or support zero copy/unified memory to avoid cost prohibitive copies. IGP acceleration is similarly problematic.

Its theoretically not impossible... For instance, Tencent ncnn has some kind of Vulkan zero copy mechanism for "unified memory" devices.

I think the library AMD uses for the demo is this: https://github.com/Xilinx/Vitis-AI

[ggerganov]

Don't know the specifics of this hardware, but most likely you need to follow the steps in #1642 to make it work

[EwoutH]

AMD's Unified Inference Frontend (UIF) might also be useful.

[audiovention]

Here's some more information:
https://onnxruntime.ai/docs/execution-providers/Vitis-AI-ExecutionProvider.html
This onnxruntime provider is what the only AMD-provided example uses. Things seem very early stage at this point (and considering AMD's history might forever remain in that stage).
I suspect it's more likely these accelerators would have to be utilized similarly to the Apple neural engine.

[marty1885]

Second this. I'm willing to look into and actually writing it. But where TF is the API?

[EwoutH]

Second this. I'm willing to look into and actually writing it. But where TF is the API?

I think it's here: https://www.xilinx.com/products/design-tools/vitis/vitis-ai.html

[marty1885]

@EwoutH Ohh.. thanks. Dang I used to do work with Vitis in collage. It'll be quite a task to gain access to low level API on that.

Update: Waiting for Acer to release the new Swift edge in my country.

[EwoutH]

There is now Ryzen AI Software Platform documentation: GitHub, ReadTheDocs.

The Ryzen AI cloud-to-client demo has also been updated over the last weeks.