Introducing NXP Neutron runtime #10563

jirioc · 2025-04-29T22:36:48Z

Summary

Introducing NXP Neutron runtime.

cc @digantdesai @JakeStevens @robert-kalmar

pytorch-bot · 2025-04-29T22:36:52Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10563

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit 8a79f2d with merge base 91c9ffa ():

NEW FAILURES - The following jobs have failed:

trunk / test-arm-cortex-m-size-test / linux-job (gh)
RuntimeError: Command docker exec -t fc1641cb69fd5f601b5c58910570ac252979a67cc4a15408f0ea76ef3097c81e /exec failed with exit code 127
trunk / test-models-macos (ic4, xnnpack-quantization-delegation) / macos-job (gh)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

jirioc · 2025-04-30T14:08:39Z

@pytorchbot label "module: nxp" "release notes: nxp"

jirioc · 2025-04-30T14:09:11Z

cc @digantdesai @JakeStevens @robert-kalmar

tarun292 · 2025-05-02T22:13:21Z

@digantdesai are you the POC for reviewing this on our end?

backends/nxp/runtime/NeutronBackend.cpp

digantdesai

Is this still in progress?

At a high level looks OK to me.

digantdesai · 2025-05-06T04:46:13Z

backends/nxp/runtime/NeutronDriver.h

+/// - Initialize the Neutron Driver library, setting initial values, do memory
+/// allocation
+///   for internal data structures, do memory mapping.
+NeutronError neutronInit();


can these functions be in a neutron namespace?
Where are we calling this specific function to init the lib?

It is called in surrounding rtos code, much like ethosu_init

backends/nxp/runtime/NeutronBackend.cpp

digantdesai · 2025-05-06T04:47:57Z

backends/nxp/runtime/NeutronBackend.cpp

+
+// Applied on outputs.
+template <typename T>
+void transposeToChannelFirst(


OK for now, but I wonder is there another way to do this, i.e. set the dim_order on the output and let the portable or someone else take care of this?

From what @jirioc explained below, I think this could be removed by inserting a transpose node instead of setting a flag

digantdesai · 2025-05-06T04:48:12Z

backends/nxp/runtime/NeutronBackend.cpp

+
+// Applied on inputs.
+template <typename T>
+void transposeToChannelLast(


digantdesai · 2025-05-06T04:53:07Z

backends/nxp/runtime/NeutronBackend.cpp

+
+    auto* cfg = allocator->allocateInstance<NeutronConfig>();
+
+    // The following data is read from the "processed" data blob.


Nit: make this a helper function?

JakeStevens

LGTM, still outstanding seems to be:

namespace for neutron backend
consider removing NeutronDriver/NeutronErrors.h from ET repo and pull in to link separately like ARM does

backends/nxp/runtime/NeutronBackend.cpp

JakeStevens · 2025-05-12T13:29:38Z

backends/nxp/runtime/NeutronDriver.h

+/// - Initialize the Neutron Driver library, setting initial values, do memory
+/// allocation
+///   for internal data structures, do memory mapping.
+NeutronError neutronInit();


It is called in surrounding rtos code, much like ethosu_init

JakeStevens · 2025-05-12T13:32:33Z

backends/nxp/runtime/NeutronBackend.cpp

+
+// Applied on outputs.
+template <typename T>
+void transposeToChannelFirst(


From what @jirioc explained below, I think this could be removed by inserting a transpose node instead of setting a flag

digantdesai · 2025-05-12T15:55:09Z

Any update on this?

robert-kalmar · 2025-05-13T18:30:27Z

LGTM, still outstanding seems to be:
1. namespace for neutron backend

Thanks for advice, will do

2. consider removing NeutronDriver/NeutronErrors.h from ET repo and pull in to link separately like ARM does

So far we don't have the NeutronDriver/NeutronErrors.h. It is part of the ExecuTorch example in the MCUXpresso SDK but I can imagine to have it distributed alongside with the Neutron Converter, Neutron Functional Simulator or as a separated part of the SDK. Let us check internally and update as we discuss it internally.

digantdesai

I guess transpose is the only fix pending? I am ok with checking in header file for now.

jirioc · 2025-05-14T12:29:09Z

The namespace has been added.

JakeStevens · 2025-05-14T15:01:13Z

I guess transpose is the only fix pending? I am ok with checking in header file for now.

FWIW, it is how ARM does it currently as well.

executorch/backends/arm/runtime/EthosUBackend.cpp

If there are well supported tests for the runtime, this might be a good candidate for a task for new contributors interested in ET.

backends/nxp/CMakeLists.txt

robert-kalmar · 2025-05-15T07:30:37Z

I guess transpose is the only fix pending? I am ok with checking in header file for now.

FWIW, it is how ARM does it currently as well.

executorch/backends/arm/runtime/EthosUBackend.cpp

If there are well supported tests for the runtime, this might be a good candidate for a task for new contributors interested in ET.

Compared to Arm's approach we carry the transposition information with us in the Neutron Payload's header, as we get this by running the "tensor format inference" instead of Dim Orders.

digantdesai · 2025-05-22T04:57:45Z

Not sure I fully understand the need for transposing input/output tensors in the delegate runtime unless you want to be flexible to in terms of supporting both dim_orders, but ET doesn't support that i.e. it specializes on the dim_order at PTE generation time. So you can also use the portable op for transposing through a pass instead of this. But this might also work.

Also can we add at least a build test with armv8-m toolchain?

digantdesai · 2025-06-03T15:29:16Z

Let's re-merge the CI PR, this PR depends on that (touches trunk.yaml), then we can merge this. Thanks.

jirioc requested review from manuelcandales and swolchok as code owners April 29, 2025 22:36

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 29, 2025

pytorch-bot bot added module: nxp Issues related to NXP Neutron NPU delegation and code under backends/nxp/ release notes: nxp Changes to the NXP Neutron backend delegate labels Apr 30, 2025

JakeStevens reviewed May 5, 2025

View reviewed changes

backends/nxp/runtime/NeutronBackend.cpp Show resolved Hide resolved

backends/nxp/runtime/NeutronBackend.cpp Show resolved Hide resolved

backends/nxp/runtime/NeutronBackend.cpp Show resolved Hide resolved

digantdesai reviewed May 6, 2025

View reviewed changes

JakeStevens reviewed May 12, 2025

View reviewed changes

digantdesai reviewed May 14, 2025

View reviewed changes

jirioc requested review from jathu, larryliu0820 and kirklandsign as code owners May 14, 2025 12:26

jathu reviewed May 14, 2025

View reviewed changes

backends/nxp/CMakeLists.txt Outdated Show resolved Hide resolved

backends/nxp/CMakeLists.txt Outdated Show resolved Hide resolved

backends/nxp/CMakeLists.txt Outdated Show resolved Hide resolved

digantdesai approved these changes May 22, 2025

View reviewed changes

jirioc added 3 commits June 25, 2025 00:19

Fix sys/types.h file not found issue for IAR and MDK toolchains

068f11c

Adding NXP runtime.

a07e13c

Adding the build test for the neutron backend library.

8a79f2d

jirioc force-pushed the EIEX-289-upstream-neutron-runtime branch from 9534c72 to 8a79f2d Compare June 24, 2025 22:21

jirioc added the ciflow/trunk label Jun 25, 2025


		auto* cfg = allocator->allocateInstance<NeutronConfig>();

		// The following data is read from the "processed" data blob.

Introducing NXP Neutron runtime #10563

Are you sure you want to change the base?

Introducing NXP Neutron runtime #10563

Conversation

jirioc commented Apr 29, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

pytorch-bot bot commented Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10563

❌ 2 New Failures

Uh oh!

jirioc commented Apr 30, 2025

Uh oh!

jirioc commented Apr 30, 2025

Uh oh!

tarun292 commented May 2, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

digantdesai left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JakeStevens left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

digantdesai commented May 12, 2025

Uh oh!

robert-kalmar commented May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

digantdesai left a comment

Choose a reason for hiding this comment

Uh oh!

jirioc commented May 14, 2025

Uh oh!

JakeStevens commented May 14, 2025 • edited by digantdesai Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

robert-kalmar commented May 15, 2025 • edited by digantdesai Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

digantdesai commented May 22, 2025

Uh oh!

digantdesai commented Jun 3, 2025

Uh oh!

Uh oh!

jirioc commented Apr 29, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Apr 29, 2025 •

edited

Loading

robert-kalmar commented May 13, 2025 •

edited

Loading

JakeStevens commented May 14, 2025 •

edited by digantdesai

Loading

robert-kalmar commented May 15, 2025 •

edited by digantdesai

Loading