Add aot example with Neutron Backend #10871

robert-kalmar · 2025-05-14T11:40:54Z

Summary

This PR add a AoT example with the eIQ Neutron Backend. The Backend is demonstrated on tiny CNN model named CifarNet, trained on Cifar10 dataset, which is part of the PR.

Test plan

Manual testing, executing the example based on steps in the Readme.md and validating the PTE on i.MX RT700 platform with the Neutron Backend runtime.

Resolves #10898

cc @digantdesai @JakeStevens , @JakeStevens , @skywall , @jirioc

pytorch-bot · 2025-05-14T11:40:58Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10871

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Unrelated Failure

As of commit c7d4b49 with merge base 77f16dc ():

NEW FAILURE - The following job has failed:

Apple / build-demo-ios / macos-job (gh)
Library not loaded: @rpath/liblz4.1.dylib

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

trunk / test-arm-cortex-m-size-test / linux-job (gh) (trunk failure)
/pytorch/executorch/backends/arm/runtime/EthosUBackend.cpp:460:74: error: array subscript 4 is above array bounds of 'int [4]' [-Werror=array-bounds=]

This comment was automatically generated by Dr. CI and updates every 15 minutes.

robert-kalmar · 2025-05-14T11:57:45Z

@pytorchbot label "module: nxp" "release notes: nxp"

pytorch-bot · 2025-05-14T11:57:48Z

Didn't find following labels among repository labels: ,,label

JakeStevens · 2025-05-14T15:04:02Z

examples/nxp/aot_neutron_compile.py

+        action="store_true",
+        required=False,
+        default=False,
+        help="Flag for producing ArmBackend delegated model",


Suggested change

help="Flag for producing ArmBackend delegated model",

help="Flag for producing NeutronBackend delegated model",

JakeStevens · 2025-05-14T15:07:15Z

examples/nxp/aot_neutron_compile.py

+        model, example_inputs, strict=True
+    )
+
+    # TODO: Add Neutron ATen Passes, once https://github.com/pytorch/executorch/pull/10579 is merged


nit: file a task so we can track and not lose this

#10579 is now merged!

JakeStevens · 2025-05-14T15:13:06Z

examples/nxp/aot_neutron_compile.py

+                "_portable_lib.cpython* using --portable_lib CLI options. \n"
+                "This is required for running quantized models with unquantized input."
+            )
+            sys.exit(-1)


Can you either: (1) just not sys.exit entirely and let it fail loudly later when it will hit the runtime exception or (2) add a CLI arg to allow skipping this part-- and the part below for the torch.loads

In internal infra, these libraries are loaded a slightly different way and I do not actually pass the .so on command line, and it is not loaded a few lines below.

✅
Ok, so reverted back to our original solution. There is only a warning raised and normally fails later when exporting to ExecuTorch Program:

# 6. Export to ExecuTorch program try: exec_prog = edge_program.to_executorch( config=ExecutorchBackendConfig(extract_delegate_segments=False) ) except RuntimeError as e: if "Missing out variants" in str(e.args[0]): raise RuntimeError( e.args[0] + ".\nThis likely due to an external so library not being loaded. Supply a path to it with the " "--portable_lib flag." ).with_traceback(e.__traceback__) from None else: raise e

JakeStevens · 2025-05-14T15:14:23Z

examples/nxp/experimental/cifar_net/cifar_net.py

+        x = self.conv3(x)
+        x = self.pool2(x)
+
+        # The output of the previous MaxPool has shape [batch, 64, 4, 4] ([batch, 4, 4, 64] in TFLite). When running


Suggested change

# The output of the previous MaxPool has shape [batch, 64, 4, 4] ([batch, 4, 4, 64] in TFLite). When running

# The output of the previous MaxPool has shape [batch, 64, 4, 4] ([batch, 4, 4, 64] in Neutron IR). When running

JakeStevens · 2025-05-14T15:14:35Z

examples/nxp/experimental/cifar_net/cifar_net.py

+        x = self.pool2(x)
+
+        # The output of the previous MaxPool has shape [batch, 64, 4, 4] ([batch, 4, 4, 64] in TFLite). When running
+        #  inference of the `FullyConnected`, TFlite will automatically collapse the channels and spatial dimensions and


Suggested change

# inference of the `FullyConnected`, TFlite will automatically collapse the channels and spatial dimensions and

# inference of the `FullyConnected`, Neutron IR will automatically collapse the channels and spatial dimensions and

skywall · 2025-05-15T09:40:22Z

examples/nxp/aot_neutron_compile.py

+    parser.add_argument(
+        "-p",
+        "--portable_lib",
+        required=True,


This probably shouldn't be required because portable library is loaded only when --quantize=True.

✅ Thanks, fixed in latest push.

skywall · 2025-05-15T09:42:00Z

examples/nxp/aot_neutron_compile.py

+
+        # For quantization we need to build the quantized_ops_aot_lib.so and _portable_lib.*.so
+        # Use this CMake options
+        # -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON


Is this documentation up to date? Is portable lib built just by specifying these two flags?

The quantized_ops_aot_lib links to portable_lib

$ ldd ./venv3.10/lib/python3.10/site-packages/executorch/kernels/quantized/libquantized_ops_aot_lib.so _portable_lib.cpython-310d-x86_64-linux-gnu.so => not found ....

For some reason we must load the portable_lib manually prior to libquantized_ops_aot_lib.so, the dlopen does not not find is by its own.

✅
FYI @skywall , we do not need any custom library loading for the quantized kernels out variants. There are already a python packages for this:

import executorch.extension.pybindings.portable_lib import executorch.kernels.quantized

Thanks to @digantdesai for the review items which helped me to find it out.

digantdesai · 2025-05-22T05:27:15Z

examples/nxp/README.md

+2. After building the ExecuTorch you shall have the `libquantized_ops_aot_lib.so` and `_portable_lib.<python_version>.so` located in the `pip_out/lib` folder. We will need this library when generating the quantized cifarnet ExecuTorch model. So as first step we will find it:
+```commandline
+$ find . -name "libquantized_ops_aot_lib.so"  
+./pip-out/lib.linux-x86_64-cpython-310-pydebug/executorch/kernels/quantized/libquantized_ops_aot_lib.so


FYI I added optimized cortex-M q/dq int8 op if you want to use that , it is still quite early days for that lib

digantdesai · 2025-05-22T05:27:53Z

examples/nxp/README.md

+./pip-out/lib.linux-x86_64-cpython-310-pydebug/executorch/kernels/quantized/libquantized_ops_aot_lib.so
+
+$ find . -name "_portable_lib.cpython-310d-x86_64-linux-gnu.so"
+./pip-out/lib.linux-x86_64-cpython-310-pydebug/executorch/extension/pybindings/_portable_lib.cpython-310d-x86_64-linux-gnu.so


is this using selective build?

Not sure what you mean.

Ok, I understand where you are heading. We needed the quantized_aot_lib to get the out variants for quantize/dequantize_per_tensor operators.
I find there are already python bindings and modules to solve it:

import executorch.extension.pybindings.portable_lib import executorch.kernels.quantized

✅

digantdesai · 2025-05-22T05:30:56Z

examples/nxp/aot_neutron_compile.py

+            torch.ops.load_library(args.portable_lib)
+            torch.ops.load_library(args.so_library)


why do we need these? just include the python module perhaps?

You are right (obviously) , we don't. Importing the python modules instead.

import executorch.extension.pybindings.portable_lib import executorch.kernels.quantized

Thanks for the finding, it helped me to locate these python modules.

digantdesai · 2025-05-22T05:34:18Z

examples/nxp/experimental/cifar_net/cifar_net.py

+        logger.info(f"Using pre-trained weights from `{state_dict_file}`.")
+        cifar_net.load_state_dict(torch.load(state_dict_file, weights_only=True))
+
+    if train:


digantdesai

Looks great. Thanks.

digantdesai · 2025-05-22T05:59:31Z

Ready to merge? Fix linter please?

robert-kalmar · 2025-05-22T13:58:17Z

Ready to merge? Fix linter please?

Not yet, updating quantizer to the recent changes : moving torchao to torch.ao

robert-kalmar · 2025-05-23T11:38:16Z

Linting error - fixed.
Quantizer invocation (using torchao instead of torch.ao) in the aot_neutron_example to align with updates in #10294 -fixed
Importing quantized operators instead of loading the *.so library - fixed

Now it is ready to merge.

robert-kalmar · 2025-05-24T13:28:39Z

3 checks failed. All with missing the "llm" preset. They were added in a later commit (c256723#diff-fc10486ef573a9c92fe4a135b8a1b20157154af6e83dacfd1ea046bda7814c84). I guess, those failures are unrelated with changes in the PR.

Although I wonder, why those tests got even triggered, as they are not in the .github/workflows of this codebase.

digantdesai · 2025-06-03T15:28:21Z

Let's re-merge the CI PR, and then we can merge this, so we have some confidence in this and know we won't be regressing. Thanks.

Co-authored-by: Martin Pavella <[email protected]>

digantdesai · 2025-06-10T14:48:12Z

examples/nxp/README.md

+
+2. Now run the `aot_neutron_compile.py` example with the `cifar10` model 
+```commandline
+$ python -m examples.nxp.aot_neutron_compile --quantize \


Should we include this in the CI?

I was thinking about it, but as the simulator is not yet ready only reasonable check is if the example not crash and produce some output.

I can include in the CI. No preference here.

digantdesai

Looks good. Is the setup.sh empty for a reason?

robert-kalmar · 2025-06-10T19:36:44Z

Looks good. Is the setup.sh empty for a reason?

It is not empty, just it content has not changed - https://github.com/pytorch/executorch/blob/2941a74be7f4d49198087d3983d591911c614260/examples/nxp/setup.sh
The change is the file mode - adding the execute bit chmod +x .

The WebUI is misleading here. By "empty file" it evidently means empty diff 🙃

robert-kalmar · 2025-06-18T12:11:37Z

Converting to draft unless the NXP Backend CI is back (#11756)

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label May 14, 2025

robert-kalmar force-pushed the upstream/release-mcux-25.03-full/aot-example branch from e5ed112 to 46c2a58 Compare May 14, 2025 11:56

pytorch-bot bot added module: nxp Issues related to NXP Neutron NPU delegation and code under backends/nxp/ release notes: nxp Changes to the NXP Neutron backend delegate labels May 14, 2025

JakeStevens reviewed May 14, 2025

View reviewed changes

skywall reviewed May 15, 2025

View reviewed changes

robert-kalmar force-pushed the upstream/release-mcux-25.03-full/aot-example branch from 46c2a58 to 2397cb0 Compare May 15, 2025 11:10

digantdesai reviewed May 22, 2025

View reviewed changes

digantdesai approved these changes May 22, 2025

View reviewed changes

robert-kalmar force-pushed the upstream/release-mcux-25.03-full/aot-example branch 2 times, most recently from 3018aea to 2941a74 Compare May 23, 2025 11:33

Add aot example with Neutron Backend

c7d4b49

Co-authored-by: Martin Pavella <[email protected]>

robert-kalmar force-pushed the upstream/release-mcux-25.03-full/aot-example branch from 2941a74 to c7d4b49 Compare June 10, 2025 09:10

robert-kalmar added the ciflow/trunk label Jun 10, 2025

digantdesai reviewed Jun 10, 2025

View reviewed changes

digantdesai requested changes Jun 10, 2025

View reviewed changes

robert-kalmar marked this pull request as draft June 18, 2025 12:10

	help="Flag for producing ArmBackend delegated model",
	help="Flag for producing NeutronBackend delegated model",

	# The output of the previous MaxPool has shape [batch, 64, 4, 4] ([batch, 4, 4, 64] in TFLite). When running
	# The output of the previous MaxPool has shape [batch, 64, 4, 4] ([batch, 4, 4, 64] in Neutron IR). When running

	# inference of the `FullyConnected`, TFlite will automatically collapse the channels and spatial dimensions and
	# inference of the `FullyConnected`, Neutron IR will automatically collapse the channels and spatial dimensions and

		torch.ops.load_library(args.portable_lib)
		torch.ops.load_library(args.so_library)

Add aot example with Neutron Backend #10871

Are you sure you want to change the base?

Add aot example with Neutron Backend #10871

Uh oh!

Conversation

robert-kalmar commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

pytorch-bot bot commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10871

❌ 1 New Failure, 1 Unrelated Failure

Uh oh!

robert-kalmar commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented May 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robert-kalmar May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robert-kalmar May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robert-kalmar May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robert-kalmar May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robert-kalmar May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

digantdesai left a comment

Choose a reason for hiding this comment

Uh oh!

robert-kalmar commented May 14, 2025 •

edited

Loading

pytorch-bot bot commented May 14, 2025 •

edited

Loading

robert-kalmar commented May 14, 2025 •

edited

Loading

robert-kalmar May 15, 2025 •

edited

Loading

robert-kalmar May 15, 2025 •

edited

Loading

robert-kalmar May 15, 2025 •

edited

Loading

robert-kalmar May 23, 2025 •

edited

Loading

robert-kalmar May 23, 2025 •

edited

Loading

robert-kalmar commented Jun 10, 2025 •

edited

Loading