Add onnxruntime as wasi-nn backend #4485

dongsheng28849455 · 2025-07-14T03:40:46Z

Steps to verify:

Install the onnx runtime (official release), assuming in /opt/onnxruntime
Build iwasm with WAMR_BUILD_WASI_NN_ONNX enabled
Using an onnx model of ssd-mobilenetv1 from ONNX Model Zoo
Generate the data file of input_tensor.bin, from origin picture for wasi-nn (with shape [1, 383, 640, 3])
Use nn-cli for test, eg.

--load-graph=file=./ssd_mobilenet_v1.onnx,id=graph1,encoding=1 
--init-execution-context=graph-id=graph1,id=exec0 
--set-input=file=./input_tensor.bin,context-id=exec0,dim=1,dim=383,dim=640,dim=3,type=3 
--compute=context-id=exec0 
--get-output=context-id=exec0,file=output.bin

Generate output.bin, with shape [1, 100, 4] and f32 type, which contents match the sample's output

yamt

what's the relationship with #4304?

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

dongsheng28849455 · 2025-07-14T07:01:25Z

what's the relationship with #4304?

1, Adapt to latest wasi-nn arch and support WAMR_BUILD_WASI_EPHEMERAL_NN
2, Test with models and nn-cli

core/iwasm/libraries/wasi-nn/include/wasi_nn.h

core/iwasm/libraries/wasi-nn/include/wasi_nn_types.h

core/iwasm/libraries/wasi-nn/src/wasi_nn.c

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

yamt · 2025-08-04T06:16:54Z

restarting SGX 143
https://github.com/bytecodealliance/wasm-micro-runtime/actions/runs/16643007953/job/47097105368?pr=4485

1, type converter btw wasi-nn and onnx runtime returns bool instead of type 2, out_buffer_size does not hold the expected size. 3, onnx runtime does not need calculate input_tenser size.

core/iwasm/libraries/wasi-nn/cmake/wasi_nn.cmake

core/iwasm/libraries/wasi-nn/cmake/Findonnxruntime.cmake

yamt · 2025-08-05T08:49:09Z

* Using an onnx model of [ssd-mobilenetv1](https://github.com/onnx/models/tree/main/validated/vision/object_detection_segmentation/ssd-mobilenetv1) from [ONNX Model Zoo](https://github.com/onnx/models/blob/main/README.md#onnx-model-zoo)

* Generate the data file of input_tensor.bin, from origin picture for wasi-nn (with shape [1, 383, 640, 3])

* Use [nn-cli](https://github.com/bytecodealliance/wasm-micro-runtime/pull/4373) for test, eg.

--load-graph=file=./ssd_mobilenet_v1.onnx,id=graph1,encoding=1 
--init-execution-context=graph-id=graph1,id=exec0 
--set-input=file=./input_tensor.bin,context-id=exec0,dim=1,dim=383,dim=640,dim=3,type=3 
--compute=context-id=exec0 
--get-output=context-id=exec0,file=output.bin

Generate output.bin, with shape [1, 100, 4] and f32 type, which contents match the sample's output

using this model, i had to use non-zero index for get_output. thus i had to fix nn-cli bug.
which model have you used?

dongsheng28849455 · 2025-08-05T09:32:47Z

* Using an onnx model of [ssd-mobilenetv1](https://github.com/onnx/models/tree/main/validated/vision/object_detection_segmentation/ssd-mobilenetv1) from [ONNX Model Zoo](https://github.com/onnx/models/blob/main/README.md#onnx-model-zoo)

* Generate the data file of input_tensor.bin, from origin picture for wasi-nn (with shape [1, 383, 640, 3])

* Use [nn-cli](https://github.com/bytecodealliance/wasm-micro-runtime/pull/4373) for test, eg.

--load-graph=file=./ssd_mobilenet_v1.onnx,id=graph1,encoding=1 
--init-execution-context=graph-id=graph1,id=exec0 
--set-input=file=./input_tensor.bin,context-id=exec0,dim=1,dim=383,dim=640,dim=3,type=3 
--compute=context-id=exec0 
--get-output=context-id=exec0,file=output.bin

Generate output.bin, with shape [1, 100, 4] and f32 type, which contents match the sample's output

using this model, i had to use non-zero index for get_output. thus i had to fix nn-cli bug. which model have you used?

I'm using this one: https://github.com/onnx/models/blob/main/validated/vision/object_detection_segmentation/ssd-mobilenetv1/model/ssd_mobilenet_v1_10.onnx

yamt · 2025-08-05T09:41:16Z

I'm using this one: https://github.com/onnx/models/blob/main/validated/vision/object_detection_segmentation/ssd-mobilenetv1/model/ssd_mobilenet_v1_10.onnx

thank you. however, this model looks same in the regard. (have 4 outputs)

yamt · 2025-08-05T09:56:27Z

I'm using this one: https://github.com/onnx/models/blob/main/validated/vision/object_detection_segmentation/ssd-mobilenetv1/model/ssd_mobilenet_v1_10.onnx

thank you. however, this model looks same in the regard. (have 4 outputs)

maybe you somehow interpreted only the first (idx=0) output, which contains bounding boxes?

dongsheng28849455 · 2025-08-05T10:18:31Z

I'm using this one: https://github.com/onnx/models/blob/main/validated/vision/object_detection_segmentation/ssd-mobilenetv1/model/ssd_mobilenet_v1_10.onnx

thank you. however, this model looks same in the regard. (have 4 outputs)

maybe you somehow interpreted only the first (idx=0) output, which contains bounding boxes?

output shape is [1, 100, 4]:
hexdump output.bin:
0000000 c05e 3ef8 5d0f 3e2f 06ad 3f09 693f 3e5a
0000010 a9d3 3f03 52ae 3d37 4e7d 3f18 1dd4 3e02
0000020 5914 3dc3 41a7 3efa 2644 3f3a 0000 3f80
0000030 73c5 3ef8 af0e 3e65 02ea 3f0b 69e3 3e84
0000040 50f7 3ef7 e514 3e04 d723 3f04 bb62 3e1f
0000050 5a2c 3ef3 75e2 3e84 4f2e 3f0e 329c 3e9e
0000060 c04a 3ef2 ca20 3e99 df0d 3f10 8744 3eb9
0000070 f5c0 3ec3 61ce 3e0f 066a 3ed6 aada 3e1c
0000080 0000 0000 0000 0000 0000 0000 0000 0000

in which, one bounding box for example:
c05e 3ef8 (0.48584265) stands for the ymin
5d0f 3e2f (0.17125343) for xmin
06ad 3f09 (0.5352581) is ymax
693f 3e5a(0.2132921) is xmax

it looks good for my test picure (http://images.cocodataset.org/val2017/000000088462.jpg)

yamt · 2025-08-06T01:16:46Z

I'm using this one: https://github.com/onnx/models/blob/main/validated/vision/object_detection_segmentation/ssd-mobilenetv1/model/ssd_mobilenet_v1_10.onnx

thank you. however, this model looks same in the regard. (have 4 outputs)

maybe you somehow interpreted only the first (idx=0) output, which contains bounding boxes?

output shape is [1, 100, 4]: hexdump output.bin: 0000000 c05e 3ef8 5d0f 3e2f 06ad 3f09 693f 3e5a 0000010 a9d3 3f03 52ae 3d37 4e7d 3f18 1dd4 3e02 0000020 5914 3dc3 41a7 3efa 2644 3f3a 0000 3f80 0000030 73c5 3ef8 af0e 3e65 02ea 3f0b 69e3 3e84 0000040 50f7 3ef7 e514 3e04 d723 3f04 bb62 3e1f 0000050 5a2c 3ef3 75e2 3e84 4f2e 3f0e 329c 3e9e 0000060 c04a 3ef2 ca20 3e99 df0d 3f10 8744 3eb9 0000070 f5c0 3ec3 61ce 3e0f 066a 3ed6 aada 3e1c 0000080 0000 0000 0000 0000 0000 0000 0000 0000

in which, one bounding box for example: c05e 3ef8 (0.48584265) stands for the ymin 5d0f 3e2f (0.17125343) for xmin 06ad 3f09 (0.5352581) is ymax 693f 3e5a(0.2132921) is xmax

it looks good for my test picure (http://images.cocodataset.org/val2017/000000088462.jpg)

ok. i understood.

actually, the model has 4 outputs as documented
and (with this nn-cli fix) you can get them as the following.

--load-graph=file=ssd_mobilenet_v1_10.onnx,encoding=1 \
--init-execution-context \
--set-input=file=input.bin,dim=1,dim=383,dim=640,dim=3,type=3 \
--compute \
--get-output=idx=0,file=output0.bin \
--get-output=idx=1,file=output1.bin \
--get-output=idx=2,file=output2.bin \
--get-output=idx=3,file=output3.bin

as wasi-nn doesn't have get-output-by-name, you need to use integer indexes.
output tensors of this model are (detection_boxes, detection_classes, detection_scores, num_detections). (in this order)

you were only looking at detection_boxes.

dongsheng28849455 · 2025-08-06T01:34:41Z

I'm using this one: https://github.com/onnx/models/blob/main/validated/vision/object_detection_segmentation/ssd-mobilenetv1/model/ssd_mobilenet_v1_10.onnx

thank you. however, this model looks same in the regard. (have 4 outputs)

maybe you somehow interpreted only the first (idx=0) output, which contains bounding boxes?

output shape is [1, 100, 4]: hexdump output.bin: 0000000 c05e 3ef8 5d0f 3e2f 06ad 3f09 693f 3e5a 0000010 a9d3 3f03 52ae 3d37 4e7d 3f18 1dd4 3e02 0000020 5914 3dc3 41a7 3efa 2644 3f3a 0000 3f80 0000030 73c5 3ef8 af0e 3e65 02ea 3f0b 69e3 3e84 0000040 50f7 3ef7 e514 3e04 d723 3f04 bb62 3e1f 0000050 5a2c 3ef3 75e2 3e84 4f2e 3f0e 329c 3e9e 0000060 c04a 3ef2 ca20 3e99 df0d 3f10 8744 3eb9 0000070 f5c0 3ec3 61ce 3e0f 066a 3ed6 aada 3e1c 0000080 0000 0000 0000 0000 0000 0000 0000 0000
in which, one bounding box for example: c05e 3ef8 (0.48584265) stands for the ymin 5d0f 3e2f (0.17125343) for xmin 06ad 3f09 (0.5352581) is ymax 693f 3e5a(0.2132921) is xmax
it looks good for my test picure (http://images.cocodataset.org/val2017/000000088462.jpg)

ok. i understood.

actually, the model has 4 outputs as documented and (with this nn-cli fix) you can get them as the following.
--load-graph=file=ssd_mobilenet_v1_10.onnx,encoding=1 \
--init-execution-context \
--set-input=file=input.bin,dim=1,dim=383,dim=640,dim=3,type=3 \
--compute \
--get-output=idx=0,file=output0.bin \
--get-output=idx=1,file=output1.bin \
--get-output=idx=2,file=output2.bin \
--get-output=idx=3,file=output3.bin
as wasi-nn doesn't have get-output-by-name, you need to use integer indexes. output tensors of this model are (detection_boxes, detection_classes, detection_scores, num_detections). (in this order)

you were only looking at detection_boxes.

OK, Thank you!

yamt

lgtm

dongsheng28849455 · 2025-08-08T04:58:30Z

@lum1n0us could you help review the code?

lum1n0us · 2025-08-11T08:50:06Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

+
+/* Helper functions */
+static void
+check_status_and_log(const OnnxRuntimeContext *ctx, OrtStatus *status)


lum1n0us · 2025-08-11T08:50:26Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

+}
+
+static bool
+convert_ort_type_to_wasi_nn_type(ONNXTensorElementDataType ort_type,


lum1n0us · 2025-08-11T08:54:31Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

+        err = convert_ort_error_to_wasi_nn_error(ctx, status);
+        NN_ERR_PRINTF("Failed to create ONNX Runtime environment: %s",
+                      error_message);
+        ctx->ort_api->ReleaseStatus(status);


seems convert_ort_error_to_wasi_nn_error() will ReleaseStatus(status). therefore, L194 might not be necessary

lum1n0us · 2025-08-11T09:03:18Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

+        wasi_nn_error err = convert_ort_error_to_wasi_nn_error(ctx, status);
+        NN_ERR_PRINTF("Failed to create ONNX Runtime session: %s",
+                      error_message);
+        ctx->ort_api->ReleaseStatus(status);


seems convert_ort_error_to_wasi_nn_error() will ReleaseStatus(status). therefore, L352 might not be necessary

lum1n0us · 2025-08-11T09:09:29Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

+        NN_ERR_PRINTF("Maximum number of graphs reached");
+        return runtime_error;
+    }
+


just my doubt, add a protector about name? like name is empty or NULL

lum1n0us · 2025-08-11T09:11:56Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

+__attribute__((visibility("default"))) wasi_nn_error
+load(void *onnx_ctx, graph_builder_array *builder, graph_encoding encoding,
+     execution_target target, graph *g)
+{


add a protector about onnx_ctx like others?

if (!onnx_ctx) { return runtime_error; }

lum1n0us · 2025-08-11T09:12:07Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

+__attribute__((visibility("default"))) wasi_nn_error
+load_by_name(void *onnx_ctx, const char *name, uint32_t filename_len, graph *g)
+{
+    OnnxRuntimeContext *ctx = (OnnxRuntimeContext *)onnx_ctx;


add a protector about onnx_ctx like others?

if (!onnx_ctx) { return runtime_error; }

lum1n0us · 2025-08-11T09:23:02Z

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp

+                ctx->ort_api->ReleaseValue(output.second);
+            }
+            ctx->ort_api->ReleaseMemoryInfo(ctx->exec_ctxs[i].memory_info);
+            ctx->exec_ctxs[i].is_initialized = false;


input_names and output_names?

dongsheng28849455 requested review from loganek, lum1n0us, no1wudi, TianlongLiang, wenyongh, xujuntwt95329 and yamt as code owners July 14, 2025 03:40

lum1n0us added the new feature Determine if this Issue request a new feature or this PR introduces a new feature. label Jul 14, 2025

yamt reviewed Jul 14, 2025

View reviewed changes

yamt added the wasi-nn label Jul 14, 2025

dongsheng28849455 requested a review from yamt July 17, 2025 06:49

yamt reviewed Jul 23, 2025

View reviewed changes

dongsheng28849455 force-pushed the feature/support_onnx_for_wasi-nn branch from aa88085 to 1fb25ad Compare July 29, 2025 08:09

dongsheng28849455 requested a review from yamt July 29, 2025 08:13

dongsheng28849455 force-pushed the feature/support_onnx_for_wasi-nn branch 2 times, most recently from 1e60909 to 29e4dd5 Compare July 29, 2025 08:17

yamt reviewed Jul 29, 2025

View reviewed changes

dongsheng28849455 requested a review from yamt July 29, 2025 09:06

yamt reviewed Aug 4, 2025

View reviewed changes

core/iwasm/libraries/wasi-nn/src/wasi_nn_onnx.cpp Outdated Show resolved Hide resolved

dongsheng28849455 added 6 commits August 5, 2025 09:40

Add onnxruntime as wasi-nn backend

56b6195

follow up some review comments

6cab94c

1, type converter btw wasi-nn and onnx runtime returns bool instead of type 2, out_buffer_size does not hold the expected size. 3, onnx runtime does not need calculate input_tenser size.

clang-format

c866d05

remove global context

4b50341

put checks under the lock

801eb2b

use WASM_ENABLE_WASI_EPHEMERAL_NN

cd3cb6c

dongsheng28849455 force-pushed the feature/support_onnx_for_wasi-nn branch from 0dcdcab to cd3cb6c Compare August 5, 2025 01:44

dongsheng28849455 requested a review from yamt August 5, 2025 01:46

tensor type will not support legacy wasi-nn abi

38e9d2b

yamt reviewed Aug 5, 2025

View reviewed changes

core/iwasm/libraries/wasi-nn/cmake/wasi_nn.cmake Outdated Show resolved Hide resolved

core/iwasm/libraries/wasi-nn/cmake/Findonnxruntime.cmake Show resolved Hide resolved

using the CMake config file

fa6c3a3

dongsheng28849455 requested a review from yamt August 8, 2025 02:17

Manually set the imported target with name space

0c164b6

yamt approved these changes Aug 8, 2025

View reviewed changes

lum1n0us reviewed Aug 11, 2025

View reviewed changes

Add onnxruntime as wasi-nn backend #4485

Are you sure you want to change the base?

Add onnxruntime as wasi-nn backend #4485

Conversation

dongsheng28849455 commented Jul 14, 2025

Uh oh!

yamt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dongsheng28849455 commented Jul 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yamt commented Aug 4, 2025

Uh oh!

Uh oh!

Uh oh!

yamt commented Aug 5, 2025

Uh oh!

dongsheng28849455 commented Aug 5, 2025

Uh oh!

yamt commented Aug 5, 2025

Uh oh!

yamt commented Aug 5, 2025

Uh oh!

dongsheng28849455 commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yamt commented Aug 6, 2025

Uh oh!

dongsheng28849455 commented Aug 6, 2025

Uh oh!

yamt left a comment

Choose a reason for hiding this comment

Uh oh!

dongsheng28849455 commented Aug 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dongsheng28849455 commented Aug 5, 2025 •

edited

Loading