use v1.9.0 to load ONNX Opset 15 failed.

[Li-chunming]

Describe the bug
I have updated my ORT version to the latest v1.9.0, but when I try to load ONNX with opset 11,13,15, it always report this BUG:

Exception has occurred: Fail (note: full exception trace is shown but execution is paused at: <module>) [ONNXRuntimeError] : 1 : FAIL : Load model from D:\src\ONNXmodel\my_mrcnn_opset15.onnx failed:D:\a\_work\1\s\onnxruntime\core/graph/model_load_utils.h:57 onnxruntime::model_load_utils::ValidateOpsetForDomain ONNX Runtime only *guarantees* support for models stamped with official released onnx opset versions. Opset 15 is under development and support for this is limited. The operator schemas and or other functionality may change before next ONNX release and in this case ONNX Runtime will not guarantee backward compatibility. Current official support for domain com.microsoft is till opset 1. File "D:\Mask_RCNN-master\onnx_runsession.py", line 169, in <module> (Current frame) sess = onnxruntime.InferenceSession(model_file_name)

I was confused because I saw that the version description said：
ONNX 1.10 support
opset 15
ONNX IR 8 (SparseTensor type, model local functionprotos, Optional type not yet fully supported this release)
So what's the reason for this bug? And what do " opset 1" mean?
supplement：The model is converted from Tf1.8.0 and Keras2.2.0; I used to load it successfully with old ORT version 1.1.2.

System information

• OS Platform and Distribution (Windows10):
• ONNX Runtime version:1.9.0
• ONNX (python packages) version:1.8.0
• Tensorflow version;1.8.0
• Python version:3.6.5
• CUDA/cuDNN version:9.0/7.0
• GPU model and memory:2070

[skottmckay]

1.9 is the latest official release and supports opset 14. 1.10 will be the next release but is currently in development.

Do you need something specific from opset 15 or could you convert your model to use opset 14?

[Li-chunming]

1.9 is the latest official release and supports opset 14. 1.10 will be the next release but is currently in development.

Do you need something specific from opset 15 or could you convert your model to use opset 14?

Thank you for your explanation @skottmckay I have changed my ONNX to 14 but the BUG is still appeared like this:
failed:D:\a\_work\1\s\onnxruntime\core/graph/model_load_utils.h:57 onnxruntime::model_load_utils::ValidateOpsetForDomain ONNX Runtime only *guarantees* support for models stamped with official released onnx opset versions. Opset 14 is under development and support for this is limited. The operator schemas and or other functionality may change before next ONNX release and in this case ONNX Runtime will not guarantee backward compatibility. Current official support for domain com.microsoft is till opset 1.
Have you ever met this situation?
I confirmed it again that my ORT is built by pip install onnxruntime==1.9.0

[skottmckay]

My apologies. ORT 1.9 does support ONNX 1.10 and opset 15. Slightly confusing when the ORT and ONNX release versions are close but not quite the same.

I'm not sure it's loading ORT from the 1.9.0 version. I did a test and was able to successfully load a model I manually updated to opset 15. Based on that, can you confirm you're actually loading the 1.9 version of ORT in your script?

e.g.

if I do this

import onnxruntime
help(onnxruntime)

I see this at the end of the output

VERSION
1.9.0

It should also say where it was loaded from a few lines after that.

[Li-chunming]

@skottmckay Thank you very much! I have now found out that this is a problem with the model‘s OPs and has nothing to do with its version. But I can load it successfully when I first use the old version v1.1.2 with ONNX Opset 11. So I tried to upgrade the version gradually and found:

1. use v1.1.2 --> Loading successfully.
2. use v1.2.0 --> [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Failed to load model with error: Unknown model file format version.
3. use v1.3.0 -->[ONNXRuntimeError] : 1 : FAIL : Fatal error: CropAndResize is not a registered function/op
4. use v1.4.0 -->[ONNXRuntimeError] : 1 : FAIL : Fatal error: CropAndResize is not a registered function/op
5. use v1.5.2 and later versions keep the same error message with the latest v1.9.0.

It seems like ORT doesn't support custom op since v1.3.0？ And my ONNX is converted from Keras2.2.0+Tensorflow1.8.0 with CUDA9.0/Cudnn7.0，but now I want to use CUDA11.4/Cudnn8.0 to run it ,do you think this is the reason for this problem? Must the loading environment of onnx be similar to its generation environment?

[skottmckay]

Would help if you can share the model as it doesn't quite make sense to me why you'd get an error message about the ONNX opset for a model using contrib ops like CropAndResize as that domain has a different version number to the ONNX operators.

What converter are you using and what version is it? There was an issue around the CropAndResize contrib op (not an official ONNX operator) that was fixed last year: Wondering if that is relevant. #3625

In general there shouldn't be any issue with running an ONNX model with different versions of CUDA/cuDNN.

[Li-chunming]

@skottmckay Thanks a lot! I met the same problem with #3625, after adding op_version and follow the changes it can now work with v1.9.0 and all related version.