I am using onnxruntime to infer the onnx model of float16 type, the following error occurs, how should I solve it[Build]

[1615070057]

Describe the issue

I am using onnxruntime to infer the onnx model of float16 type, the following error occurs, how should I solve it[Build]
error:
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from model_fp16.onnx failed:Node (Resize__139) Op (Resize) [ShapeInferenceError] Either sizes or scales must be provided, but not both of them

Urgency

The project is urgent, the deadline is August 23

Target platform

linux 64

Build script

import onnx
from onnxconverter_common import float16

model = onnx.load("model.onnx")
model_fp16 = float16.convert_float_to_float16(model)
onnx.save(model_fp16, "model_fp16.onnx", size_threshold=2048)

Error / output

onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from model_fp16.onnx failed:Node (Resize__139) Op (Resize) [ShapeInferenceError] Either sizes or scales must be provided, but not both of them

Visual Studio Version

No response

GCC / Compiler Version

No response

[snnn]

It doesn't like a build issue. Please use onnx model checker to check your model first.

[snnn]

Please tell us more about how you generate the model file. I don't know which tool you used, so I don't know who can help you. So far, it doesn't seem to relate to onnxruntime. You probably report the issue to a wrong place.

[1615070057]

The onnx.checker verification is successful, but the onnxruntime loading model reports an error: onnxruntime.capi.onnxruntime_pybind11_state.InvalidGraph: [ONNXRuntimeError] : 10 : INVALID_GRAPH : Load model from model_fp16.onnx failed:This is an invalid model. Type Error: Type 'tensor(float16 )' of input parameter (const_empty_float__132) of operator (Resize) in node (Resize__139) is invalid.

[snnn]

But you said you "checked the onnx model and reported an error: onnx.onnx_cpp2py_export.checker.ValidationError: " So I am confused. Was it successful or not?

[1615070057]

When quantizing model.onnx to model_float16.onnx, For operator like Resize, some of the optional inputs doesn't support float16. We need exclude fp16 conversion to them. So onnx.checker verification is successful. But still inference model in onnxruntime , an error occurred: onnxruntime.capi.onnxruntime_pybind11_state.InvalidGraph: [ONNXRuntimeError] : 10 : INVALID_GRAPH : Load model from model_fp16.onnx failed:This is an invalid model. Type Error: Type 'tensor(float16 )' of input parameter (const_empty_float__132 ) of operator (Resize) in node (Resize__139) is invalid.

[tianleiwu]

@1615070057,

Type Error: Type 'tensor(float16 )' of input parameter (const_empty_float__132) of operator (Resize) in node (Resize__139) is invalid. means that the fp16 model is invalid.

You may try the following since Resize is handled specially in the conversion:

from onnxruntime.transformers.onnx_model import OnnxModel
model = OnnxModel(onnx.load_model(fp32_onnx_model_path, load_external_data=True))
model.convert_float_to_float16(keep_io_types=False)
model.save_model_to_file(fp16_onnx_model_path, use_external_data_format=False)

There are many parameters you can use to fine tune the conversion:

onnxruntime/onnxruntime/python/tools/transformers/onnx_model.py

Lines 723 to 736 in 1fb2e71

	use_symbolic_shape_infer (bool, optional): use symbolic shape inference instead of onnx shape inference.
	Defaults to True.
	keep_io_types (Union[bool, List[str]], optional): boolean or a list of float32 input/output names.
	If True, model inputs/outputs should be left as float32.
	Defaults to True.
	op_block_list (List[str], optional): List of operator types to leave as float32.
	Defaults to None, which will use `float16.DEFAULT_OP_BLOCK_LIST`.
	node_block_list (List[str], optional): List of node names to leave as float32. Defaults to None.
	force_fp16_initializers(bool): force converting all float initializers to float16.
	Default to false.
	min_positive_val (float, optional): minimal positive value. Defaults to 1e-7.
	max_finite_val (float, optional): maximal finite value. Defaults to 1e4.
	force_fp16_inputs(Dict[str, List[int]]): Force the conversion of the inputs of some operators to float16, even if
	this script's preference it to keep them in float32.

[JIANG3330]

I have the same issue.
After converting the model to an FP16 model with the following code and then inferring with ONNX Runtime, this error occurs.

import onnx
from onnxconverter_common import float16

model = onnx.load("test.onnx")
model_fp16 = float16.convert_float_to_float16(model)
onnx.save(model_fp16, "test_fp16.onnx")

If inference is done with FP32, there is no problem.

import onnxruntime as ort
import numpy as np

model_path = 'test.onnx'

input_data = np.random.randn(1, 3, 488, 488).astype(np.float32)

ort_session = ort.InferenceSession(model_path, providers=['CPUExecutionProvider'])
input_name = ort_session.get_inputs()[0].name
output_name = ort_session.get_outputs()[0].name
pred_onnx = ort_session.run([output_name], {input_name: input_data})[0]

If inference is done with an FP16 model, an error is reported.

import onnxruntime as ort
import numpy as np

model_path = 'test_fp16.onnx'

input_data = np.random.randn(1, 3, 488, 488).astype(np.float16)

ort_session = ort.InferenceSession(model_path, providers=['CPUExecutionProvider'])
input_name = ort_session.get_inputs()[0].name
output_name = ort_session.get_outputs()[0].name
pred_onnx = ort_session.run([output_name], {input_name: input_data})[0]

onnx_model.zip

[tianleiwu]

@JIANG3330, your fp32 model exported form paddle paddle is invalid.

You can see the Resize op spec says Only one of ‘scales’ and ‘sizes’ can be specified.:
https://onnx.ai/onnx/operators/onnx__Resize.html

It seems that FP16 behavior expected. We'll check why FP32 inference did not raise the exception.