1 Background
After training a network model using a deep learning open source framework, a format conversion is usually required before deployment, and the Horizon Toolchain model conversion currently supports Caffe1.0 and ONNX(opset_version=10/11 and ir_version≤7). ONNX(Open Neural Network Exchange) format is a common open source neural network format, supported by many inference engines, such as Pytorch, PaddlePaddle, TensorFlow, etc. This article describes in detail how to export the resulting model from TensorFlow2 to ONNX format.
2 Experimental environment
The experimental environment for this tutorial is as follows:
Python library
Version
tensorflow-cpu
2.11.0
tensorflow-intel
2.11.0
tf2onnx
1.13.0
protobuf
3.20.2
onnx
1.13.0
onnxruntime
1.14.0
3 tf2onnx tool introduction
tf2onnx can convert the TensorFlow/Keras model to ONNX through the command line. The main configuration parameters of the tool are as follows:
python -m tf2onnx.convert
--saved-model
--output
--opset
--inputs
--outputs
tf2onnx： https://github.com/onnx/tensorflow-onnx
4 codes
TensorFlow2ONNX。
import tensorflow as tf
import os
import onnx
def MyNet():
input1 = tf.keras.layers.Input(shape=(7, 7, 3))
x = tf.keras.layers.Conv2D(16, (3, 3),
activation='relu',
padding='same',
name='conv1')(input1)
x = tf.keras.layers.Conv2D(16, (3, 3),
activation='relu',
padding='same',
name='conv2')(x)
x = tf.keras.layers.Flatten(name='flatten')(x)
x = tf.keras.layers.Dense(100, activation='relu', name='fc1')(x)
output = tf.keras.layers.Dense(2, activation='softmax', name='predictions')(x)
input_1 = input1
model = tf.keras.models.Model(inputs=[input_1], outputs=output)
return model
model = MyNet()
model.save('model')
os.system("python -m tf2onnx.convert --saved-model model --output model.onnx --opset 11")
ONNX correctness verification
You can verify the correctness of the ONNX model with the following code, which checks the version of the model, the structure of the graph, the nodes, and the input and output. If the output is Check: None, no error information is reported and the model is correctly exported.
import onnx
onnx_model = onnx.load("./model.onnx")
check = onnx.checker.check_model(onnx_model)
print('Check: ', check)
Consistency check between TensorFlow2 and ONNX You can use the following code to check that the derived ONNX model and the original PaddlePaddle model have the same calculations.
import tensorflow as tf
import onnxruntime
import numpy as np
input1 = np.random.random((1, 7, 7, 3)).astype('float32')
ort_sess = onnxruntime.InferenceSession("./model.onnx")
ort_inputs = {ort_sess.get_inputs()[0].name: input1}
ort_outs = ort_sess.run(None, ort_inputs)
tf_model = tf.saved_model.load(export_dir="model")
tf_outs = tf_model(inputs=input1)
print(ort_outs[0])
print(tf_outs.numpy())
np.testing.assert_allclose(tf_outs.numpy(), ort_outs[0], rtol=1e-03, atol=1e-05)
print("onnx model check finsh.")
Case of multiple inputs
If your model has multiple inputs, you can refer to the code below to save TensorFlow2 in save-model mode and convert to ONNX format.
import tensorflow as tf
import os
def MyNet():
input1 = tf.keras.layers.Input(shape=(7, 7, 3))
input2 = tf.keras.layers.Input(shape=(7, 7, 3))
x = tf.keras.layers.Conv2D(16, (3, 3),
activation='relu',
padding='same',
name='conv1')(input1)
y = tf.keras.layers.Conv2D(16, (3, 3),
activation='relu',
padding='same',
name='conv2')(input2)
z = tf.keras.layers.Concatenate(axis=-1)([x, y])
z = tf.keras.layers.Flatten(name='flatten')(z)
z = tf.keras.layers.Dense(100, activation='relu', name='fc1')(z)
output = tf.keras.layers.Dense(2, activation='softmax', name='predictions')(z)
input_1 = input1
input_2 = input2
model = tf.keras.models.Model(inputs=[input_1,input_2], outputs=output)
return model
model = MyNet()
model.save('model')
os.system("python -m tf2onnx.convert --saved-model model --output model.onnx --opset 11")
Set input/output nodes
Sometimes, due to the difficulty of deployment, we do not want the pre - and post-processing parts of the TensorFlow network structure to be imported into the ONNX model. At this point, we can use the inputs and outputs parameters of the tf2onnx tool to specify the first and last exported nodes, so that nothing before and after the first node is imported into the ONNX model.
5 ONNX model visualization
Once exported into the ONNX model, you can use the open source visualization tool Netron to view the network structure and related configuration information. Netron use mainly divided into two kinds, one kind is to use the online web page version of https://netron.app/, another kind is to download the installation program at https://github.com/lutzroeder/netron. The visualizations of the model in this tutorial are:
6 ir_version and opset_version are modified
The ONNX model supported by Horizon Toolchain must meet opset_version=10/11 and ir_version≤7. When the ONNX model obtained does not meet these two requirements, you can modify the code and re-export it, or try to write scripts to directly modify the corresponding attributes of the ONNX model. Example code for the second approach is as follows:
import onnx
model = onnx.load("./model.onnx")
model.ir_version = 6
model.opset_import[0].version = 11
onnx.save_model(model, "./model_version.onnx")
** Note: ** There may be problems when switching from the high version to the low version, here is only one solution to try.
7 ONNX I/O dimension changed
When it is found that the input and output nodes of the ONNX model saved by the tf2onnx tool have abnormal values, such as the following situations: We can edit it with codes below.
import onnx
onnx_model = onnx.load("./model.onnx")
onnx_model.graph.input[0].type.tensor_type.shape.dim[0].dim_value = 1
onnx_model.graph.output[0].type.tensor_type.shape.dim[0].dim_value = 1
onnx.save(onnx_model, './model_dim.onnx')
Open the saved ONNX model file and you can see that the dimensions of the input and output nodes are normal: At this point, the ONNX model has satisfied the transformation condition of the Horizon toolchain.