A wealth of operators are already supported in the Horizon toolchain, and in most cases your model should be able to be converted and deployed on the Horizon chip without a problem by using the hb_mapper tool. In the case that a few operators are not supported, we recommend that you first try the possibility of replacing operators, which is conducive to the full use of the Horizon chip capabilities.
At present, only CPU operator development capabilities are provided, and onnx operators and caffe operators can be customized. A complete custom operator application process includes four stages: registration operator, operator realization, model conversion with custom operator and running with custom op model.
1 Customize the onnx operator
1.1 Export the pytorch model with custom operators to ONNX
Register the custom operator with torch.onnx.register_custom_op_symbolic and then export the onnx model. Note the following configuration parameters:
1. The first parameter of the register_custom_op_symbolic function ‘::adaptive_avg_pool2d’ is the pytorch operator name. An incorrect parameter will cause the registration of the custom operator to fail
2. The operation domain must be set to horizon.custom and the operator type to PyOp
3. class_name_s must correspond to the class name in the operator implementation file
4. module_s is the same as the operator implementation file name. If the operator implementation file is in the subdirectory (custom_op) of the current directory, the relative path must be included: “custom_op/sample_custom”.
5. You must specify the input_types_i, output_types_i, and output_shape_s parameters
6. Note that opset_version is set to 10 or 11
Reference code:
import torch
from horizon_nn.horizon_onnx.onnx_pb import TensorProto
from torch.onnx.utils import register_custom_op_symbolic
#prepare your model and input_data
def horizon_pool(g, input, output_size):
return g.op(
'horizon.custom::PyOp', #required, ! must be 'horizon.custom' domain !
input,
class_name_s="GlobalAveragePool", #required ! must match the class def name in sample_custom python file !
compute_s="compute", #optional, 'compute' by default
module_s="sample_custom", #required ! must match the file name of the "op_register_files" !
input_types_i=[TensorProto.FLOAT], #required
output_types_i=[TensorProto.FLOAT], #required
output_shape_s=["1, 1024, 1, 1"]) #required
register_custom_op_symbolic('::adaptive_avg_pool2d',
horizon_pool,
opset_version=11)
torch.onnx.export(model, input_data, "custom_op.onnx", opset_version=11)
1.2 Operator Implementation
Correspond to the operator implementation file configured when registering the custom operator in the previous section (class_name must be the same).
#sample_custom.py
import numpy as np
from horizon_nn.custom import op_implement_register
@op_implement_register("CustomIdentity")
class CustomIdentity(object):
def __init__(self, kernel_size, threshold):
self._kernel_size = kernel_size
self._default_threshold = threshold
def compute(self, X):
return X
@op_implement_register("GlobalAveragePool")
class GlobalAveragePool(object):
def __init__(self):
pass
def compute(self, X):
return np.nanmean(X, axis=(2, 3)).reshape(-1, 1024, 1, 1)
2 Customize the caffe operator
2.1 Modifying prototxt
In the original model file, mark the corresponding type of the Custom operator as “Custom” and set custom_param. params is the parameter passed to the operator, specified as’ param_name ': param_value, separated by \n.
layer {
name: "hr_op"
type: "Custom"
bottom: "res3d_in"
top: "res3d"
custom_param {
kind: "CustomIdentity"
shape {
dim: 1
dim: 512
dim: 28
dim: 28
}
params: "'kernel_size': 10 \n'threshold': 0.5"
}
}
2.2 Operator Implementation
In contrast to the onnx model, a custom operator implementation of the caffe model also needs to provide the output dimensions of the operator.
#sample_custom.py
from horizon_nn.custom.op_registration import op_implement_register, op_shape_infer_register
@op_implement_register("CustomIdentity")
class CustomIdentity(object):
def __init__(self, kernel_size, threshold):
self._kernel_size = kernel_size
self._default_threshold = threshold
def compute(self, X):
return X
@op_shape_infer_register("CustomIdentity")
def infer_shape(inputs_shape):
"""Infer the output shapes of the custom operator.
Arguments:
input_shapes: A list of input shapes.
Returns:
Return a list of custom operator's output shapes.
"""
outputs_shape = inputs_shape
return outputs_shape
3 Model conversion with custom operators
In the model transformation configuration file, add custom operator-related parameters as shown in the following example:
custom_op_method: specifies a fixed register.
op_register_files: The implementation file of the custom operator calculation. If there are multiple implementations, use '; Just separate the individual files.
4 Model inference with custom operators
In order to successfully deploy the.bin model containing the custom operator to the development board, you also need to provide the C++ code implementation of the custom operator. You can modify it using the template provided below:
Header file:
// custom_identity.h
#ifndef ADVANCED_SAMPLES_CUSTOM_IDENTITY_H_
#define ADVANCED_SAMPLES_CUSTOM_IDENTITY_H_
#include <string>
#include <vector>
#include "dnn/hb_dnn.h"
#include "dnn/plugin/hb_dnn_layer.h"
#include "dnn/plugin/hb_dnn_ndarray.h"
namespace hobot {
namespace dnn {
Layer *CustomIdentity_layer_creator();
class CustomIdentity : public Layer {
public:
CustomIdentity() = default;
~CustomIdentity() override = default;
public:
int32_t Init(const Attribute &attributes) override;
int32_t Forward(const std::vector<NDArray *> &bottomBlobs,
std::vector<NDArray *> &topBlobs,
const hbDNNInferCtrlParam *inferCtrlParam) override;
std::string GetType() const override { return "CustomIdentity"; }
private:
std::string module_;
};
} // namespace dnn
} // namespace hobot
#endif
cpp:
// custom_identity.cpp
#include "custom_identity.h"
namespace hobot {
namespace dnn {
Layer *CustomIdentity_layer_creator() { return new CustomIdentity; }
int32_t CustomIdentity::Init(const Attribute &attributes) {
// unused attribute, just demonstrating
attributes.GetAttributeValue(&module_, "module");
return 0;
}
int32_t CustomIdentity::Forward(const std::vector<NDArray *> &bottomBlobs,
std::vector<NDArray *> &topBlobs,
const hbDNNInferCtrlParam *inferCtrlParam) {
const NDArray *input = bottomBlobs[0];
NDArray *out = topBlobs[0];
const auto *input_data = input->Dptr<float>();
auto *out_data = out->Dptr<float>();
uint32_t size = input->Size();
for (uint32_t i = 0U; i < size; i++) {
out_data[i] = input_data[i];
}
return 0;
}
} // namespace dnn
} // namespace hobot
After the above two files are placed in the current project directory, it is only necessary to add the registration of the operator before loading the model when compiling the infer code. The registration can refer to the following code:
//infer.cpp
#include "custom_identity.h"
// register custom layer
hbDNNRegisterLayerCreator("CustomIdentity",
hobot::dnn::CustomIdentity_layer_creator)