In recent years, deep learning-based methods have made great achievements in image classification, object detection, instance segmentation and other tasks, thanks to the deep structure of neural network models and the large number of model parameters. However, the large number of parameters means that it is difficult to deploy the model to resource-constrained edge devices, such as smartphones, smart wearables, drones, robots, self-driving cars, etc., which often have strict time constraints on the execution process of the neural network or have strict requirements for power consumption during long-term execution. Therefore, there is an urgent need for techniques to optimize models to reduce model size, achieve lower power consumption and faster inference speed.
Quantization approximates the continuous value of a signal (float32) to a finite number of discrete values (int8, etc.), and is the most common means of reducing the computation time and power consumption of neural network models. However, the quantization/inverse quantization node introduced after the quantization of neural network consumes a lot of time for data traversal in the process of point-by-point elementwise operation, which seriously affects the inference performance of the quantization model. In order to reduce the computing time, in the practice of model deployment, the inverse quantization is usually incorporated into the post-processing part of the model for calculation to reduce the time consumption of repeated data traversal, which is called inverse quantization node fusion.
The following table shows the performance of the inverse quantization node of the efficientdetd0 model on the Horzion XJ3 development board before and after the fusion. Obviously, when the inverse quantization node is integrated into the post-processing of the model, the performance of the model changes as follows:
- The post-processing time of inverse quantization node fusion is much less than that of Dequantize node + post-processing before fusion, and the effect of reducing the time of one data traversal is obvious;
- The model’s workflow full frame rate running FPS increased by a staggering 34.18% compared to before the inverse quantization node fusion.
Therefore, this paper takes the target detection model efficientdetd0 as an example to describe the relevant background and basic process of inverse quantization node fusion.
1. Quantization and antiquantization calculation
Quantize is to establish a mapping relationship between floating-point data and fixed-point data, and convert featuremap or common operations (convolution, activation, pooling, homogenization, etc.) into equivalent integer operations. In this way, large benefits are realized at a small cost of precision loss. Dequantize refers to the inverse operation of quantization, converting integer data to floating point data.
1.1 Quantize
The Quantize node is used to quantify the input data of the model type float to type int. The Horizon quantization tool adopts symmetric quantization. The formulas and codes of the PTQ and QAT quantization tools are introduced below. 1.1.1 Quantization of PTQ **1. Quantization formula **
xint=clamp(round(xfloat/scale+zero_point),−2b−1,2b−1−1)x_{int}=clamp(round(x_{float}/scale+zero\_{point}),-2^{b-1},2^{b-1}-1)xint=clamp(round(xfloat/scale+zero_point),−2b−1,2b−1−1)
The round(x) function implements rounding of floating point numbers, and the clamp(x) function implements clamping of data to integer values between [−2b−1,2b−1−1][-2^{b-1},2^{b-1}-1][−2b−1,2b−1−1]. zero_point is the zero offset value of asymmetric quantization, and zero_point = 0 in symmetric quantization. scale, called the scale factor or quantization factor, determines the number of quantized partitions (a total of 2b−2^b-2b−1 partitions) and is usually expressed as a floating point number.
**2. Quantized code **
int8 Quantizes python code:
import numpy as np
x= input_float/scale
x=np.clip(np.round(x),-128,127)
x = x.astype(np.int8)
int16 Quantizes python code:
import numpy as np
x= input_float/scale
x=np.clip(np.round(x),-32768,32767)
x = x.astype(np.int16)
int8 Quantizes C++ code:
static inline float32_t _round(float32_t const input) {
std::fesetround(FE_TONEAREST);
float32_t const result{std::nearbyintf(input)};
return result;
}
static inline int8_t int_quantize(float32_t value, float32_t const scale) {
value = _round(value / scale);
value = std::min(std::max(value, -128.0f), 127.0f);
return static_cast<int8_t>(value);
}
int16 Quantizes C++ code:
static inline float32_t _round(float32_t const input) {
std::fesetround(FE_TONEAREST);
float32_t const result{std::nearbyintf(input)};
return result;
}
static inline int16_t int_quantize(float32_t value, float32_t const scale) {
value = _round(value / scale);
value = std::min(std::max(value, -32768.0f), 32767.0f);
return static_cast<int16_t>(value);
}
1.1.2 QAT quantization
**1. Quantization formula **
xint=clamp(floor(xfloat/scale+0.5,−2b−1,2b−1−1)x_{int}=clamp(floor(x_{float}/scale+0.5,-2^{b-1},2^{b-1}-1)xint=clamp(floor(xfloat/scale+0.5,−2b−1,2b−1−1)
floor(x+0.5) implements rounding of floating-point numbers, and clamp(x) implements clamping of data The integer value between [−2b−1,2b−1−1].scale,calledthescalefactororquantizationfactor,determinesthenumberofquantizedpartitions(atotalof[-2^{b-1},2^{b-1}-1]. scale, called the scale factor or quantization factor, determines the number of quantized partitions (a total of[−2b−1,2b−1−1].scale,calledthescalefactororquantizationfactor,determinesthenumberofquantizedpartitions(atotalof2^b- $1 partitions) and is usually expressed as a floating point number. **2. Quantized code **
int8 Quantizes python code:
import numpy as np
x= input_float/scale
x=np.clip(np.floor(x+0.5),-128,127)
x = x.astype(np.int8)
int16 Quantizes python code:
import numpy as np
x= input_float/scale
x=np.clip(np.floor(x+0.5),-32768,32767)
x = x.astype(np.int16)
int8 Quantizes C++ code:
float features_tmp = std::floor(static_cast<float>(x_float/scale)+0.5);
features_tmp = std::min(std::max(features_tmp, -128.f), 127.f);
features_tmp= static_cast<int8_t>(features_tmp);
int16 Quantizes C++ code:
float features_tmp = std::floor(static_cast<float>(x_float/scale)+0.5);
features_tmp = std::min(std::max(features_tmp, -32768.f), 32767.f);
features_tmp = static_cast<int16_t>(features_tmp);
1.2 Dequantize
The Dequantize node is used to inversely quantize the output data of the integer type in the model back to the float type, and its calculation formula is as follows:
deqx=(x−zero_point)∗scaledeqx = (x - zero\_point) * scaledeqx=(x−zero_point)∗scale
Here is the C++ implementation code for the Dequantize node:
static_cast<float>(value) * scale
Here the zero_point and scale have the same meaning as in the Quantize node.
2. The basic process of inverse quantization node fusion
Under normal circumstances, for the BPU/CPU hybrid heterogeneous model, the time taken under single frame serial includes the following aspects:
‘Input quantized CPU node + model BPU operator + output inverse quantized CPU node +CPU post-processing’
- Enter a quantized CPU node: float32->int8. Only the featuremap input model includes this node. The image input model does not include this node.
- Output inverse quantization CPU node: int8->float32;
In addition, according to the background knowledge of neural network quantitative calculation, we can know that:
- The essence of the quantization/inverse quantization operation of the neural network in the model deployment stage is the point-by-point elementwise multiplication and addition operation of the input tensor, that is, the operation of the weights in the neural network and the quantization parameters scale and zero point;
- Multiplication and addition operations and data transmission consume much of the energy consumed during neural network reasoning; The multiplication and addition operation requires traversing each element of the tensor, which takes more time than the multiplication and addition operation.
For the network model with pre and post processing, it must perform data traversal operations, so the Quantize and Dequantize nodes at the head and tail of the model can be manually removed during the model quantization process, and then the Quantize and Dequantize operations can be integrated into the code of the model pre and post processing. To reduce the loss of repeated data traversal.
2.1 Inverse quantization node is deleted
There are two main methods for removing the Dequantize node in the tail of the model provided by the Horizon AI toolchain:
1. Configure yaml file
Before converting the model using hb_mapper makertbin tool, parameters in the yaml file need to be configured. In the model parameter group of the yaml file, the type of node to be deleted can be set:
remove_node_type: "Quantize;Dequantize"
This removes the Quantize and Dequantize nodes on the model input and output sides. It should be mentioned here that yaml parameters are set to remove only nodes at the beginning or end of the model.
2. hb_model_modifier tool
After the model is compiled, the ‘hb_model_modifier’ tool can be used to delete the node on the input and output of the specified runtime model, and the information about the deleted node is stored in the BIN model. Delete the Dequantize node in the tail of the compiled.bin model as follows:
hb_model_modifier model.bin -a Dequantize
hb_model_modifier model.bin -r node1
2.2 scale value view
Quantization scale is a necessary factor for inverse quantization calculation. After deleting the Dequantize node, it is still necessary to obtain the scale value of the deleted node for subsequent inverse quantization fusion operation. You can view the scale value in the following ways:
1. Way 1
On the developer side, when a Dequantize node is removed using either of the above methods, the ‘hb_model_Modifer.log’ file is generated, which contains the name and scale value of the removed Dequantize node. As shown in the picture below:
2. Way 2
You can also run the hb_model_info tool on the development machine to view the information of the deleted node. The name of the deleted node will be printed at the end of the output information, and the deleted_nodes_info.txt file will be generated. Each line in the file records the initial information of the corresponding deleted node. For example, the name and scale value of the Dequantize node are deleted. Run the ‘hb_model_info’ tool on the developer side:
hb_model_info model.bin
The resulting ‘deleted_nodes_info.txt’ file contains the following information:
3. Way 3
In addition, you can also run the ‘hrt_model_exec’ tool on the board to obtain the deleted node information, and the terminal will print out the scale information of the deleted node. Run the hrt_model_exec tool on the board:
hrt_model_exec model_info --model_file model.bin
The information about the node to be deleted is as follows:
4. Way 4
In addition, the runtime also provides the BPU SDK apis’ hbDNNTensor ‘and’ hbDNNTensorProperties’ to obtain scale values. First, ‘APIhbDNNTensor’ is used to define the Tensor that stores the input and output information of the model, and then the member scale of ‘hbDNNTensorProperties’ is used to obtain the scale information of the Dequantize node on the output side of the model. See the example in section 2.3 for details. The following is the definition and membership of the two apis:
//hbDNNTensor
typedef struct {
hbSysMem sysMem[4];
hbDNNTensorProperties properties;
} hbDNNTensor;
Member name
Description
sysMem
The memory in which the tensor is stored
properties
Information about the tensor
//hbDNNTensorProperties
typedef struct {
hbDNNTensorShape validShape;
hbDNNTensorShape alignedShape;
int32_t tensorLayout;
int32_t tensorType;
hbDNNQuantiShift shift;
hbDNNQuantiScale scale;
hbDNNQuantiType quantiType;
int32_t alignedByteSize;
} hbDNNTensorProperties;
Member name
Description
validShape
The shape of the valid content of the tensor
alignedShape
The shape of the tensor that aligns the content
tensorLayout
The tensor is arranged in the form
tensorType
Indicates the offset of fixed-point to floating-point
shift
Indicates the shift from fixed point to floating point
scale
The scale of the fixed-point to floating point
quantiType
The fixed-point to floating-point quantization type
alignedByteSize
The size of memory for the aligned content of the tensor
2.3 Example of inverse quantization node fusion
Horizon provides several examples of de-quantization node fusion in the Open Explorer development package horizon_xj3_open_explorer_v1.14.3_20220727/ddk/samples/ai_benchmark. The object detection model efficientdetd0 was used as an example to describe the process of inverse quantization node fusion.
- Remove the Dequantize node. The Dequantize node in the tail of the model can be removed in two ways described in Section 2.1.
- Obtain the value of the quantized scaling factor scale. Model of Dequantize after deleting nodes, delete nodes in the model of information will be stored in a BIN, you can use
tensor. The properties, scale, scaleDatato obtain scale values. - Integrate the obtained scale value into the post-processing part of the model for inverse quantization calculation, so as to avoid the redundant operation of a data traversal.
The C++ implementation code of the inverse quantization node in runtime is as follows:
static_cast<float>(value) * scale
In efficientdetd0 / DDK sample post-processing code/samples/ai_benchmark/code/SRC/method/ptq_efficientdet_post_process_method cc, Use the BPU SDK API ‘hbDNNTensor’ to store the classification header information c_tensor and detection header information bbox_tensor:
hbDNNTensor *c_tensor,
hbDNNTensor *bbox_tensor,
Then use the pointer to get the quantized scale factor scale values’ cls_scale_data ‘and’ box_scale_data 'from c_tensor and bbox_tensor:
float *cls_scale_data = c_tensor->properties.scale.scaleData;
float *box_scale_data = bbox_tensor->properties.scale.scaleData;
Finally, the inverse quantization is calculated according to the above formula, and then other post-processing operations are carried out. The following is part of the C++ code for the inverse quantization of the efficientdetd0 example:
auto *raw_cls_data =
reinterpret_cast<int32_t *>(c_tensor->sysMem[0].virAddr);
auto *raw_box_data =
reinterpret_cast<int32_t *>(bbox_tensor->sysMem[0].virAddr);
for (int i = 0; i < box_num; i++) {
// score and cls
uint32_t res_id_cur_anchor = i * class_num;
int cls_scale_index = res_id_cur_anchor % 720;
float *cls_scale_data = c_tensor->properties.scale.scaleData;
auto cls_scales = cls_scale_data + cls_scale_index;
auto max_score_id =
MaxScoreID(raw_cls_data + res_id_cur_anchor, cls_scales, class_num);
float max_score = max_score_id.first;
int max_id = max_score_id.second;
// box
if (max_score <= score_threshold_) continue;
// stride is 40, if do not have dequanti node
int start = i * 4 + (4 * i) / 36 * 4;
int scale_index = (i * 4) % 36;
float *box_scale_data = bbox_tensor->properties.scale.scaleData;
auto box_scales = box_scale_data + scale_index;
float dx = raw_box_data[start] * box_scales[0];
float dy = raw_box_data[start + 1] * box_scales[1];
float dw = raw_box_data[start + 2] * box_scales[2];
float dh = raw_box_data[start + 3] * box_scales[3];
float width = anchors[i].w_;
float height = anchors[i].h_;
float ctr_x = anchors[i].c_x_;
float ctr_y = anchors[i].c_y_;
float pred_ctr_x = dx * width + ctr_x;
float pred_ctr_y = dy * height + ctr_y;
float pred_w = std::exp(dw) * width;
float pred_h = std::exp(dh) * height;
float xmin = (pred_ctr_x - 0.5f * (pred_w - 1.f));
float ymin = (pred_ctr_y - 0.5f * (pred_h - 1.f));
float xmax = (pred_ctr_x + 0.5f * (pred_w - 1.f));
float ymax = (pred_ctr_y + 0.5f * (pred_h - 1.f));
Bbox bbox;
bbox.xmin = std::max(xmin, 0.f);
bbox.ymin = std::max(ymin, 0.f);
bbox.xmax = std::min(xmax, img_w);
bbox.ymax = std::min(ymax, img_h);
dets.push_back(
Detection(max_id,
max_score,
bbox,
efficient_det_config_.class_names[max_id].c_str()));
}
}
return 0;
}
3. References
- Nagel M, Fournarakis M, Amjad R A, et al. A white paper on neural network quantization[J]. arXiv preprint arXiv:2106.08295, 2021.
- Gholami A, Kim S, Dong Z, et al. A survey of quantization methods for efficient neural network inference[J]. arXiv preprint arXiv:2103.13630, 2021.
- https://pytorch.org/blog/introduction-to-quantization-on-pytorch/
- https://www.tensorflow.org/api_docs/python/tf/quantization