hbdk-model-verifier, hbdk-sim, hbdk-hbm-attach Tool Usage Tutorial

Application Development Algorithm Toolchain
,
1

1. hbdk-model-verifier

1.1 Introduction to the tool The hbdk-model-verifier tool is developed by Horizon to verify the result consistency of specified fixed-point pt models and hbm deployment models and output the model’s predicted execution time. ** The following two issues need to be noted when using the hbdk-model-verifier tool: **

  1. The version of the HBDk-Model-verifier tool must be consistent with the hbdk version used to compile the hbm model.
  2. If you do not care about the time required, use the random input data automatically generated by the tool. No manual input data is required.

1.2 Obtaining method

  • docker image The docker image of the Horizon algorithm toolchain already contains the hbdk-model-verifier tool, which can be used directly in docker.

  • Local installation If you want to install this tool locally, the whl package is provided in the ddk/package/host/ai_toolchain directory of the OE development kit. You only need to install the following whl package in your local python environment.

    pip install hbdk-{version}-cp38-cp38-linux_x86_64.whl
    pip install hbdk_model_verifier-{version}-py3-none-linux@

1.3 Usage Scenarios and methods

1.3.1 pyramid Input model consistency verification

1.3.1.1 Obtaining Input Data

When the model input is pyramid and the input data format is NV12 images, jpg, png or yuv files can be used as input data. Here are some things to note:

  1. If jpg and png images are used as inputs, hbdk-model-verifier only does decoding (such as jpg->bgr) and color space conversion (bgr->yuv444) when processing image input, and does not scale the image, so it needs to prepare pictures with the same size as the model input.
  2. If the yuv file is used as input, you need to use the --yuv-shape parameter to give the yuv file size HxW. The code for converting rgb/bgr image to NV12 image can refer to PTQ&QAT scheme board end verification precautions.

The model name and input and output information can be obtained by running the hrt_model_exec tool on the x86 or development board.

hrt_model_exec model_info --model_file model.hbm

If only model consistency verification is performed, random input data automatically generated by the tool is sufficient, and manual input data preparation is not required.

1.3.1.2 Run the command

  1. Verify the consistency of pyramid input model inference and hbm model inference results. The command line example is as follows:

    hbdk-model-verifier --hbm model.hbm --model-pt model.pt --model-input nv12.yuv --yuv-shape {HxW} --ip {board_ip} --times 10

If there is no development board, you can use x86 emulator instead, just configure the --skip-bpu parameter, the command line example is as follows:

hbdk-model-verifier --hbm model.hbm --model-pt model.pt --model-input nv12.yuv --yuv-shape {HxW} --skip-bpu

** Parameter Description: ** --hbm (required) : hbm model compiled by Horizon algorithm Toolchain; – – Model-pt (required) : torchscript fixed-point model pt;

  • -- Model-input (optional) : model input data. If this parameter is not specified, the tool automatically generates input data.
  • --yuv-shape: Enter the shape of NV12. The value is pyramid/resizer. The configuration format is HxW.
  • --ip: indicates the IP address of the development board.
  • --times: indicates the number of frames for inference. The default value is 1. When evaluating the model performance, it is generally necessary to deduce several times, and then take the average time. This parameter is not configured when verifying the model consistency. --skip-bpu: Use x86 emulator instead of BPU; – --force-run-simulator: No matter whether the BPU development board is connected remotely or not, the x86 simulator is invoked for inference, and the simulator results are compared.

To obtain more parameter descriptions, run hbdk-model-verifier --help.

1.3.2 resizer input model consistency verification

1.3.2.1 Obtaining Data

When the model input is resizer and the input data format is also an NV12 image, jpg, png or yuv file can be used as the input data, but the starting coordinates or dimensions of ROI need to be specified in the image as the model input.

1.3.2.2 Running commands
hbdk-model-verifier --hbm model.hbm --model-pt model.pt --model-input nv12.yuv --yuv-shape {HxW} --roi-coord {YxX} --roi-size {roi_hxroi_w} --ip {board_ip}

** Parameter Description ** :

  • --roi-coord/–roi-size: The option is followed by a string representing the coordinates/sizes of the ROI starting point, in the format of Y coordinates /ROI height + “x” + X coordinates /ROI width. The ROI of the resizer model and the model input size need to meet the hardware limitations of resizer itself. The ROI size of the pyramid model needs to be consistent with the model input.

    resizer model introduction can refer to the use and deployment] [resizer model (https://developer.horizon.cc/forumDetail/116476319759489550).

1.3.3 ddr input model consistency verification

1.3.3.1 Obtaining Input Data

When the model input is ddr, then you need to use binary data bin file or txt format data as input. It should be noted that regardless of the layout required by the model itself, the data entered by the hbdk-model-verfier tool must be stored using NHWC_NATIVE.

  1. If the binary data is used, the data type of the binary data must be consistent with that of the model input.

  2. If txt text is used, numpy.loadtxt can be read correctly.

    NHWC_Native data arrangement: means that the data arrangement of a multidimensional array is NHWC, and only the order of dimensions affects the location of data storage.

1.3.3.2 Run the command

To verify the consistency of ddr input model inference and hbm model inference results, the command line example is as follows:

hbdk-model-verifier --hbm model.hbm --model-pt model.pt --model-input featuremap.bin --ip {board_ip} --times 100

If you do not have a development board, you can configure the --skip-bpu parameter on the command line to use the x86 emulator to reason about the hbm model.

1.3.4 Multi-input model consistency verification

1.3.4.1 Obtaining Input Data

When the model has multiple inputs, input data should be prepared according to different input types. Refer to the previous input data preparation for pyramid, resizer, and ddr models. What should be noted is:

  1. If the model itself has multiple inputs, the input data are separated by commas, and the corresponding parameters describing the input information such as --yuv-shape/–roi-coord/–roi-size/–image-stride are also separated by commas;
  2. If multiple inputs to the model come from different sources, and some inputs do not have a parameter like --yuv-shape, then commas can be left empty.
1.3.4.2 Running commands

Suppose the model has 4 inputs, namely ddr, resizer, ddr and pyramid inputs. The HxW input of resizer is 540x960, the starting coordinate of ROI is 0x0, and the size is 128x128. pyramid input HxW is 540x960; The file names are input_0.bin, input_1.yuv, input_2.bin, input_3.yuv. The following is an example of the command line:

hbdk-model-verifier --hbm model.hbm --model-pt model.pt --model-input input_0.bin,input_1.yuv,input_2.bin,input_3.yuv --yuv-shape,540x960,,540x960 
--roi-coord ,0x0,, --roi-size ,128x128,, --image-stride ,960,,960 --ip {board_ip}

** Parameter Description: ** --image-stride: Adjust the W span of pyramid/resizer. The default is “None”. In some scenarios, image-stride may not be equal to W dimension, so you need to adjust this parameter.

1.4 Output results

The hbdk-model-verifier log output information includes the execution time of the model on the development board and the verifier result. The following is an interpretation of the output log that verifies the consistency of the mobilenetv1 sample pt model and hbm model using the hbdk-model-verifier tool.

1.4.1 Time consuming information

09-22-2023 14：26：35 root：INFO：Verifier Total Time on dev board with connection time (including BPU， CPU， IO， network and time to wait for lock)： 9.485120 ms
09-22-2023 14：26：35 root：INFO：Verifier Total Time on dev board without connection time (including BPU， CPU， IO and network time)： 8.821064 ms
09-22-2023 14：26：35 root：INFO：
======> Model execution time (including BPU and CPU)： 0.811332 ms
======> BPU execution time (BPU function call consumed time)： 0.682924 ms
======> CPU execution time (including context switch and cpu operator)： 0.128408 ms

Note:

  1. The time consuming information about a single thread with a single core is displayed in the log file.
  2. If the --skip-bpu parameter is configured, that is, the x86 emulator is used to reason the model, and the output log does not include the time consuming information.
  3. If random input is used, the output time may be different from the actual time.

1.4.2 verifier result

If [SUCCESS] output file [{output_node_name}.txt] is same in the log, the consistency verification succeeds; otherwise, the verification fails. The following information is displayed when the consistency verification is successful:

09-22-2023 14：26：42 root：INFO：======> Compare Results on BPU Board vs. Framework
09-22-2023 14：26：42 root：INFO： [SUCCESS] output file [hbdk_output__backbone_output_1_0_hz_conv2d.txt] is same.

2. hbdk-sim

2.1 Tool Introduction

The hbdk-sim tool is developed by Horizon for simulating BPU inference to hbm models in x86 environments.

2.2 Obtaining method

The docker image of the Horizon algorithm toolchain already contains the hbdk-sim tool, which can be used directly in docker. 2.3 Usage Scenarios and Modes

2.3.1 Obtaining Input Data

Refer to sections 1.3.1, 1.3.2, 1.3.3 to prepare different types of input data.

2.3.2 Run the command

  1. Simulate the pyramid input model. The command line example is as follows:

    hbdk-sim --model-name {model_name} --hbm model.hbm --input-binary nv12.yuv --yuv-size {HxW} --output ./result

  2. Simulate inference resizer input model. The command line example is as follows:

    hbdk-sim --model-name {model_name} --hbm model.hbm --input-binary nv12.yuv --yuv-shape {HxW} --roi-coord {YxX} --roi-size {roi_hxroi_w} --output ./result

The ROI of the resizer model and the model input size need to meet the hardware limitations of resizer itself.

  1. Simulate the inference ddr input model. The command line example is as follows:

    hbdk-sim --model-name {model_name} --hbm model.hbm --input-binary featuremap.bin --output ./result

  2. Simulate inference complex input model. The command line example is as follows: Suppose the model has 4 inputs, namely ddr, resizer, ddr and pyramid inputs. The HxW input of resizer is 540x960, the starting coordinate of ROI is 0x0, and the size is 128x128. pyramid input HxW is 540x960; The file names are input_0.bin, input_1.yuv, input_2.bin, input_3.yuv. The following is an example of the command line:

    hbdk-sim --hbm model.hbm --input-binary input_0.bin，input_1.yuv,input_2.bin,input_3.yuv --yuv-shape,540x960,540x960 --roi-coord ,0x0, --roi-size ,128x128, --image-stride ,960,960

** Parameter Description: **

  • --model-name(Mandatory) : specifies the name of the hbm model. --hbm(required) : hbm model compiled by Horizon algorithm Toolchain;
  • --input-binary (Mandatory) : indicates model input data.
  • --output (Mandatory) : output saving path.
  • --yuv-size: indicates the shape of the NV12 input. The value is pyramid/resizer. The configuration format is HxW. – --yuv-roi-coord/–yuv-roi-size: The initial coordinates and dimensions of ROI entered by resizer. The configuration format is Y coordinates /ROI height + “x” + X coordinates /ROI width. To obtain more parameter descriptions, run the hbdk-sim --help command.

2.4 Output results

The output of hbdk-sim includes time consuming information, bandwidth usage information, memory usage information, and inference results. Inference results are saved in a.txt file under the configured --output path. Other information is displayed in the log of the terminal, as follows:

****************************************************             
                     Latency Info                   
* Estimated model latency： 567 us

                  DDR Bandwidth Info                
* Total DDR bandwidth： 4156608 bytes
* Bpu read           ： 4140608 bytes
* Bpu write          ： 16000 bytes

                  Memory Usage Info                 
* Bpu side memory usage peak value： 5.191 MB
* Input memory                    ： 75264 Bytes
* Output memory                   ： 16000 Bytes

3. hbdk-hbm-attach

3.1 Tool Introduction

hbdk-hbm-attach tool is an hbm file assistant tool developed by Horizon. It is used to update or read desc(description) information in hbm file. It has the following functions:

  1. Update/extract desc (tag) information in hbm file, desc information of model, and desc information of model input and output feature;
  2. Modify the input and output feature name of the model.

desc information is information that describes model attributes (such as version, task type, developer, etc.) and is added by the user.

3.2 Use process

The hbdk-hbm-attach tool can update the description of the hbm model in the following three steps:

  1. Extract the json template for updating the description information.
  2. Modify the description in the json file.
  3. Run the description update command.
  4. Result verification.

This paper will take the hbm model of mobilenetv1 as an example to explain the flow of updating desc information of the model.

3.2.1 Extract the json template used to update the description

Before updating the description information of the hbm model, we need to extract the json template of the description information of the hbm model. The reference command is as follows:

hbdk-hbm-attach model.hbm


{
    "models": {
        "mobilenetv1_cls": {
            "desc": "",
            "desc_type": "string",
            "input_features": {
                "arg0[img]": {
                    "desc": "",
                    "desc_type": "string"
                }
            },
            "output_features": {
                "_backbone_output_1_0_hz_conv2d": {
                    "desc": "",
                    "desc_type": "string"
                }
            }
        }
    },
    "tag": ""
}

The model.json extracted in the preceding process may not be modified due to permission issues. You can use chmod 777 model.json to modify the file permission.

3.2.2 Modifying the json File

After extracting the json template of the desc information of the model, you can modify the desc information and feature name in the following ways:

  • If you want to modify the desc information of a feature; The value of the “desc” field can be edited directly;
  • If you want to modify the name of a feature, you only need to add the reserved keyword “new_name” to the dictionary corresponding to the feature name to modify the input/output node name.

If you do not want to modify the desc information of a feature, do not include the key “new_name” in the corresponding dictionary.

For example, we can modify the json template extracted in Step 1 as follows:

{
    "models": {
        "mobilenetv1_cls": {
            "desc": "This is mobilenetv1 hbm",
            "desc_type": "string",
            "input_features": {
                "arg0[img]": {
                    "desc": "shape of input is 1x3x224x224",
                    "desc_type": "string",
                    "new_name" :"horizon_input"

                }
            },
            "output_features": {
                "_backbone_output_1_0_hz_conv2d": {
                    "desc": "shape of output is 1x1000x1x1",
                    "desc_type": "string",
                    "new_name" :"horizon_output"
                }
            }
        }
    },
    "tag": "Journey to AD"
}

3.2.3 Run the description update command

When the desc information of the hbm model is updated, the json file is the tool input. The file must be stored in the same directory as the hbm file to be updated, and the file name must meet certain rules. For example, the desc file corresponding to the xxx.hbm file is named xxx_desc.json. After modifying the json file, run the following command to update the desc information of the hbm:

hbdk-hbm-attach model.hbm --update

After the command is executed, the following information is displayed on the terminal:

INFO: Will update hbm
INFO: Update feature name from "arg0[img]" to "horizon_input"
INFO: Update feature name from "_backbone_output_1_0_hz_conv2d" to "horizon_output"

3.2.4 Verification Result

After the desc information of the hbm model is updated, we can run the following command again to verify that the desc information has been successfully modified:

hbdk-hbm-attach model.hbm

After the command is executed, the model.json file generated is as follows:

{
    "models": {
        "mobilenetv1_cls": {
            "desc": "This is mobilenetv1 hbm",
            "desc_type": "string",
            "input_features": {
                "horizon_input": {
                    "desc": "shape of input is 1x3x224x224",
                    "desc_type": "string"
                }
            },
            "output_features": {
                "horizon_output": {
                    "desc": "shape of output is 1x1000x1x1",
                    "desc_type": "string"
                }
            }
        }
    },
    "tag": "Journey to AD"
}

As you can see from the extracted model.json file, the desc information for input_features and output_features and the model tag have been successfully updated.