Getting Your AI Model Running on RDK with the OE Toolchain

Shockley · July 3, 2026, 7:09am

0. Toolchain Versions — Which One Do You Need?

D-Robotics ships a separate toolchain package per hardware platform.
Before anything else, match your board to the right Docker image and SDK.

Platform Overview

Platform	BPU Architecture	TOPS	Typical Use Case	Toolchain
RDK X5	bayes-e	~10	Edge vision (this workshop)	OE v1.2.8
RDK X3	bernoulli2	~5	Entry-level edge vision	OE v2.6.6
RDK S100	nash-e	~80+	High-perf vision + DSP	OE v3.7.0
LLM on S100	nash-e	—	Large language model deployment	OELLM

OE Toolchain Downloads

Docker login (required for registry pulls):

docker login -u "ccr$deliver-ronly" registry.d-robotics.cc \
             -p 'VLaeatrjF9yGf6I44trT74zKhUpZSVlr'

RDK X5 — OE v1.2.8 ← (this workshop)

# Docker (online)
docker pull registry.d-robotics.cc/deliver/ai_toolchain_ubuntu_20_x5_cpu:v1.2.8
docker pull registry.d-robotics.cc/deliver/ai_toolchain_ubuntu_20_x5_gpu:v1.2.8

# Offline tarball
wget https://d-robotics-aitoolchain.oss-cn-beijing.aliyuncs.com/oe_x5/1.2.8/docker_openexplorer_ubuntu_20_x5_cpu_v1.2.8.tar.gz
wget https://d-robotics-aitoolchain.oss-cn-beijing.aliyuncs.com/oe_x5/1.2.8/docker_openexplorer_ubuntu_20_x5_gpu_v1.2.8.tar.gz

# SDK + Docs
wget https://d-robotics-aitoolchain.oss-cn-beijing.aliyuncs.com/oe_x5/1.2.8/horizon_x5_open_explorer_v1.2.8-py310_20240926.tar.gz
wget https://d-robotics-aitoolchain.oss-cn-beijing.aliyuncs.com/oe_x5/1.2.8/x5_doc-v1.2.8-py310-en.zip

RDK S100 — OE v3.7.0

# Docker (online)
docker pull registry.d-robotics.cc/deliver/ai_toolchain_ubuntu_22_s100_s600_cpu:v3.7.0
docker pull registry.d-robotics.cc/deliver/ai_toolchain_ubuntu_22_s100_s600_gpu:v3.7.0

# Offline tarball
wget https://d-robotics-aitoolchain.oss-cn-beijing.aliyuncs.com/oe/3.7.0/ai_toolchain_ubuntu_22_s100_s600_cpu_v3.7.0.tar

# SDK + Docs
wget https://d-robotics-aitoolchain.oss-cn-beijing.aliyuncs.com/oe/3.7.0/oe-package-3.7.0-s100-s600.tgz
wget https://d-robotics-aitoolchain.oss-cn-beijing.aliyuncs.com/oe/3.7.0/oe-doc-3.7.0-s100-s600.zip

RDK X3 — OE v2.6.6

docker pull registry.d-robotics.cc/deliver/ai_toolchain_ubuntu_20_xj3_cpu:v2.6.6-py38

wget https://d-robotics-aitoolchain.oss-cn-beijing.aliyuncs.com/oe_x3/2.6.6/horizon_xj3_open_explorer_v2.6.6_py38_20240717.tar.gz

OELLM — Large Language Model Quantization

For deploying LLMs (Qwen, LLaMA, etc.) on D-Robotics hardware, there is a dedicated toolchain separate from the standard OE:

Version	Platform	Notes
OELLM v1.0.0	S100	LLM quantization and deployment

# S100
wget https://d-robotics-aitoolchain.oss-cn-beijing.aliyuncs.com/llm_s100/1.0.0/D-Robotics_LLM_S100_1.0.0_SDK.tar.gz
wget https://d-robotics-aitoolchain.oss-cn-beijing.aliyuncs.com/llm_s100/1.0.0/D-Robotics_LLM_S100_1.0.0_Doc.zip

Why Do We Need Quantization?

A standard PyTorch model runs as 32-bit floating-point (FP32) on a GPU.
The Horizon BPU is an INT8 hardware accelerator — it runs on 8-bit integers.

	Float32 Model	INT8 Quantized Model
Compute	GPU/CPU (FP32)	BPU (INT8)
Memory	~4× larger	~1× baseline
Speed (ResNet18)	~30 ms (CPU)	~2.3 ms (BPU)
Accuracy loss	baseline	typically < 0.5%

Post-Training Quantization (PTQ) converts a trained FP32 model to INT8 without retraining.
All you need is:

The exported ONNX model
A small set of representative calibration images (~50–200)
A YAML config file

1. The Full PTQ Pipeline

┌──────────────────────────────────────────────────────────────────┐
│  Training  (PyTorch, your GPU machine)                           │
│  resnet18 pretrained ──► export_onnx ──► resnet18_opset11.onnx   │
└────────────────────────────┬─────────────────────────────────────┘
                             │ ONNX (FP32)
┌────────────────────────────▼─────────────────────────────────────┐
│  Quantization  (OE Toolchain, Docker on x86)                     │
│                                                                  │
│  calibration images (33 jpgs)                                    │
│       └─► prepare_calibration_data.py  ──► .fm.bin files         │
│                                                                  │
│  hb_mapper makertbin --config resnet18_config.yaml               │
│       ├─ Calibrate   (find optimal INT8 scales per layer)        │
│       ├─ Quantize    (FP32 weights → INT8)                       │
│       └─ Compile     (BPU instruction scheduling)                │
│            └─► resnet18_224x224_featuremap.bin   ◄── BPU model   │
└────────────────────────────┬─────────────────────────────────────┘
                             │ .bin
┌────────────────────────────▼─────────────────────────────────────┐
│  Runtime Inference  (RDK X5 board)                               │
│                                                                  │
│  Python:  hbm_runtime  ──► infer_resnet18.py                     │
│  C++:     hb_dnn       ──► infer_resnet18.cpp                    │
└──────────────────────────────────────────────────────────────────┘

1.5 PTQ vs QAT — Which One Do You Need?

The Horizon toolchain offers two paths to get a quantized BPU model.
This workshop covers PTQ. Here is why, and when you’d choose the other.

The Two Paths at a Glance

Your trained FP32 model
        │
        ├─── PTQ (this workshop) ──────────────────────────────────────┐
        │    Tool: hb_mapper makertbin                                  │
        │    Input: ONNX + ~50 calibration images + YAML config        │
        │    Time: minutes, no GPU needed                               │
        │    Output: .bin  ready to deploy                              │
        │                                                               │
        └─── QAT ──────────────────────────────────────────────────────┐
             Tool: horizon_plugin_pytorch  (PyTorch plugin)            │
             Input: your original training code + dataset              │
             Time: hours–days of retraining on GPU                     │
             Output: fixed-point model → compile to .bin               │

Side-by-Side Comparison

	PTQ (Post-Training Quantization)	QAT (Quantization-Aware Training)
What it does	Calibrates INT8 scales from a small dataset; no retraining	Inserts fake-quantization nodes, fine-tunes the whole model with the loss function
Tool	`hb_mapper makertbin` (CLI)	`horizon_plugin_pytorch` (Python/PyTorch)
Prerequisite	Exported ONNX + 50–200 representative images	Access to original training code, training dataset, GPU
Time	Minutes	Hours to days
Accuracy	Typically < 0.5% drop for CNNs	Recovers most of PTQ’s accuracy loss
When to use	Default first choice. Standard CNNs almost always pass.	PTQ cosine similarity < 0.99, or per-layer accuracy unacceptable
Skill required	Minimal — just fill in the YAML	Must understand model internals; requires modifying training code

2. Environment Setup

2.1 Get the OE Toolchain Docker Image

The toolchain ships as a Docker image. There are two ways to get it:

Option A — Pull directly from Horizon’s registry (recommended):

docker pull registry.d-robotics.cc/deliver/ai_toolchain_ubuntu_20_x5_cpu:v1.2.8

Option B — Download the image tarball via FTP (offline / slow network):

# Download (~7 GB, resume-capable)
wget -c ftp://oeftp@sdk.d-robotics.cc/OE/v1.2.8/ai_toolchain_ubuntu_20_x5_cpu_v1.2.8.tar.xz \
     --ftp-password=Oeftp~123$%

# Load into Docker
docker load -i ai_toolchain_ubuntu_20_x5_cpu_v1.2.8.tar.xz

The FTP server also hosts model conversion samples and documentation:
ftp://oeftp@sdk.d-robotics.cc/ (password: Oeftp~123$%)

Other toolchain versions are listed at:
https://developer.d-robotics.cc/rdk_x_doc/en/Advanced_development/toolchain_development/expert/environment_config

2.2 Create the Persistent Container

We create the container once and reuse it every session:

# First time only — create the container
docker run -it \
  --name oe_x5_ptq \
  -v ~/resnet18/ptq_demo:/workspace/resnet18/ptq_demo \
  registry.d-robotics.cc/deliver/ai_toolchain_ubuntu_20_x5_cpu:v1.2.8 \
  bash

# Every subsequent time — just start it
docker start oe_x5_ptq

# Run a command without entering the shell
docker exec -w /workspace/resnet18/ptq_demo/config oe_x5_ptq \
  hb_mapper makertbin --model-type onnx --config resnet18_config.yaml

Why Docker? The toolchain (hb_mapper) has specific Python and library version requirements. Packaging it in Docker avoids conflicts with your host system.

2.3 Project Structure

This section introduces the demo project used throughout the workshop. The full source code is available on GitHub:

https://github.com/shockley6668/resnet_ptq_demo

resnet18/ptq_demo/
├── model/                          # ONNX models
├── config/
│   └── resnet18_config.yaml        # Quantization config ← KEY FILE
├── calibration_data/               # Raw calibration images (33 jpgs)
├── calibration_data_featuremap/    # Preprocessed binary files (33 .fm.bin)
├── model_output/
│   └── resnet18_224x224_featuremap.bin   ← Final BPU model
├── scripts/
│   ├── export_resnet18_onnx.py
│   └── prepare_calibration_data_featuremap.py
└── board_infer/
    ├── infer_resnet18.py           # Python inference (hbm_runtime)
    └── infer_resnet18.cpp          # C++ inference (hb_dnn)

3. Step 1 — Export Your Model to ONNX

The BPU toolchain starts from ONNX, not PyTorch .pth files.

# scripts/export_resnet18_onnx.py
import torch
import torchvision.models as models

model = models.resnet18(pretrained=True)
model.eval()

dummy = torch.randn(1, 3, 224, 224)
torch.onnx.export(
    model, dummy,
    "model/resnet18_opset11.onnx",
    opset_version=11,          # OE supports opset 9–13
    input_names=["input"],
    output_names=["output"],
    dynamic_axes=None,         # Fixed shape — required for BPU
)

Run inside Docker:

docker exec -w /workspace/resnet18/ptq_demo oe_x5_ptq \
  python3 scripts/export_resnet18_onnx.py

Generalize to your own model:

Keep opset_version=11 (safe choice)

Use fixed input shape — dynamic shapes are not supported on BPU

Check supported operators: hb_mapper checker --model-type onnx --model your_model.onnx

Unsupported ops fall back to CPU automatically — expect a speed penalty

Validate the ONNX Export

docker exec -w /workspace/resnet18/ptq_demo oe_x5_ptq \
  hb_mapper checker --model-type onnx \
  --march bayes-e \
  --model model/resnet18_opset11.onnx

This reports which operators run on BPU vs CPU. Aim for 100% BPU.

4. Step 2 — Prepare Calibration Data

What Is Calibration?

PTQ works by finding the best scale factor for each INT8 tensor.
To do that, the quantizer needs to observe real activation values from your data.

calibration image → forward pass (FP32) → record min/max per layer
                                         → compute optimal INT8 scale

Rule of thumb: 50–200 diverse, representative images are sufficient.
More is not always better — diversity matters more than quantity.

Our Preprocessing Pipeline

For featuremap input mode, calibration data must match exactly what you’ll feed at runtime:

# (pixel/255 - mean) / std  →  float32 NCHW binary file
img  = ShortSideResize(256) → CenterCrop(224) → RGB
arr  = arr.astype(float32) / 255.0
arr  = (arr - MEAN) / STD          # ImageNet normalization
arr  = arr.transpose(2,0,1)        # HWC → CHW (NCHW)
arr.tofile("calib_000.fm.bin")

Run:

docker exec -w /workspace/resnet18/ptq_demo/scripts oe_x5_ptq \
  python3 prepare_calibration_data_featuremap.py

5. Step 3 — The Quantization Config

This is the most important file. Let’s walk through every section:

# resnet18_config.yaml

model_parameters:
  onnx_model: '../model/resnet18_opset11.onnx'
  march: 'bayes-e'            # Target BPU: bayes-e=X5, bayes=J5, bernoulli2=X3
  output_model_file_prefix: 'resnet18_224x224_featuremap'
  working_dir: '../model_output'

input_parameters:
  input_name: 'input'         # Must match your ONNX input node name

  # --- Input type on the BOARD at runtime ---
  input_type_rt: 'featuremap' # float32 passthrough — simplest option
  input_layout_rt: 'NCHW'     # Channel-first — matches PyTorch convention

  # --- Input type the original ONNX model expects ---
  input_type_train: 'featuremap'
  input_layout_train: 'NCHW'

  input_shape: '1x3x224x224'
  input_batch: 1
  norm_type: 'no_preprocess'  # We handle normalization ourselves

calibration_parameters:
  cal_data_dir: '../calibration_data_featuremap'
  cal_data_type: 'float32'
  preprocess_on: False         # We pre-processed manually
  calibration_type: 'default'  # Auto-search for best quantization method
  per_channel: True            # Per-channel scales → better accuracy

compiler_parameters:
  compile_mode: 'latency'      # Optimize for speed (vs 'bandwidth')
  optimize_level: 'O2'
  core_num: 1                  # X5 has 2 BPU cores; use 2 for throughput
  jobs: 8

Input Type Reference

Full list of supported values (from official docs):

`input_type_rt`	Board dtype	Normalization location	Typical use
`featuremap`	float32	Your code	Custom models, safest choice
`rgb`	int8 (pixel−128)	Inside model (HzPreprocess node)	Standard RGB image pipeline
`bgr`	int8 (pixel−128)	Inside model	OpenCV BGR pipeline
`nv12`	uint8	Inside model	Zero-copy camera (ISP output)
`yuv444`	uint8	Inside model	YUV444 image source
`gray`	uint8	Inside model	Grayscale models

Supported Type Combination Matrix

Not all (input_type_train, input_type_rt) pairs are valid.
The toolchain will error if you pick an unsupported combination:

`input_type_train` ↓ \ `input_type_rt` →	`nv12`	`yuv444`	`rgb`	`bgr`	`gray`	`featuremap`
`yuv444`
`rgb`
`bgr`
`gray`
`featuremap`

This is why we use featuremap for both train and rt — it’s the only valid combination for featuremap.
rgb → featuremap is not supported and will fail at conversion time.

`featuremap` — Full Control in Your Code

The model is a pure compute graph with no preprocessing node inside.
You handle every step: resize, crop, color conversion, normalization.

# YAML
input_type_train: 'featuremap'
input_type_rt: 'featuremap'
input_layout_rt: 'NCHW'
norm_type: 'no_preprocess'
# Do NOT specify mean_value / scale_value / std_value with featuremap

# Runtime preprocessing (Python / C++)
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img = resize_and_crop(img)                        # → HWC uint8
chw = img.transpose(2,0,1).astype(np.float32) / 255.0
chw = (chw - MEAN) / STD                          # normalize yourself
feed = chw[np.newaxis, ...]                        # (1,3,224,224) float32

# Calibration data — identical to runtime input
arr = (arr / 255.0 - MEAN) / STD
arr.tofile("calib_000.fm.bin")                     # float32 NCHW raw binary
# Alternative: np.save("calib_000.npy", arr)       # .npy also accepted

`rgb` / `bgr` — Normalization Fused into the Model

The toolchain inserts an HzPreprocess node at the front of the compiled model.
This node performs (data − mean) × scale on the BPU at inference time.

# YAML
input_type_train: 'rgb'       # what the ONNX model was trained on
input_type_rt: 'rgb'          # or 'bgr' — what you feed at runtime
input_layout_rt: 'NHWC'       # BPU image preprocessing requires NHWC
norm_type: 'data_mean_and_scale'
mean_value: '123.675 116.28 103.53'      # mean × 255 (0-255 scale)
scale_value: '0.017125 0.017507 0.017429'   # 1 / (std × 255)
# Alternative: std_value: '58.395 57.12 57.375'  (std × 255; scale_value = 1/std_value)

Layout note: When mean_value/scale_value are specified, the toolchain inserts an
HzPreprocess node which requires NHWC input. Even if your ONNX is NCHW,
the compiled .bin model will accept NHWC. This is handled automatically by the toolchain.

# Runtime preprocessing — resize/crop only, NO normalization, NO /255
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)          # → RGB HWC uint8
img = resize_and_crop(img)                           # → (224,224,3) uint8

# hbm_runtime on RDK X5 requires int8 input (pixel − 128):
feed = (img.astype(np.int16) - 128).astype(np.int8) # (224,224,3) int8
feed = feed[np.newaxis, ...]                         # (1,224,224,3) NHWC int8

# Calibration data — raw 0-255 float BEFORE normalization
arr = resize_and_crop(img).transpose(2,0,1).astype(np.float32)  # (3,224,224) 0-255
arr.tofile("calib_000.rgb.bin")

int8 on RDK X5: The official docs note that for RGB/BGR models the compiled
input dtype is int8, meaning pixel values are offset by −128. When using
hbm_runtime, you must pass int8 (format 'b'), not uint8 ('B').
Passing the wrong dtype raises: expected numpy dtype format 'b', but received 'B'.

`nv12` — Zero-Copy Camera Path

NV12 is the native ISP output of the RDK X5 camera.
Using nv12 avoids the BGR→RGB conversion and a memory copy step.

# YAML
input_type_train: 'rgb'        # your model was trained on RGB
input_type_rt: 'nv12'          # board receives NV12 from camera
# No input_layout_rt needed — NV12 models always use NHWC internally
norm_type: 'data_mean_and_scale'
mean_value: '128.0 128.0 128.0'
scale_value: '0.0078125 0.0078125 0.0078125'   # 1/128

Layout note (official docs): When input_type_rt is nv12, the quantized
model’s input layout is always NHWC internally — do not set input_layout_rt: NCHW
for nv12. The toolchain handles the layout conversion automatically.

# Runtime preprocessing — feed raw NV12 frame directly
# NV12 memory layout: Y plane (H×W) followed by interleaved UV plane (H/2 × W)
nv12_frame = camera.get_frame()     # shape (H*3//2, W), dtype uint8
feed = nv12_frame[np.newaxis, ...]  # (1, H*3//2, W) uint8

Use NV12 when: your application reads from the onboard camera or a V4L2
video source and needs maximum throughput. Not recommended for offline demos.

Summary: What Changes Per Input Type

	`featuremap`	`rgb` / `bgr`	`nv12`	`gray`
`norm_type` in YAML	`no_preprocess`	`data_mean_and_scale`	`data_mean_and_scale`	`data_mean_and_scale`
`input_layout_rt`	NCHW	NHWC (auto-forced)	NHWC (auto-forced)	NHWC
cal data values	normalized float	0–255 float	raw YUV uint8	0–255 float
cal data layout	NCHW	NCHW	H×3/2 × W	NCHW
runtime dtype	float32	int8 (pixel−128)	uint8	uint8
color convert in code?	yes (BGR→RGB)	yes	no	N/A
mean/scale in YAML?	never	required	required	required

Recommendation for custom models: Start with featuremap. It has zero ambiguity — what you preprocess is exactly what BPU receives. Switch to nv12 only when integrating with the camera pipeline for maximum throughput.

6. Step 4 — Run the Quantizer

docker exec -w /workspace/resnet18/ptq_demo/config oe_x5_ptq \
  hb_mapper makertbin --model-type onnx --config resnet18_config.yaml

This takes ~10 seconds for ResNet18. Larger models can take minutes.

Reading the Output

Node                     ON   Type                    Cosine Similarity
/conv1/Conv              BPU  HzSQuantizedConv        0.999956
/layer1/layer1.0/conv1   BPU  HzSQuantizedConv        0.999679
...
/fc/Gemm_reshape         CPU  Reshape                 0.999134
─────────────────────────────────────────────────────────────────────
output   Cosine Similarity: 0.9991

How to read this:

Metric	Meaning	Target
`ON: BPU`	Layer runs on BPU hardware	All conv/linear layers
`ON: CPU`	Layer runs on ARM CPU	Acceptable for reshape/softmax
Cosine Similarity	Per-layer accuracy vs FP32	> 0.99 per layer
Output Cosine Similarity	End-to-end accuracy	> 0.999 = excellent

Outputs in `model_output/`

hb_mapper makertbin runs four internal stages and saves an intermediate file after each one.
Understanding what each file is lets you debug at exactly the right point.

model_output/
├── resnet18_224x224_featuremap_original_float_model.onnx    ← Stage 1
├── resnet18_224x224_featuremap_optimized_float_model.onnx   ← Stage 2
├── resnet18_224x224_featuremap_calibrated_model.onnx        ← Stage 3
├── resnet18_224x224_featuremap_quantized_model.onnx         ← Stage 4
└── resnet18_224x224_featuremap.bin                          ← Stage 5 ★ deploy this

Stage-by-Stage Explanation

Your ONNX
    │
    ▼ Stage 1 ─ Parse & insert HzPreprocess
original_float_model.onnx
    │   • Input/output nodes renamed to match YAML config
    │   • HzPreprocess node inserted at the front
    │     (handles type conversion & mean/scale — for rgb/nv12 only)
    │   • For featuremap: no HzPreprocess, model is unchanged
    │   • Still FP32; useful to verify graph structure is correct
    │
    ▼ Stage 2 ─ Graph optimization
optimized_float_model.onnx
    │   • Operator fusion (Conv+BN+ReLU → single fused op)
    │   • Constant folding, dead node removal
    │   • Still FP32; this is what goes into calibration
    │   • Use this to confirm expected operators and shapes
    │
    ▼ Stage 3 ─ Calibration
calibrated_model.onnx
    │   • Forward pass run on your calibration data
    │   • Each tensor's statistical range (min/max) recorded
    │   • Optimal INT8 scale factor computed per layer
    │   • Still FP32 weights, but scale metadata attached
    │   • Cosine similarity between this and optimized = calibration quality
    │   • Debug here if a specific layer has poor scale selection
    │
    ▼ Stage 4 ─ Quantization
quantized_model.onnx
    │   • Weights converted FP32 → INT8
    │   • Activations expressed as INT8 + scale factor
    │   • Quantize/Dequantize (QDQ) nodes wrap each op
    │   • Runnable on x86 via HB_ONNXRuntime (simulates BPU behavior)
    │   • ★ Use this for accuracy validation before going to the board
    │   • Cosine similarity output here = what you'll get on the board
    │
    ▼ Stage 5 ─ BPU Compilation
resnet18_224x224_featuremap.bin
        • INT8 weights packed into BPU instruction format
        • Operator scheduling, memory layout optimized for BPU SRAM
        • Not human-readable; only runs on physical BPU or simulator
        • ★ This is the only file you need to copy to the board

Which File to Use for What

Goal	File to use	Tool
Verify graph structure after conversion	`original_float_model.onnx`	Netron, `hb_mapper checker`
Inspect fused operators	`optimized_float_model.onnx`	Netron
Debug calibration scale quality	`calibrated_model.onnx`	`hb_mapper` accuracy tools
Validate accuracy on x86 (no board needed)	`quantized_model.onnx`	`HB_ONNXRuntime`
Deploy and run on RDK X5 board	`.bin`	`hbm_runtime` / `hb_dnn`
Benchmark theoretical BPU performance	`.bin`	`hb_perf`

7. Step 5 — Board Inference

Let’s break down the Python API. The RDK platforms use the hbm_runtime package, which is a thin Python binding built on C++ (libdnn) using pybind11. It allows you to load .bin models and perform fast inference directly on the BPU.

1. Dynamic Model Metadata Query

When loading the compiled .bin model, it is a bad practice to hardcode tensor names. If you re-convert the ONNX and output nodes rename themselves slightly, your script will crash. Instead, load metadata dynamically:

from hbm_runtime import HB_HBMRuntime

# Load the compiled binary (HBM/bin format)
model = HB_HBMRuntime("resnet18_224x224_featuremap.bin")

# Extract properties dynamically
model_name  = model.model_names[0]
input_name  = model.input_names[model_name][0]
output_name = model.output_names[model_name][0]

# Check input and output specifications
input_shape = model.input_shapes[model_name][input_name]
input_dtype = model.input_dtypes[model_name][input_name]

2. Preprocessing

For featuremap models, your board-side python code must perform the identical preprocessing steps as your calibration script, including OpenCV-level floating-point division and normalization:

def preprocess(img_path: str) -> np.ndarray:
    # 1. Read and swap channels BGR -> RGB
    img = cv2.imread(img_path, cv2.IMREAD_COLOR)
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

    # 2. Resize maintaining aspect ratio (Short Side to 256)
    h, w = img.shape[:2]
    nh, nw = (256, int(round(w * 256.0 / h))) if h <= w else (int(round(h * 256.0 / w)), 256)
    img = cv2.resize(img, (nw, nh), interpolation=cv2.INTER_LINEAR)

    # 3. Center Crop to 224x224
    top, left = (nh - 224) // 2, (nw - 224) // 2
    img = img[top:top+224, left:left+224]

    # 4. HWC (uint8) -> CHW (float32) [0.0, 1.0]
    chw = img.transpose(2, 0, 1).astype(np.float32) / 255.0

    # 5. Fused Normalization: (x - mean) / std
    chw = (chw - MEAN) / STD

    # 6. Add batch dimension -> float32 NCHW (1, 3, 224, 224)
    return np.ascontiguousarray(chw[np.newaxis, ...])

3. Execution & Output Processing

To execute inference, call model.run(). It receives a dictionary mapping string input node names to numpy arrays, and returns a nested output dictionary structured as {model_name: {output_name: ndarray}}.

feed = preprocess("coffee.jpg")

# BPU execution blocks the CPU thread until complete
outputs = model.run({input_name: feed})

# Unpack the nested dict
logits = outputs[model_name][output_name].reshape(-1)

Run:

python3 infer_resnet18.py resnet18_224x224_featuremap.bin coffee.jpg

Expected output:

model  : resnet18_224x224_featuremap
input  : input  shape=(1, 3, 224, 224)  dtype=float32
feed   : shape=(1, 3, 224, 224)  dtype=float32  min=-2.118  max=2.640

latency: 2.3 ms
Top-5:
  #1  [ 967] espresso                       71.2%  ############################
  #2  [ 504] coffee mug                      8.1%  ###
  #3  [ 441] beer glass                      4.3%  #
  #4  [ 550] espresso maker                  3.1%  #
  #5  [ 968] cup                             2.4%

C++ Inference (hb_dnn) — Compile on Board

While Python is excellent for development, C++(hb_dnn) is recommended for latency-critical deployment and industrial robot applications. Let’s do an in-depth walkthrough of the native libdnn C++ APIs used in infer_resnet18.cpp.

1. Low-level Model Loading

#include <dnn/hb_dnn.h>

hbPackedDNNHandle_t packed_handle;
const char *model_files[] = {"resnet18_224x224_featuremap.bin"};

// Initialize the packed handle from .bin file (binary packages models + BPU code)
hbDNNInitializeFromFiles(&packed_handle, model_files, 1);

// Query model count and get model names packed inside the .bin
const char **model_name_list;
int model_count = 0;
hbDNNGetModelNameList(&model_name_list, &model_count, packed_handle);

// Get the handle of the specific model by its name string
hbDNNHandle_t model_handle;
hbDNNGetModelHandle(&model_handle, packed_handle, model_name_list[0]);

2. Physical Memory Allocation (Contiguous DDR)

The BPU chip cannot directly read variables allocated on standard CPU Heap or Stack (which are virtually managed and can be fragmented across physical RAM). The BPU requires physically contiguous memory buffers allocated via D-Robotics kernel allocator:

#include <dnn/hb_sys.h>

hbDNNTensor input_tensor;
// Copy tensor structural properties (shapes, valid size, layout, stride)
input_tensor.properties = input_props; 

// Allocate physically contiguous buffer for input tensor
hbSysAllocCachedMem(&input_tensor.sysMem[0], input_bytes);

// ... Prepare float32 data in std::vector<float> blob ...

// Copy preprocessed data to BPU input buffer virtual address
std::memcpy(input_tensor.sysMem[0].virAddr, blob.data(), input_bytes);

3. CPU/BPU Cache Synchronization

Because we allocated Cached memory (hbSysAllocCachedMem) for maximum CPU copy performance, the CPU keeps written data in its L1/L2 cache lines. The BPU, however, directly reads from physical DDR memory. To prevent the BPU from reading stale or incomplete memory, we must perform a flush to write back the CPU caches:

// Flush CPU cached data to physical DDR before BPU execution
hbSysFlushMem(&input_tensor.sysMem[0], HB_SYS_MEM_CACHE_CLEAN);

Similarly, after BPU writes the output results to DDR, the CPU must invalidate its old cached entries to force itself to read the fresh inference results directly from physical RAM:

// Invalidate CPU cache lines so CPU reads the fresh BPU outputs from physical DDR
hbSysFlushMem(&output_tensor.sysMem[0], HB_SYS_MEM_CACHE_INVALIDATE);
const float *logits = reinterpret_cast<float *>(output_tensor.sysMem[0].virAddr);

4. Asynchronous Execution (Overlapping Compute)

The core inference function hbDNNInfer is completely asynchronous and non-blocking, allowing you to parallelize BPU model compute with CPU image preprocessing:

hbDNNTaskHandle_t task_handle = nullptr;
hbDNNInferCtrlParam ctrl;
// Setup scheduling options (bind to any BPU core, default priority)
ctrl.bpuCoreId = HB_BPU_CORE_ANY;
ctrl.dspCoreId = HB_DSP_CORE_ANY;
ctrl.priority  = 0;

// Submit inference request to BPU hardware queue (non-blocking)
hbDNNInfer(&task_handle, &p_output, p_input, model_handle, &ctrl);

// Block CPU thread and wait for BPU compute task to finish
hbDNNWaitTaskDone(task_handle, 0); // 0 means wait indefinitely

Compile and Run:

# On the board
cd ~/resnet
mkdir -p build && cd build
cmake ..
make -j4
cd ~/resnet
./build/infer_resnet18 resnet18_224x224_featuremap.bin coffee.jpg

Python vs C++:
Python (hbm_runtime) is faster to prototype — use it first.
C++ (hb_dnn) gives lower latency and is better for production pipelines.
BPU compute time is identical; the difference is pre/post-processing overhead.

8. Accuracy Verification (Optional but Recommended)

Before deploying, verify that the quantized model matches the float model on x86:

# Inside Docker — simulate BPU behavior on x86
docker exec -w /workspace/resnet18/ptq_demo oe_x5_ptq \
  python3 scripts/classify_image.py \
  model_output/resnet18_224x224_featuremap_quantized_model.onnx \
  test_images/coffee.jpg

This uses HB_ONNXRuntime — a drop-in replacement for standard ONNX Runtime that simulates INT8 quantization behavior on the host CPU. If the x86 result matches the board result, your pipeline is correct.

9. Generalizing to Your Own Model

Here’s a checklist for bringing any PyTorch model to RDK X5:

Checklist

□ 1. Export to ONNX opset 11, fixed input shape
□ 2. Run hb_mapper checker — resolve any unsupported ops
□ 3. Collect 50–200 representative calibration images
□ 4. Write preprocessing script — same pipeline as training val set
□ 5. Write resnet18_config.yaml with correct:
      - march (bayes-e for X5)
      - input_type_rt (start with featuremap)
      - input_shape
      - norm_type (no_preprocess for featuremap)
□ 6. Run hb_mapper makertbin
□ 7. Check output cosine similarity > 0.99
□ 8. Test on x86 with HB_ONNXRuntime before going to board
□ 9. Deploy .bin to board and verify

If Accuracy Is Poor

Symptom	Likely Cause	Fix
Cosine similarity < 0.95	Bad calibration data	Use more diverse images, or `calibration_type: mix`
One layer has low similarity	Outlier activations	Try `per_channel: True` or `calibration_type: kl`
CPU fallback for many ops	Unsupported operator	Fuse/replace ops before export; see OE op support list
Board result ≠ x86 result	Preprocessing mismatch	Ensure cal data ≡ runtime preprocessing

Common Architecture Notes

Architecture	Notes
ResNet / VGG / EfficientNet	Works out of the box, 100% BPU
MobileNet / ShuffleNet	Works; depthwise conv is BPU-supported
Transformer / ViT	Attention ops may fall to CPU; check with checker
YOLO (v5/v8/v10)	Remove post-processing (NMS) from ONNX export
Custom ops	Register via `custom_op` in config; CPU fallback

10. Performance Tuning

Lock BPU/CPU Frequency

# On the board — prevents thermal throttling during benchmarking
sudo bash -c "echo performance > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor"

11. Agent-Assisted Quantization

If you want to apply this workflow to your own ONNX model, you can also install an agent skill that specializes in the RDK X5 OpenExplorer PTQ toolchain:

RDK X5 Toolchain Quantization

This skill helps an AI agent guide or generate the toolchain-level steps for RDK X5 quantization, including:

checking ONNX operator compatibility with hb_mapper checker
preparing representative calibration data
writing and debugging hb_mapper makertbin YAML configs
choosing input_type_rt, input_type_train, layout, normalization, and calibration settings
compiling ONNX models into RDK X5 deployable .bin / .hbm artifacts
comparing float vs quantized outputs with cosine similarity tools
profiling performance with hb_perf and hrt_model_exec perf
troubleshooting accuracy drops, preprocessing mismatches, CPU fallback, or poor BPU utilization

Install it in an OpenClaw-compatible agent environment:

openclaw skills install @shockley6668/rdk-x5-toolchain-quantization

After installation, you can ask your agent questions such as:

Convert my ONNX model to an RDK X5 .bin with OpenExplorer v1.2.8.
Check this hb_mapper config and explain why cosine similarity dropped.
Prepare calibration data for an nv12 deployment path.
Profile this compiled model and help improve BPU utilization.

The skill does not replace understanding the toolchain, but it can make the first custom-model bring-up much faster by turning the checklist in this workshop into an interactive debugging assistant.