- Preface
The ultimate goal of this project is to realize the access of multi-eye mipi camera on the RDK Module to achieve multi-eye vision, multi-eye depth and other functions. ** This project is also an extension of the visual part of the 2023 graduate Electronic design competition “Garbage sorting Robot based on Rising Sun x3” **. In this explanation why choose mipi camera access instead of usb camera, usb bus is controlled by usb host controller, bandwidth is limited, generally speaking, a usb host controller on a PC can access two usb 2.0 low resolution cameras, If the PC has a large number of usb host controllers, multiple cameras can be connected; otherwise, only PCIE acquisition cards can be used to collect cameras. The X3M chip has only one usb host controller, and after a simple test, in the case of simultaneous streaming, only one usb camera can stream, which does not meet the real-time requirements of multi-camera. mipi camera access has hardware acceleration inside the X3M chip, which in principle can meet the needs of multiple camera access and streaming at the same time, so it is necessary to achieve multiple mipi camera access. All content of this project is open source at: https://github.com/307321587/tros_code
Including two parts, ** multi-lens camera stream ** and **TROS communication **, this chapter first describes the ** multi-lens camera stream **. 2. RDK Module hardware analysis
RDK Module
The Rising Sun X3M chip has a total of four MIPI host inputs, namely MIPI host 0,1,2,3, and each host has 2 data Lane bandwidth, but according to the official statement, MIPI host0 and MIPI host2 can drive 2 data Lane bandwidth separately. MIPI host1 and MIPI host3 need to form 4 data lanes to use together.
Therefore, the Rising Sun X3M chip can access up to 3 MIPI cameras, and the RDK Module will lead out all the interfaces of the three cameras, which is easier for us to use. It should be noted that the CAM0 and CAM1 extracted are compatible with the Raspberry PI interface cameras, so theoretically speaking, the CAM0 and CAM1 extracted are compatible with the Raspberry PI interface cameras. There are a variety of Raspberry PI interface cameras on the market that can plug into the RDK Module.
We chose a non-Horizon recommended infrared Raspberry PI camera, and the current testing is normal, link below:
Three, binocular stream program
3.1 Program run
Run on the computer:
cd mipi_camera_board
bash build.sh
Run at the board end:
bash start.sh
You can see the print log as follows:
msg send completed!
try to get img...
normal pipe_id (1)type(11)frame_id(0)buf_index(2)w x h(1920x1080) data_type 11 img_format 0
mat success
get img cost time 2 ms
3.2 Program analysis
The file structure is as follows:
--- mipi_camera_board
--- include
-- encoder.h defines the jpeg encoding class encode
-- image_publish.h defines the tros publisher for publishing image messages
-- -- sensor.h Defines the camera class SensorProcess
-- sp_codec.h Horizon official coding header file
-- lib Required libraries, including camera driver libraries and isp libraries
--- sensor
-- sensor_imx219.cpp Defines the driver port that the sensor is connected to
--- src
-- -- encoder.cpp Implements the encoding class
-- main.cpp main program
-- sensor.cpp camera class implementation
3.2 mipi camera access analysis
** Horizon API Analysis **
To achieve multiple mipi camera access, you need to use the horizon’s underlying multimedia interface VIN interface, which requires at least the following files to achieve mipi camera access:
sensor_xxx.cpp
libxxx.so
libxxx_linear.so
-
sensor_xxx.cpp defines some required parameters for VIN api to start mipi cameras, such as I2C bus number, frame rate, and mipi index. Take imx219 as an example. Some key configurations are as follows:
MIPI_SENSOR_INFO_S SENSOR_2LANE_IMX219_30FPS_10BIT_LINEAR_INFO_2 = {
.deseEnable = 0,
.inputMode = INPUT_MODE_MIPI,
.sensorInfo = {
.port = 1, // sensor logical number. The value must start from 0
.dev_port = 1, // Driver nodes for each sensor operation. One driver supports multiple nodes. The dev_port in snsinfo must be equal to the pipeId, which needs special attention when setting up the multi-camera
.bus_type = 0,
bus_num = 0, // Bus number. The value is determined based on the hardware schematic diagram of the board. The default i2c5 is not configured
.fps = 30,
.resolution = 1080,
.sensor_addr = 0x10,
.entry_index = 2, // sensor mipi index used. The value ranges from 0 to 3, which corresponds to the serial number of the mipi host
.sensor_mode = NORMAL_M,
.reg_width = 16,
.sensor_name = “imx219”,
}
};
We use two identical imx219 cameras, so most of the configurations are the same. However, it should be noted that several parameters given in the remarks above, such as port, bus_num, entry_index, etc., are different due to the hardware interface. For example, CAM2 accesses I2C 0. mipi host2. In addition, the flow inside the image is carried out based on pipe pipe, so CAM1 occupies pipe0, i.e. dev_port=0, and CAM2 needs to use pipe1, i.e. dev_port=1
- libxxx.so defines functions related to register operations, such as on stream, off stream, and modifying exposure time.
- libxxx_linear.so defines isp calibration parameters.
All three files are available in the hobot-camera project on Horizon github.
However, it is worth noting that files cannot be directly copied when binocular streaming is performed. When CAM2 host2 is used to access the camera,.mclk needs to be changed from 2400 to 24. In addition, host2 is used to start the script when the camera is started.
.mclk = 24, // Master clock of the mipi module. The default clock is 24 MHZ
In addition, when the camera is started, host2 needs to add the stop command in the startup script, so that the sensor is configured to stop state before initialization
echo 1 > /sys/class/vps/mipi_host1/param/stop_check_instart
Otherwise, an error will be reported:
[ERROR]["LOG"][src/hb_vin_mipi_host.c:286] !!! host2 MIPIHOSTIOC_INIT error, ret = -1
[ERROR]["LOG"][src/hb_vin.c:124] mipi_host 2 init error!
[ERROR]["mipi"][mipi/hb_mipi_api.c:461] hb_vin_init fail
hb mipi set mipi attr error!
[ERROR]["venc"][video/src/hb_venc.c:302] [9857.657464]HB_VENC_CreateChn[302]: [HB_VENC] HB_VENC_CreateChn:302 Failed to FindCtxByChn VeChn = 0 s32Ret = -268958736
2023/08/20 17:12:11.622 ERROR [x3_venc_init][0173]HB_VENC_CreateChn 0 failed, eff803f0.
** Self-encapsulated library **
Through the above analysis, it is very good to package the horizon official API as follows:
class SensorProcess {
public:
SensorProcess(SensorSetting sensor);
~SensorProcess();
int sensorSifDevInit();
void print_sensor_info(MIPI_SENSOR_INFO_S *snsinfo);
cv::Mat getImage();
cv::Mat transRgbToYuv(cv::Mat rgb_img);
int getSensorWidth(){return width_; };
int getSensorHeight(){return height_; };
private:
SensorSetting sensor_setting_;
bool sensor_init_flag = false;
int width_=0;
int height_=0;
};
During class initialization, the data in sensor_xxx.cpp is packaged and passed into SensorSetting, and sensorSifDevInit is called for initialization. After initialization, the current image of the sensor can be obtained through getImage(). And is presented in opencv mat format, the final effect is as follows: