6. Deployment of Yolov8 Model for General Use¶

6.1. Introduction¶

This document introduces the operation process of deploying the YOLOv8 architecture model on the CV181x development board. The main steps include:

Convert YOLOv8 model Pytorch version to ONNX model
Convert onnx model to cvi model format
Finally, write a calling interface to obtain the inference results

6.2. Convert pt Model to onnx¶

Firstly, obtain the official warehouse code for YOLOv8 [ultra tics/ultra tics: NEW - YOLOv8 🚀 In PyTorch>ONNX>OpenVINO>CoreML>TFLite (github. com)](https://github.com/ultralytics/ultralytics) .. code-block:: shell

git clone https://github.com/ultralytics/ultralytics.git

Download the corresponding yolov8 model file again to [yolov8n](https://github.com/ultralytics/assets/releases/download/v0.0.0/yolov8n.pt)For example, then place the downloaded yolov8n. pt in the ultra tics/weights/directory, as shown on the following command line

cd ultralytics & mkdir weights cd weights wget https://github.com/ultralytics/assets/releases/download/v0.0.0/yolov8n.pt

Adjust the output branch of YOLOv8, remove the decoding part of the forward function, and separate the boxes and cls of three different feature maps to obtain six branches

Specifically, you can create a new file in the Ultratics/directory and paste the following code

from ultralytics import YOLO
import types

input_size = (640, 640)

def forward2(self, x):
    x_reg = [self.cv2[i](x[i]) for i in range(self.nl)]
    x_cls = [self.cv3[i](x[i]) for i in range(self.nl)]
    return x_reg + x_cls

model_path = "./weights/yolov8s.pt"
model = YOLO(model_path)
model.model.model[-1].forward = types.MethodType(forward2, model.model.model[-1])
model.export(format='onnx', opset=11, imgsz=input_size)

After running the above code, you can run it in/ Obtain the yolov8n. onnx file in the weights/directory, and then convert the onnx model to the cvi model

6.3. Onnx Model Conversion cvi model¶

The CVIModel conversion operation can refer to the onnx model conversion CVIModel section in the YOLO-V5 porting section.

6.4. TDL SDK Interface Description¶

First, create a cvitdl_handle, and then open the corresponding cvi model. Before running the inference interface, you can set two thresholds for your own model

CVI_TDL_SetModelThreshold
CVI_TDL_SetModelNmsThreshold

The final inference result is analyzed through cvtdl_object_t. Information acquisition

// create handle
cvitdl_handle_t tdl_handle = NULL;
ret = CVI_TDL_CreateHandle(&tdl_handle);
  if (ret != CVI_SUCCESS) {
    printf("Create tdl handle failed with %#x!\n", ret);
    return ret;
  }

// read image
VIDEO_FRAME_INFO_S bg;
ret = CVI_TDL_ReadImage(strf1.c_str(), &bg, PIXEL_FORMAT_RGB_888_PLANAR);

// open model and set conf & nms threshold
ret = CVI_TDL_OpenModel(tdl_handle, CVI_TDL_SUPPORTED_MODEL_YOLOV8_DETECTION, path_to_model);
CVI_TDL_SetModelThreshold(tdl_handle, CVI_TDL_SUPPORTED_MODEL_YOLOV8_DETECTION, 0.5);
CVI_TDL_SetModelNmsThreshold(tdl_handle, CVI_TDL_SUPPORTED_MODEL_YOLOV8_DETECTION, 0.5);
if (ret != CVI_SUCCESS) {
  printf("open model failed with %#x!\n", ret);
    return ret;
}

// start infer
cvtdl_object_t obj_meta = {0};
CVI_TDL_YOLOV8_Detection(tdl_handle, &bg, &obj_meta);

// analysis result
std::stringstream ss;
ss << "boxes=[";
for (uint32_t i = 0; i < obj_meta.size; i++) {
ss << "[" << obj_meta.info[i].bbox.x1 << "," << obj_meta.info[i].bbox.y1 << ","
  << obj_meta.info[i].bbox.x2 << "," << obj_meta.info[i].bbox.y2 << ","
  << obj_meta.info[i].classes << "," << obj_meta.info[i].bbox.score << "],";
}

6.5. Test Result¶

The Yolov8n and Yolov8s models on the official website were converted and tested on the COCO2017 dataset, with the threshold set to:

Conf: 0.001
Nms_Threshold: 0.6

All resolutions are 640 x 640

The official export method of the YOLOV8n model onnx performance:

platform	Inference time (ms)	bandwidth (MB)	ION(MB)	MAP 0.5	MAP 0.5-0.95
pytorch	N/A	N/A	N/A	53	37.3
cv181x	54.91	44.16	8.64	Quantification failure	Quantification failure
cv182x	40.21	44.32	8.62	Quantification failure	Quantification failure
cv183x	17.81	40.46	8.3	Quantification failure	Quantification failure

TDL of yolov8n model_SDK export method onnx performance：

platform	Inference time (ms)	bandwidth (MB)	ION(MB)	MAP 0.5	MAP 0.5-0.95
onnx	N/A	N/A	N/A	51.32	36.4577
cv181x	45.62	31.56	7.54	51.2207	35.8048
cv182x	32.8	32.8	7.72	51.2207	35.8048
cv183x	12.61	28.64	7.53	51.2207	35.8048

The official export method of the yolov8s model onnx performance:

platform	Inference time (ms)	bandwidth (MB)	ION(MB)	MAP 0.5	MAP 0.5-0.95
pytorch	N/A	N/A	N/A	61.8	44.9
cv181x	144.72	101.75	17.99	Quantification failure	Quantification failure
cv182x	103	101.75	17.99	Quantification failure	Quantification failure
cv183x	38.04	38.04	16.99	Quantification failure	Quantification failure

TDL of yolov8s model SDK export method onnx performance：

platform	Inference time (ms)	bandwidth (MB)	ION(MB)	MAP 0.5	MAP 0.5-0.95
onnx	N/A	N/A	N/A	60.1534	44.034
cv181x	135.55	89.53	18.26	60.2784	43.4908
cv182x	95.95	89.53	18.26	60.2784	43.4908
cv183x	32.88	58.44	16.9	60.2784	43.4908