6.1. Algorithm Migration Overview
contents
Video/image analysis tasks based on deep learning usually include the following steps:
Video/picture decoding
Input preprocessing
Model reasoning
Output post-processing
Video/picture coding
In actual tasks, the algorithm often contains several different neural network models, so steps 2-4 will be repeated as required.
Hardware acceleration support
Practice has proved that the acceleration of neural network operation can no longer meet the needs of real scenes. In order to improve the running efficiency of the algorithm, BM1684/BM1684X integrates several hardware acceleration modules for codec, image processing and other operations in addition to the hardware acceleration unit TPU for tensor computation. Users can accelerate the above steps by using the corresponding software interface library provided in SophonSDK. Thus, it is convenient to develop efficient algorithms and applications.
In order to meet customers’ preference for different styles of interfaces, we have also packaged the hardware-accelerated interface library for several times. Users can select appropriate interface libraries for development by themselves. The specific situation is summarized as follows:
Task flow |
Whether hardware acceleration is supported |
SAIL Advanced interface library |
OPENCV Interface library |
FFMPEG Interface library |
Native Interface library |
Video/picture decoding |
Support |
sail::Decoder |
Y |
Y |
BMCV(picture) |
Input preprocessing |
Support |
sail::Bmcv |
Y |
N |
BMCV |
Model reasoning |
Support |
sail::Engine |
N |
N |
BMruntime |
Output post-processing |
Partial support |
sail::Bmcv |
N |
N |
BMCV |
Video/picture coding |
Support |
sail::Bmcv |
Y |
Y |
BMCV(picture) |
It is worth mentioning that in order to improve the efficiency of the algorithm and the requirements of hardware characteristics, users should pay attention to the following aspects when calling the hardware acceleration interface, which will be elaborated by examples in the subsequent documents:
Memory zero copy
Example Apply for physical continuous memory
Merge multiple preprocessing steps
Use 4batch to deduce
C/C++/Python programming interface
Currently support C/C++/Python three programming interfaces:
The BMRuntime module supports C/C++ interface programming, and the BMCV and BMLib support C interface programming. The Python/C++ programming interface is implemented based on the SAIL library.
SAIL (Sophon Artificial Intelligent Library) is an advanced encapsulation of BMRuntime, BMCV, BMLib and BMDecoder libraries in SophonSDK. SophonSDK original “load BModel and drive TPU reasoning”, “drive TPU+VPP to do image processing”, “drive VPU to do image and video decoding” and other functions abstract into a more simple C++ interface to provide external; It is repackaged using pybind11 to provide a concise and easy-to-use Python interface. Currently, all classes, enumerations, and functions in SAIL module are in the “sail” namespace. For details about SAIL C++/Python interface, please read SAIL User Development Manual.