7.8. BM1684(X)_to_BM1688(CV186AH) Compatibility Documentation

This document is intended for users with some experience in using SOPHONSDK. It is still being developed and improved, and if user discover any new issues or have suggestions for improvement, please feel free to provide feedback to our staff.

7.8.1. Obtaining the BM1688/CV186AH SOPHONSDK

The SOPHONSDK for BM1688/CV186AH is not the same as that for BM1684/BM1684X, so it needs to be obtained separately from the official website. After downloading, use the tool to extract it, and user will get installation packages for various modules, sample programs, etc. The SDK directory structure is as follows:

Folder Name

Remarks

sophon-img

SoC mode installation package, etc.

sophon_media

Multimedia library supporting hardware acceleration for SOPHON devices

tpu-mlir

TPU compiler toolchain

tpu-perf

Model performance and accuracy verification toolkit

sophon-stream

High-performance inference framework based on pipeline

sophon-demo

Comprehensive routines for single models or scenarios

sophon-sail

Interface library for encapsulating underlying interfaces with C++/Python API

isp-tools

Parameter adjustment tools for various modules of ISP

doc

Documentation collection for various modules

7.8.2. Preparing the BM1688/CV186AH bmodel

To recompile the bmodel that can run on BM1688/CV186AH, user need to use tpu-mlir that supports BM1688/CV186AH.

Generally, there are several steps:

  1. Prepare an x86 ubuntu with corresponding specifications, depending on the size of the model to be compiled. It is recommended to have more than 32GB of RAM to compile most models.

  2. Run the following command to set up the tpu-mlir environment:

    # Pull the latest version of Docker, currently corresponding to tpuc_dev:v3.2. If this image is already present in the environment, re-execute the following command.
docker pull sophgo/tpuc_dev:latest

# Here, the local directory is mapped to the /workspace directory in the docker. Users need to map the directory of the demo to the docker according to the actual situation.
# myname is just an example name, please specify the name user want for the container.
docker run --privileged --name myname -v $PWD:/workspace -it sophgo/tpuc_dev:latest

# At this point, user have entered the docker and are in the /workspace directory.

# Download tpu_mlir via pip. If user can't download it, user can use Tsinghua source to accelerate.
pip install tpu_mlir

# TPU-MLIR requires different dependencies when processing models from different frameworks. For model files generated by onnx or torch, use a command like the following to install additional dependency environments:
pip install tpu_mlir[onnx]
pip install tpu_mlir[torch]

# Currently supports five configurations: onnx, torch, tensorflow, caffe, paddle. user can install multiple configurations with one command, or user can install all dependency environments directly:
pip install tpu_mlir[onnx,torch]
pip install tpu_mlir[all]

  3. Compile the model:

    As long as user change the value of the --processor parameter in the previous compilation command or script to bm1688/cv186x, and leave the other parts unchanged, user can get the bmodel that can run on BM1688/CV186AH after running.

  4. Compiling BM1688 dual-core model:

    BM1688 consists of two identical cores and supports the data partitioning of one model to run on two cores. Tpu-mlir model_deploy.py adds a --num_core parameter for BM1688. If user want one model to run on two cores, user can add a --num_core 2 parameter in model_deploy.py, so that the compiled model will be executed by two cores.

  5. Quantization considerations:

    The previous calibration_table and qtable can be reused. If user encounter errors such as unsupported operators or poor model accuracy during the mlir compilation of the bmodel, regenerate the calibration_table or qtable.

7.8.3. Preparing the compilation-dependent SDK

If user are programming in C++, user still need to obtain the libsophon_soc_${x.y.z}/_aarch64.tar.gz under the sophon-img directory and the sophon-media-soc_${x.y.z}/_aarch64.tar.gz under the sophon_media directory as the header files and library files for cross-compilation dependencies.

If user are programming in C++, user still need to obtain the libsophon_soc_${x.y.z}/_aarch64.tar.gz and sophon-media-soc_${x.y.z}/_aarch64.tar.gz release packages under the sophon-img and sophon_media directories, respectively, as the header files and library files for cross-compilation dependencies.

If user previously manually linked libraries such as bmion/bmjpulite/bmjpuapi/bmvpulite/bmvpuapi/bmvideo/bmvppapi, user can remove them directly, as these libraries do not actually expose interfaces to users and are intended for underlying calls.

One difference from BM1684/BM1684X is that the libraries provided in sophon_media currently depend on the libisp module. If user are using the GeminiSDK1.3 or later version for BM1688/CV186AH, user need to perform the following operations:

Obtain the sophon-soc-libisp_${x.y.z}_arm64.deb package from the sophon-img/bsp-debs/ directory in the SDK, and then run the following command:

dpkg -x sophon-soc-libisp_${x.y.z}_arm64.deb sophon-libisp
cp -rf sophon-libisp/opt/sophon/sophon-soc-libisp_${x.y.z}/lib ${soc-sdk} # Add the libisp library to the existing dependency library.

If user are programming in Python, user only need to recompile the sophon-sail.whl installation package. The compilation process is the same as before, and user can refer to the sophon-sail development guide. The BM1688 SoC comes with libsophon and sophon_media runtimes.

7.8.4. Compatibility Adaptation for SDK Versions ≥1.7

BM1688/CV186AH SOPHONSDK versions 1.7 and above have further optimized compatibility issues. It is only necessary to ensure that the code includes the following header file:

#include <bmcv_api.h>

7.8.5. Compatibility Adaptation for SDK Versions <1.7

The interfaces provided by BM1688/CV186AH SOPHONSDK versions below 1.7 have some changes compared to the BM1684/BM1684X SDK. The following sections will explain the changes by module and provide a compatibility solution for application layer code to be compatible with both SDKs:

7.8.5.1. Header File Changes

  1. bmcv_api.h has been deprecated. The compatibility solution is as follows:

    #if!(BMCV_VERSION_MAJOR > 1)
#include <bmcv_api.h>
#endif

7.8.5.2. bmcv Changes

  1. Function parameters become pointers:

    DECL_EXPORT bm_status_t bm_image_destroy(bm_image image)

    Becomes

    DECL_EXPORT bm_status_t bm_image_destroy(bm_image *image)

    Compatibility solution:

    1. Add the following code in the common header file:

    #if BMCV_VERSION_MAJOR > 1
    static inline bm_status_t bm_image_destroy(bm_image& image){
    return bm_image_destroy(&image);
    }
#endif

  2. Function name changes:

    bm_image_dev_mem_alloc

    Becomes

    bm_image_alloc_dev_mem

    Compatibility solution:

    1. Encapsulate an old interface with the same functionality in the common header file.

  3. Interface name correction:

    bm_image_dettach_contiguous_mem

    Corrected to

    bm_image_detach_contiguous_mem

    Compatibility solution:

    1. Encapsulate an old interface with the same functionality in the common header file.

  4. Structure name correction

    bmcv_padding_atrr_t

    Corrected to

    bmcv_padding_attr_t

    Compatibility solution:

    1. Add typedef bmcv_padding_attr_t bmcv_padding_atrr_t; in the common header file.

  5. Interface deprecation:

    bmcv_image_crop

    Compatibility solution:

    1. Use bmcv_image_vpp_convert instead.

    2. Or encapsulate a bmcv_image_crop interface using bmcv_image_vpp_convert in the common header file.

  6. Data type deprecation:

    DATA_TYPE_EXT_4N_BYTE_SIGNED
DATA_TYPE_EXT_4N_BYTE

    Compatibility solution:

    1. 4N data types were rarely used in previous SDKs, and it is not recommended to use 4N data types anymore. Switch to 1N data types.

  7. Interface functionality changes due to memory lausert changes:

    bm_image_alloc_dev_mem(heap_id = 2)
bm_image_alloc_dev_mem_heap_mask(heap_mask = 1 << 2)
bm_image_alloc_contiguous_mem_heap_mask(heap_mask = 1 << 2)

    On BM1688/CV186AH, there are only two heaps. Memory previously allocated in heap2 needs to be moved to heap1, otherwise the allocation will fail. The heap_id/heap_mask parameters for the above interfaces need to be changed to:

    bm_image_alloc_dev_mem(heap_id = 1)
bm_image_alloc_dev_mem_heap_mask(heap_mask = 1 << 1)
bm_image_alloc_contiguous_mem_heap_mask(heap_mask = 1 << 1)

  8. Interfaces not yet supported:

    bmcv_base64_enc()
bmcv_base64_dec()
bmcv_faiss_indexflatIP()
bmcv_nms()
bmcv_nms_ext()
bmcv_fft_1d_create_plan()
bmcv_fft_2d_create_plan()
bmcv_fft_execute()
bmcv_fft_execute_real_input()
bmcv_fft_destroy_plan()

7.8.5.3. Changes to ffmpeg

The ffmpeg version has been upgraded from 4.1 to 6.0, which has introduced some changes to ffmpeg itself, as follows:

  1. Interface deprecation:

    av_register_all

    Compatibility solution:

    1. Remove it.

    2. Or encapsulate an av_register_all.

    avcodec_decode_video2
avcodec_encode_video2

    Compatibility solution:

    1. According to the ffmpeg documentation, change to the form of avcodec_send_packet and avcodec_receive_frame.

    2. User can encapsulate avcodec_decode_video2 independently by refer to https://github.com/sophgo/sophon-demo/blob/release/include/ff_decode.hpp.

    3. User can encapsulate avcodec_encode_video2 independently by referring to sophon-sail src/internal.h.

  2. Struct or function type becomes const:

    struct AVOutputFormat *oformat;
AVInputFormat* av_find_input_format
AVCodec* avcodec_find_decoder
AVOutputFormat* av_guess_format
AVCodec* avcodec_find_decoder_by_name
AVCodec* avcodec_find_decoder_by_name

    Becomes

    const struct AVOutputFormat *oformat
const AVInputFormat* av_find_input_format
const AVCodec* avcodec_find_decoder
const AVOutputFormat* av_guess_format
const AVCodec* avcodec_find_decoder_by_name
const AVCodec* avcodec_find_decoder_by_name

    Compatibility solution:

    1. Use const_cast to remove the const modifier.

  3. Struct member variable changes:

    AVStream->codec

    Becomes

    AVStream->codecpar

    Compatibility solution:

    1. Use avcodec_parameters_to_context to retrieve the context again, and use avcodec_parameters_from_context to copy the context content back to the stream.

7.8.5.4. Changes to bmlib

  1. New interface:

    DECL_EXPORT bm_status_t bm_thread_sync_from_core(bm_handle_t handle, int core_id);

    Function description:

    1.Can specify to sync only a certain core, which is used when each core runs a model.

  2. Interface function changes due to memory lausert changes:

    bm_malloc_device_byte_heap(heap_id = 2)
bm_malloc_device_byte_heap_mask(heap_mask = 1 << 2)

    On BM1688/CV186AH, there are only two heaps, and the memory previously placed in heap2 needs to be moved to heap1, otherwise the allocation will fail. The heap_id/heap_mask parameters for the above interfaces need to be changed to:

    bm_malloc_device_byte_heap(heap_id = 1)
bm_malloc_device_byte_heap_mask(heap_mask = 1 << 1)

7.8.5.5. Changes to bmrt

  1. New interface:

    DECL_EXPORT bool bmrt_launch_tensor_multi_cores(void *p_bmrt,
                                                const char *net_name,
                                                const bm_tensor_t input_tensors[],
                                                int input_num,
                                                bm_tensor_t output_tensors[],
                                                int output_num,
                                                bool user_mem,
                                                bool user_stmode,
                                                const int *core_list,
                                                int core_num);

    Function description:

    This interface can specify to run a single-core model on different cores. For details, please refer to the bmruntime interface documentation or source code in the SOPHONSDK.

7.8.6. Preparing the BM1688/CV186AH operating environment

The flashing steps for BM1688/CV186AH are the same as those for BM1684/BM1684X. For details, please refer to the relevant product user manual. In the SDK downloaded in the Obtaining the BM1688/CV186AH SOPHONSDK section, the sophon-img/sdcard.tgz file is the flashing package.

After preparing the operating environment, user can copy the compiled program to the BM1688/CV186AH SoC for execution.

7.8.7. Other precautions

  1. For bmrt and bmcv related operations, it is best to isolate the memory allocated by each other. The memory required by bmrt should be placed in heap0 (i.e., npu heap), and the memory required by bmcv should be placed in heap1 (i.e., vpp heap).

  2. Due to the reduced memory of BM1688, the ddr is fully interleaved without bandwidth allocation issues, so the vpu heap is no longer reserved, and the multimedia modules share the vpp heap.

  3. The bm1688 sdk currently only divides the ddr into two heaps: npu and vpp. The viewing method is changed to:

    sudo cat /sys/kernel/debug/ion/cvi_vpp_heap_dump/summary
sudo cat /sys/kernel/debug/ion/cvi_npu_heap_dump/summary

  4. Memory lausert modification tool: https://doc.sophgo.com/sdk-docs/v23.09.01-lts/docs_latest_release/docs/SophonSDK_doc/zh/html/appendix/2_mem_edit_tools.html

    It is compatible with BM1688/CV186AH Gemini v1.5 and above SDKs, and the vpu heap needs to be set to 0.