6.5. Graphics Operation Acceleration Module

BMCV provides a set of machine vision library based on Computational AI chip optimization. At present, it can complete color space conversion, scale transformation, affine transformation, projection transformation, linear transformation, frame, JPEG encoding, BASE64 encoding, NMS, sorting, feature matching and other operations. For details about the BMCV module, please read the BMCV Development Reference Manual.

See the SAIL User Development Manual for an implementation of the Python interface.

The BMCV API is built around bm_image. A bm_image structure corresponds to an image.

6.5.1. C Language Programming Interface

bm_image structure

1struct bm_image {
2    int width;
3    int height;
4    bm_image_format_ext image_format;
5    bm_data_format_ext data_type;
6    bm_image_private* image_private;
7};

The bm_image structure members include the width and height of the image, the image format, the image data format, and the private data of the structure.

image_format Enumeration type

 1typedef enum bm_image_format_ext_{
 2    FORMAT_YUV420P,
 3    FORMAT_NV12,
 4    FORMAT_NV21,
 5    FORMAT_NV16,
 6    FORMAT_NV61,
 7    FORMAT_RGB_PLANAR,
 8    FORMAT_BGR_PLANAR,
 9    FORMAT_RGB_PACKED,
10    FORMAT_BGR_PACKED,
11    FORMAT_GRAY,
12    FORMAT_COMPRESSED
13}bm_image_format_ext;

Format

Description

FORMAT_YUV420P

Indicates that a YUV420 format image is pre-created, with three planes

FORMAT_NV12

An image in NV12 format is pre-created and has two planes

FORMAT_NV21

An image in NV21 format is pre-created and has two planes

FORMAT_RGB_PLANAR

Indicates a pre-created image in RGB format. RGB is arranged separately and has a plane

FORMAT_BGR_PLANAR

Indicates that an image in BGR format is pre-created. BGR is arranged separately and has a plane

FORMAT_RGB_PACKED

Indicates a pre-created image in RGB format. RGB is interleaved and has a plane

FORMAT_BGR_PACKED

Indicates that an image in BGR format is pre-created. BGR is interleaved and has a plane

FORMAT_GRAY

Represents a pre-created picture in grayscale format, with a plane

FORMAT_COMPRESSED

Indicates that an image in the compressed format of the VPU is pre-created. There are four planes

Enumeration of data store formats

1typedef enum bm_image_data_format_ext_{
2    DATA_TYPE_EXT_FLOAT32,
3    DATA_TYPE_EXT_1N_BYTE,
4    DATA_TYPE_EXT_4N_BYTE,
5    DATA_TYPE_EXT_1N_BYTE_SIGNED,
6    DATA_TYPE_EXT_4N_BYTE_SIGNED,
7}bm_image_data_format_ext;

Data format

Description

DATA_TYPE_EXT_FLOAT32

Indicates that the created image data format is a single-precision floating-point number

DATA_TYPE_EXT_1N_BYTE

Indicates that the data format of the created image is common signed 1N INT8

DATA_TYPE_EXT_4N_BYTE

Indicates that the data format of the created image is 4N INT8, that is, four INT8 images with symbols are interleaved

DATA_TYPE_EXT_1N_BYTE_SIGNED

It UINT8 indicates the format of the created picture data is ordinary unsigned 1N UINT8

DATA_TYPE_EXT_4N_BYTE

It indicates the data format of the created picture is 4N UINT8 which is four unsigned INT8 picture data interlaced

For bm_image initialization, we do not recommend that the user fill the bm_image structure directly, but create/destroy a bm_image structure through the following API

  • bm_image_create_batch

Create multiple bm images with contiguous physical memory.

 1 /*
 2 * @param [in]     handle       handle of low level device
 3 * @param [in]     img_h        image height
 4 * @param [in]     img_w        image width
 5 * @param [in]     img_format   format of image: BGR or YUV
 6 * @param [in]     data_type    data type of image: INT8 or FP32
 7 * @param [out]    image        pointer of bm image object
 8 * @param [in]     batch_num    batch size
 9 */
10 static inline bool bm_image_create_batch (bm_handle_t              handle,
11                                           int                      img_h,
12                                           int                      img_w,
13                                           bm_image_format_ext      img_format,
14                                           bm_image_data_format_ext data_type,
15                                           bm_image                 *image,
16                                           int                      batch_num)

  • bm_image_destroy_batch

Releases multiple bm images in a row in physical memory. To be used in pairs with the bm_image_create_batch interface.

1/*
2* @param [in]     image        pointer of bm image object
3* @param [in]     batch_num    batch size
4*/
5static inline bool bm_image_destroy_batch (bm_image *image, int batch_num)

  • bm_image_alloc_contiguous_mem

Allocate contiguous memory for multiple images

1bm_status_t bm_image_alloc_contiguous_mem(
2                int image_num,
3                bm_image *images,
4                int bmcv_image_usage
5);

Parameter

Description

int image_num

Number of images to be allocated memory

bm_image *images

Pointer to the image to which memory is to be allocated

int bmcv_image_usage

Parameters have been set for the customer by default (if the customer has requirements for the allocated memory location, this parameter can be specified).

  • bm_image_free_contiguous_mem

Frees the memory that is applied by bm_image_alloc_contiguous_mem

1bm_status_t bm_image_free_contiguous_mem(
2                int image_num,
3                bm_image *images
4        );

Parameter

Description

int image_num

Number of images to be allocated memory

bm_image *images

Pointer to the image to which memory is to be allocated

  • bmcv_image_vpp_convert

The bm1684 has special video post-processing hardware. If certain conditions are met, the csc + crop + resize function can be realized at one time, which is faster than the TPU.

1bmcv_image_vpp_convert(
2bm_handle_t           handle,
3    int                   output_num,
4    bm_image              input,
5    bm_image *            output,
6    bmcv_rect_t *         crop_rect,
7    bmcv_resize_algorithm algorithm = BMCV_INTER_LINEAR);

The API converts the input image format into the output image format and supports crop + resize function, which supports crop output from one input and resize to the output image size.

Parameter

Description

bm_handle_t handle

Device environment handle, obtained by calling bm_dev_request

int output_num

The output bm_image number is the same as the crop number of src image. A src crop outputs a dst bm_image

bm_image input

Enter the bm_image object

bm_image* output

Outputs a pointer to the bm_image object

bmcv_rect_t * crop_rect

The crop parameter corresponding to each output bm_image object on the input image includes the x coordinate of the start point, y coordinate of the start point, the width of the crop image and the height of the crop image. For details, please refer to the BMCV user development manual

bmcv_resize_algorithm algorithm = BMCV_INTER_LINEAR)

The value can be BMCV_INTER_NEAREST or BMCV_INTER_LINEAR. The default value is bilinear difference

  • bmcv_convert_to

To realize the linear change of image pixels, the specific data relationship can be expressed by the formula

\[y=\alpha*x+\beta\]
1bm_status_t bmcv_convert_to (bm_handle_t handle,int input_num,
2                        bmcv_convert_to_attr convert_to_attr,
3                        bm_image* input, bm_image* output)

Parameter

Description

bm_handle_thandle

Input bm_handle handle

int input_num

Enter the number of pictures. A maximum of 4 is supported

bmcv_convert_to_attr convert_to_attr

The corresponding configuration parameters for each picture

bm_image* input

Enter bm_image. Each bm_image outside needs to be created by calling bmcv_image_create. image memory can be used to create new memory using bmcv_image_dev_mem_alloc or bmcv_image_copy_to_device, or to attach existing memory using bmcv_image_attach.

bm_image*output

Print bm_image. Each bm_image outside needs to be created by calling bmcv_image_create. image memory can be opened up with bmcv_image_dev_mem_alloc, or existing memory can be attached with bmcv_image_attach. If it is not actively allocated, it will be self-allocated within the api

Structure bmcv_convert_to_attr_s

1typedef struct bmcv_convert_to_attr_s {
2    float alpha_0;
3    float beta_0;
4    float alpha_1;
5    float beta_1;
6    float alpha_2;
7    float beta_2;
8} bmcv_convert_to_attr;

Parameter

Description

alpha_0

The coefficient of linear transformation on the 0th channel is described

beta_0

Describes the offset of the 0 th channel for linear transformation

alpha_1

The coefficient of linear transformation for the first channel is described

beta_1

Describes the shift of the first channel for linear transformation

alpha_2

The linear transformation coefficient of the second channel is described

beta_2

The offset of the second channel for linear transformation is described

The structure describes alpha and beta in the tee. The actual parameter configuration is based on how many channels the inference input data is.

6.5.2. Python Language Programming Interface

This section covers only the interface functions used in the use case YOLOv5.

See the SAIL User Development Manual for more interface definitions.

  • init

1def __init__(handle):
2""" Constructor.
3Parameters
4---------
5handle : sail.Handle Handle instance
6"""

  • convert_to

 1def convert_to(input, alpha_beta):
 2""" Applies a linear transformation to an image.
 3Parameters
 4---------
 5input : sail.BMImage Input image
 6alpha_beta: tuple (a0, b0), (a1, b1), (a2, b2) factors
 7Returns
 8---------
 9output : sail.BMImage Output image
10"""

  • convert_format

 1def convert_format(input, output):
 2"""Convert input to output format.
 3
 4Parameters
 5----------
 6input : sail.BMImage
 7    BMimage instance
 8output : sail.BMImage
 9    output image
10"""

  • vpp_crop_and_resize_padding

 1def vpp_crop_and_resize_padding(input, crop_x0, crop_y0, crop_w, crop_h, resize_w, resize_h, padding):
 2""" Crop then resize an image using vpp.
 3
 4Parameters
 5----------
 6input : sail.BMImage
 7    Input image
 8crop_x0 : int
 9    Start point x of the crop window
10crop_y0 : int
11    Start point y of the crop window
12crop_w : int
13    Width of the crop window
14crop_h : int
15    Height of the crop window
16resize_w : int
17    Target width
18resize_h : int
19    Target height
20padding : PaddingAtrr
21    padding info
22
23Returns
24----------
25output : sail.BMImage
26    Output image
27"""