5.10. sail.Engine

5.10.1. __init__

Constructor without or with bmodel loaded.

Interface:

def __init__(tpu_id: int)

Parameters

• tpu_id : int

TPU ID. You can use bm-smi to see available IDs

Interface:

def __init__(self, handle: Handle)

Parameters

• hanle : Handle

A Handle instance

Interface:

def __init__(self,
            bmodel_path: str,
            tpu_id: int,
            mode: sail.IOMode)

Parameters

• bmodel_path : str

Path to bmodel

• tpu_id : int

TPU ID. You can use bm-smi to see available IDs

• mode : sail.IOMode

Specify the input/output tensors are in system memory or device memory

Interface:

def __init__(self,
            bmodel_bytes:str,
            bmodel_size: int,
            tpu_id: int,
            mode: sail.IOMode)

Parameters

• bmodel_bytes : bytes

Bytes of bmodel in system memory

• bmodel_size : int

Bmodel byte size

• tpu_id : int

TPU ID. You can use bm-smi to see available IDs

• mode : sail.IOMode

Specify the input/output tensors are in system memory or device memory

5.10.2. get_handle

Get Handle instance.

Interface:

def get_handle(self)->sail.Handle

Returns

• handle: sail.Handle

Handle instance

5.10.3. load

Load bmodel from file.

Interface:

def load(self, bmodel_path: str)

Parameters

• bmodel_path : str

Path to bmodel

Interface:

def load(self, bmodel_bytes: bytes, bmodel_size: int)

Parameters

• bmodel_bytes : bytes

Bytes of bmodel in system memory

• bmodel_size : int

Bmodel byte size

5.10.4. get_graph_names

Get all graph names in the loaded bmodels.

Interface:

def get_graph_names(self)-> list

Returns

• graph_names : list

Graph names list in loaded context

5.10.5. set_io_mode

Set IOMode for a graph.

Interface:

def set_io_mode(self, graph_name: str, mode: sail.IOMode)

Parameters

• graph_name: str

The specified graph name

• mode : sail.IOMode

Specified io mode

5.10.6. get_input_names

Get all input tensor names of the specified graph.

Interface:

def get_input_names(self, graph_name: str) -> list

Parameters

• graph_name : str

Specified graph name

Returns

• input_names : list

All the input tensor names of the graph

5.10.7. get_output_names

Get all output tensor names of the specified graph.

Interface:

def get_output_names(self, graph_name: str)-> list

Parameters

• graph_name : str

Specified graph name

Returns

• input_names : list

All the output tensor names of the graph

5.10.8. get_max_input_shapes

Get max shapes of input tensors in a graph. For static models, the max shape is fixed and it should not be changed. For dynamic models, the tensor shape should be smaller than or equal to the max shape.

Interface:

def get_max_input_shapes(self, graph_name: str)-> dict {str : list}

Parameters

• graph_name : str

The specified graph name

Returns

• max_shapes : dict {str : list}

Max shape of the input tensors

5.10.9. get_input_shape

Get the maximum dimension shape of an input tensor in a graph. There are cases that there are multiple input shapes in one input name, This API only returns the maximum dimension one for the memory allocation in order to get the best performance.

Interface:

def get_input_shape(self, graph_name: str, tensor_name: str)-> list

Parameters

• graph_name : str

The specified graph name

• tensor_name : str

The specified input tensor name

Returns

• tensor_shape : list

The maxmim dimension shape of the tensor

5.10.10. get_max_output_shapes

Get max shapes of input tensors in a graph. For static models, the max shape is fixed and it should not be changed. For dynamic models, the tensor shape should be smaller than or equal to the max shape.

Interface:

def get_max_output_shapes(self, graph_name: str)-> dict {str : list}

Parameters

• graph_name : str

The specified graph name

Returns

• max_shapes : dict {str : list}

Max shape of the output tensors

5.10.11. get_output_shape

Get the shape of an output tensor in a graph.

Interface:

def get_output_shape(self, graph_name: str, tensor_name: str)-> list

Parameters

• graph_name : str

The specified graph name

• tensor_name : str

The specified output tensor name

Returns

tensor_shape : list

The shape of the tensor

5.10.12. get_input_dtype

Get scale of an input tensor. Only used for int8 models.

Interface:

def get_input_dtype(self, graph_name: str, tensor_name: str)-> sail.Dtype

Parameters

• graph_name : str

The specified graph name

• tensor_name : str

The specified output tensor name

Returns

• scale: sail.Dtype

Data type of the input tensor

5.10.13. get_output_dtype

Get scale of an output tensor. Only used for int8 models.

Interface:

def get_output_dtype(self, graph_name: str, tensor_name: str)-> sail.Dtype

Parameters

• graph_name : str

The specified graph name

• tensor_name : str

The specified output tensor name

Returns

• scale: sail.Dtype

Data type of the output tensor

5.10.14. get_input_scale

Get scale of an input tensor. Only used for int8 models.

Interface:

def get_input_scale(self, graph_name: str, tensor_name: str)-> float32

Parameters

• graph_name : str

The specified graph name

• tensor_name : str

The specified output tensor name

Returns

• scale: float32

Scale of the input tensor

5.10.15. get_output_scale

Get scale of an output tensor. Only used for int8 models.

Interface:

def get_output_scale(self, graph_name: str, tensor_name: str)-> float32

Parameters

• graph_name : str

The specified graph name

• tensor_name : str

The specified output tensor name

Returns

• scale: float32

Scale of the output tensor

5.10.16. process

Inference with provided system data of input tensors, with or without input shapes and output tensors.

Interface:

def process(self,
            graph_name: str,
            input_tensors: dict {str : numpy.array}
            )-> dict {str : numpy.array}

Parameters

• graph_name : str

The specified graph name

• input_tensors : dict {str : numpy.array}

Data of all input tensors in system memory

Returns

• output_tensors : dict {str : numpy.array}

Data of all output tensors in system memory

Interface:

def process(self,
            graph_name: str,
            input_tensors: dict {str : sail.Tensor},
            output_tensors: dict {str : sail.Tensor})

Parameters

• graph_name : str

The specified graph name

• input_tensors : dict {str : sail.Tensor}

Input tensors managed by user

• output_tensors : dict {str : sail.Tensor}

Output tensors managed by user

Interface:

def process(self,
            graph_name: str,
            input_tensors: dict {str : sail.Tensor},
            input_shapes: dict {str : list},
            output_tensors: dict {str : sail.Tensor})

Parameters

• graph_name : str

The specified graph name

• input_tensors : dict {str : sail.Tensor}

Input tensors managed by user

• input_shapes : dict {str : list}

Shapes of all input tensors

• output_tensors : dict {str : sail.Tensor}

Output tensors managed by user

5.10.17. get_device_id

Get device id of this engine

Interface:

def get_device_id(self)-> int

Returns

• tpu_id : int

tpu id of this engine

5.10.18. create_input_tensors_map

Create input tensors map, according to and bmodel.

Interface:

def create_input_tensors_map(self,
                            graph_name: str,
                            create_mode: int)-> dict[str,Tensor]

Parameters:

• graph_name : str

The specified graph name.

• create_mode: int

Tensor Create mode, case 0: only allocate system memory; case 1: only allocate device memory; case other: according to engine IOMode.

Returns

• output: dict[str,Tensor]

Output result.

5.10.19. create_output_tensors_map

Create output tensors map, according to and bmodel.

Interface:

def create_output_tensors_map(self,
                            graph_name: str,
                            create_mode: int)-> dict[str,Tensor]

Parameters:

• graph_name : str

The specified graph name.

• create_mode: int

Tensor Create mode, case 0: only allocate system memory; case 1: only allocate device memory; case other: according to engine IOMode.

Returns

• output: dict[str,Tensor]

Output result.