TPU-MLIR Quick Start

Legal Notices

Copyright © SOPHGO 2022. All rights reserved.

No part or all of the contents of this document may be copied, reproduced or transmitted in any form by any organization or individual without the written permission of the Company.

Attention

All products, services or features, etc. you purchased is subject to SOPHGO’s business contracts and terms. All or part of the products, services or features described in this document may not be covered by your purchase or use. Unless otherwise agreed in the contract, SOPHGO makes no representations or warranties (including express and implied) regarding the contents of this document. The contents of this document may be updated from time to time due to product version upgrades or other reasons. Unless otherwise agreed, this document is intended as a guide only. All statements, information and recommendations in this document do not constitute any warranty, express or implied.

Technical Support

Address

Building 1, Zhongguancun Integrated Circuit Design Park (ICPARK), No. 9 Fenghao East Road, Haidian District, Beijing

Postcode

100094

URL

https://www.sophgo.com/

Email

sales@sophgo.com

Tel

+86-10-57590723

+86-10-57590724

Release Record

Version	Release date	Explanation
v0.6.0	2022.11.05	support mix precision
v0.5.0	2022.10.20	support test model_zoo models
v0.4.0	2022.09.20	support convert caffe model
v0.3.0	2022.08.24	Support TFLite. Add the chapter on TFLite model conversion.
v0.2.0	2022.08.02	Add the chapter on test samples in running SDK.
v0.1.0	2022.07.29	Initial release, supporting resnet/mobilenet/vgg/ssd/yolov5s and using yolov5s as the use case.

Table of contents

• 1. TPU-MLIR Introduction
• 2. Environment Setup
• 3. Compile the ONNX model
  • 3.1. Load tpu-mlir
  • 3.2. Prepare working directory
  • 3.3. ONNX to MLIR
  • 3.4. MLIR to F16 bmodel
  • 3.5. MLIR to INT8 bmodel
    • 3.5.1. Calibration table generation
    • 3.5.2. Compile to INT8 symmetric quantized model
  • 3.6. Effect comparison
  • 3.7. Model performance test
    • 3.7.1. Install the libsophon
    • 3.7.2. Check the performance of BModel
• 4. Compile the Torch Model
  • 4.1. Load tpu-mlir
  • 4.2. Prepare working directory
  • 4.3. TORCH to MLIR
  • 4.4. MLIR to F16 bmodel
  • 4.5. MLIR to INT8 bmodel
    • 4.5.1. Calibration table generation
    • 4.5.2. Compile to INT8 symmetric quantized model
  • 4.6. Effect comparison
• 5. Compile the TFLite model
  • 5.1. Load tpu-mlir
  • 5.2. Prepare working directory
  • 5.3. TFLite to MLIR
  • 5.4. MLIR to bmodel
• 6. Compile the Caffe model
  • 6.1. Load tpu-mlir
  • 6.2. Prepare working directory
  • 6.3. Caffe to MLIR
  • 6.4. MLIR to F32 bmodel
  • 6.5. MLIR to INT8 bmodel
    • 6.5.1. Calibration table generation
    • 6.5.2. Compile to INT8 symmetric quantized model
    • 6.5.3. Compile to INT8 asymmetric quantized model
• 7. Use TPU for Preprocessing
  • 7.1. Model Deployment Example
    • 7.1.1. Deploy to BM168x
    • 7.1.2. Deploy to CV18xx
• 8. Reference for converting model to ONNX format
  • 8.1. PyTorch model to ONNX
    • 8.1.1. Step 0: Create a working directory
    • 8.1.2. Step 1: Build and save the model
    • 8.1.3. Step 2: Export ONNX model
  • 8.2. TensorFlow model to ONNX
    • 8.2.1. Step 0: Create a working directory
    • 8.2.2. Step 1: Prepare and convert the model
  • 8.3. PaddlePaddle model to ONNX
    • 8.3.1. Step 0: Create a working directory
    • 8.3.2. Step 1: Prepare the model
    • 8.3.3. Step 2: Convert the model
• 9. CV18xx Guidance
  • 9.1. Compile yolov5 model
    • 9.1.1. TPU-MLIR Setup
    • 9.1.2. Prepare working directory
    • 9.1.3. ONNX to MLIR
    • 9.1.4. MLIR to BF16 Model
    • 9.1.5. MLIR to INT8 Model
    • 9.1.6. Result Comparison
  • 9.2. Merge cvimodel Files
    • 9.2.1. Step 0: generate the cvimodel for batch 1
    • 9.2.2. Step 1: generate the cvimodel for batch 2
    • 9.2.3. Step 2: merge the cvimodel of batch 1 and batch 2
    • 9.2.4. Step 3: use the cvimodel through the runtime interface
    • 9.2.5. Overview:
  • 9.3. Compile and Run the Runtime Sample
    • 9.3.1. 1) Run the provided pre-build samples
    • 9.3.2. 2) Cross-compile samples
    • 9.3.3. 3) Run samples in docker environment
• 10. Test SDK release package with TPU-PERF
  • 10.1. Configure the system environment
  • 10.2. Get the model-zoo model
  • 10.3. Get the tpu-perf tool
  • 10.4. Test process
    • 10.4.1. Unzip the SDK and create a Docker container
    • 10.4.2. Set environment variables and install tpu-perf
    • 10.4.3. Run the test
    • 10.4.4. Configure SOC device
    • 10.4.5. Run the test
    • 10.4.6. Precision test