TPU-MLIR Technical Reference Manual

Legal Notices

Copyright © SOPHGO 2022. All rights reserved.

No part or all of the contents of this document may be copied, reproduced or transmitted in any form by any organization or individual without the written permission of the Company.

Attention

All products, services or features, etc. you purchased is subject to SOPHGO’s business contracts and terms. All or part of the products, services or features described in this document may not be covered by your purchase or use. Unless otherwise agreed in the contract, SOPHGO makes no representations or warranties (including express and implied) regarding the contents of this document. The contents of this document may be updated from time to time due to product version upgrades or other reasons. Unless otherwise agreed, this document is intended as a guide only. All statements, information and recommendations in this document do not constitute any warranty, express or implied.

Technical Support

Address

Building 1, Zhongguancun Integrated Circuit Design Park (ICPARK), No. 9 Fenghao East Road, Haidian District, Beijing

Postcode

100094

URL

https://www.sophgo.com/

Email

sales@sophgo.com

Tel

+86-10-57590723

+86-10-57590724

Release Record

Version	Release date	Explanation
v0.6.0	2022.11.05	support mix precision
v0.5.0	2022.10.20	support test model_zoo models
v0.4.0	2022.09.20	support convert caffe model
v0.3.0	2022.08.24	Support TFLite. Add the chapter on TFLite model conversion.
v0.2.0	2022.08.02	Add the chapter on test samples in running SDK.
v0.1.0	2022.07.29	Initial release, supporting resnet/mobilenet/vgg/ssd/yolov5s and using yolov5s as the use case.

Table of Contents

• 1. TPU-MLIR Introduction
• 2. Environment Setup
  • 2.1. Code Download
  • 2.2. Docker Configuration
  • 2.3. ModelZoo (Optional)
  • 2.4. Compilation
• 3. User Interface
  • 3.1. Introduction
  • 3.2. model_transform.py
  • 3.3. run_calibration.py
  • 3.4. model_deploy.py
  • 3.5. Other Tools
    • 3.5.1. model_runner.py
    • 3.5.2. npz_tool.py
    • 3.5.3. visual.py
• 4. Overall Design
  • 4.1. Layered
  • 4.2. Top Pass
  • 4.3. Tpu Pass
  • 4.4. Other Passes
• 5. Front-end Conversion
  • 5.1. Main Work
  • 5.2. Workflow
  • 5.3. Example
• 6. Quantization
  • 6.1. Basic Concepts
    • 6.1.1. Asymmetric Quantization
    • 6.1.2. Symmetric Quantization
  • 6.2. Scale Conversion
  • 6.3. Quantization derivation
    • 6.3.1. Convolution
    • 6.3.2. InnerProduct
    • 6.3.3. Add
    • 6.3.4. AvgPool
    • 6.3.5. LeakyReLU
    • 6.3.6. Pad
    • 6.3.7. PReLU
• 7. Calibration
  • 7.1. General introduction
  • 7.2. Calibration data screening and preprocessing
    • 7.2.1. Screening Principles
    • 7.2.2. Input format and preprocessing
  • 7.3. Algorithm Implementation
    • 7.3.1. KLD Algorithm
    • 7.3.2. Auto-tune Algorithm
  • 7.4. Example: yolov5s calibration
  • 7.5. visual tool introduction
• 8. Lowering
  • 8.1. Basic Process
  • 8.2. Mixed precision
• 9. SubNet
• 10. LayerGroup
  • 10.1. Basic Concepts
  • 10.2. BackwardH
  • 10.3. Dividing the Mem Cycle
  • 10.4. LMEM Allocation
  • 10.5. Divide the optimal Group
• 11. GMEM Allocation
  • 11.1. 1. Purpose
  • 11.2. 1. Principle
    • 11.2.1. 2.1. GMEM allocation in weight tensor
    • 11.2.2. 2.2. GMEM allocation in global neuron tensors
• 12. CodeGen
• 13. MLIR Definition
  • 13.1. Top Dialect
    • 13.1.1. Operations
• 14. Accuracy Validation
  • 14.1. Introduction
    • 14.1.1. Objects
    • 14.1.2. Metrics
    • 14.1.3. Datasets
  • 14.2. Validation Interface
  • 14.3. Validation Example
    • 14.3.1. mobilenet_v2
    • 14.3.2. yolov5s
• 15. quantzation aware traing
  • 15.1. Basic Principles
  • 15.2. tpu-mlir QAT implementation scheme and characteristics
    • 15.2.1. Main body flow
    • 15.2.2. Features of the Scheme
  • 15.3. Installation Method
    • 15.3.1. Install from source
    • 15.3.2. Installing the wheel file
  • 15.4. Basic Steps
    • 15.4.1. Step 1: Interface import and model prepare
    • 15.4.2. Step 2: Calibration and quantization training
    • 15.4.3. Step 3: Export tuned fp32 model
    • 15.4.4. Step 4: Initiate the training
    • 15.4.5. Step 5: Transform deployment
  • 15.5. Use Examples-resnet18
  • 15.6. Tpu-mlir QAT test environment
    • 15.6.1. Adding a cfg File
    • 15.6.2. Modify and execute run_eval.py
  • 15.7. Use Examples-yolov5s
• 16. TpuLang Interface
  • 16.1. Main Work
  • 16.2. Work Process
  • 16.3. Operator Conversion Example