Decompress test package
mkdir -p sophon/model-zoo
tar -xvf path/to/model-zoo_<date>.tar.bz2 --strip-components=1 -C sophon/model-zoo
cd sophon/model-zoo
The catalogue structure of test package is as below:
├── config.yaml
├── requirements.txt
├── data
├── dataset
├── harness
├── output
└── ...
General configurations included in config.yaml: Catalogue of dataset, root catalogue of model, etc. as well as some multiplexing parameters and commands.
requirements.txt is the python dependency of model-zoo.
dataset catalogue includes imagenet dataset pre-processing of model so as to be called by tpu_perf as plugin.
data catalogue will be used for storing lmdb dataset.
output catalogue will be used for storing bmodel output by compilation and some intermediate data.
Other catalogues include the information and configuration of different models. Catalogue corresponding to each model contains one config.yaml file, which is configured with model name, path, FLOPs, dataset production parameters and quantization compilation command of model.
Prepare dataset
ImageNet
Download the followings of imagenet 2012 dataset
ILSVRC2012_img_val.tar（MD5 29b22e2961454d5413ddabcf34fc5622）.
Decompress ILSVRC2012_img_val.tar to dataset/ILSVRC2012/ILSVRC2012_img_val catalogue:
cd path/to/sophon/model-zoo
tar xvf path/to/ILSVRC2012_img_val.tar -C dataset/ILSVRC2012/ILSVRC2012_img_val
COCO (optional)
If coco dataset is used for precision test, please download and decompress it according to the following steps:
cd path/to/sophon/model-zoo/dataset/COCO2017/
wget http://images.cocodataset.org/annotations/annotations_trainval2017.zip
wget http://images.cocodataset.org/zips/val2017.zip
unzip annotations_trainval2017.zip
unzip val2017.zip
Run performance and accuracy tests in SOC
Performance and accuracy testing only depends on the libsophon running environment, so the model compiled in the toolchain compilation environment can be packaged together with Model-zoo, and TPU_perf is used for performance and accuracy testing in the SOC environment. However, due to the limited eMMC, the complete Model-zoo and compiled output may not be fully copied to the SOC.Here is a method to compile the model in the toolchain environment and run the test in the soc by mounting the linux nfs remote file system.
First, install the nfs service on the toolchain environment server 『host system』:
sudo apt install nfs-kernel-server
Add the shared directory in /etc/exports:
/path/to/sophon *(rw,sync,no_subtree_check,no_root_squash)
*``Indicates that everyone can access the shared directory, or it can be configured to be accessible by a specific network segment or IP, such as ``192.168.43.0/24 .
Then execute the following command to make the configuration take effect:
sudo exportfs -a
sudo systemctl restart nfs-kernel-server
In addition, you need to add read permissions to the images in the dataset directory:
chmod -R +r path/to/sophon/model-zoo/dataset
Install the client in soc and mount the shared directory:
mkdir sophon
sudo apt-get install -y nfs-common
sudo mount -t nfs <IP>:/path/to/sophon ./sophon
This will make the test directory accessible in the soc environment. The remaining operations of the soc test are basically the same as those of pcie, please refer to the following for operations; the difference in the execution position of the running environment command has been added in the execution place.
Prepare running environment
Install the required dependencies for model-zoo using pip:
sudo apt-get install -y libgl1 # For OpenCV
cd path/to/sophon/model-zoo
pip3 install -r requirements.txt
If running the tests in the soc environment, execute the above command on the soc.
In addition, call tpu hardware in running environment for performance and precision test. Please install libsophon by following libsophon user manual.
Install the tpu_perf tool
Using the tpu_perf tool, you can easily verify the model performance and accuracy, and realize batch testing and verification of multiple models and batch sizes. You can get the latest version of the tpu_perf release corresponding to the architecture from here, or you can compile it from the source code. The following are the source code compilation steps.
Please compile the tpu_perf source code in the deployment environment. Compilation depends on libsophon-dev, please refer to the libsophon manual for installation.
Compilation depends on the python packaging tool, please use pip to install:
pip3 install setuptools wheel
Compilation depends on protoc.On x86 machines, please use the following installation method to maintain compatibility with the tpu-nntc environment:
wget -O /tmp/protoc-3.19.4-linux-x86_64.zip \
https://github.com/protocolbuffers/protobuf/releases/download/v3.19.4/protoc-3.19.4-linux-x86_64.zip
cd path/to/sophon
mkdir protoc
unzip -o -d ./protoc /tmp/protoc-3.19.4-linux-x86_64.zip
In the soc environment, you can use the package manager to install directly:
sudo apt-get install -y protobuf-compiler
Then, get the tpu_perf source code and unzip it, and then execute the compilation. The command is as follows:
mkdir tpu_perf
tar xvf path/to/tpu-perf-X.Y.Z.tar.gz --strip-components=1 -C tpu_perf
# If the tpu_perf/build directory already exists, it is recommended to delete it first.
# rm -r tpu_perf/build
# Execute compilation.
mkdir -p tpu_perf/build
cd tpu_perf/build
PATH=$PATH:../../protoc/bin cmake ..
make install/strip -j4
Among them, the PATH environment variable specified in the cmake generation step is to use the correct protoc program, and it can be not specified in the soc environment.
After successful compilation, the whl package will be generated under tpu_perf/python/dist.
When tpu_perf runs also depends on numpy , lmdb , protobuf==3.19.* , psutil , pyyaml numpy , lmdb , protobuf==3.19.* , psutil , pyyaml ,make sure you can connect to the Internet or install dependencies manually when installing whl.
cd ..
pip3 install --upgrade python/dist/tpu_perf-X.Y.Z-py3-none-<arch>.whl
Please note that if you compile tpu_perf in the soc environment, please download and compile it on the soc environment and the toolchain environment 『host system』 respectively to obtain a package for ‘Aarch64’ and a package for ‘x86_64’ for both machines.
Prepare tool chain compilation environment
Tool chain software is suggested to be used in docker environment. docker of the latest version can be installed by referring to official tutorial.After installation, execute the following script and add the present user into docke group to obtain docker execution authority.
sudo usermod -aG docker $USER
newgrp docker
Next, pull docker mirroring from docker hub:
docker pull sophgo/tpuc_dev:v2.1
Then decompress tool chain development package from test package catalogue. The latest tool chain development package could be obtained from official website.
cd path/to/sophon
tar xvf tpu-nntc_vx.y.z-<hash>-<date>.tar.gz
# Optional, copy the tpu_perf installation package to the working directory
cp path/to/tpu_perf-X.Y.Z-py3-none-x86_64.whl ./
Next, start docker container and map the catalogue to docker.
docker run -td -v $(pwd):/workspace --name nntc sophgo/tpuc_dev:v2.1 bash
Enter docker environment by executing the following commands:
docker exec -it nntc bash
One example
By taking resnet50 network as an example, the process of performance and precision test is executed for once here completely so that users could gain a full understanding of test process.
First, compile model in『docker tool chain environment』:
# Env setup
cd /workspace/tpu-nntc_vx.y.z-<hash>-<date>
source scripts/envsetup.sh
# Prepare working directory
cd /workspace/model-zoo
mkdir -p output/resnet50
cp vision/classification/ResNet50-Caffe/ResNet-50-* output/resnet50
cd output/resnet50
python3 -m ufw.cali.cali_model \
--model ./ResNet-50-deploy.prototxt \
--weights ./ResNet-50-model.caffemodel \
--cali_image_path /workspace/model-zoo/dataset/ILSVRC2012/caliset \
--test_iterations 10 \
--net_name resnet50 \
--postprocess_and_calc_score_class none \
--target BM1684X \
--cali_iterations 100 \
--cali_image_preprocess='
resize_side=256;
crop_w=224,crop_h=224;
mean_value=103.94:116.78:123.68,scale=1' \
--input_shapes=[1,3,224,224]
The command quantifies model into int8 model in virtue of auto_cali quantization tool. Quantization dataset is required and the pre-processing parameter needs designating.
Next, carry out performance test and precision verification of model under the running environment. First, exit tool chain docker environment:
exit
If running the tests in the soc environment, execute the following model test commands on the soc.
Verify model reasoning time via bmrt_test tool
cd path/to/sophon/model-zoo
bmrt_test --bmodel output/resnet50/resnet50_batch1/compilation.bmodel
The key performance parameters of model will be printed after execution.
Next, the running precision test program will verify bmodel precision on dataset:
python3 harness/topk.py \
--mean 103.94,116.78,123.68 --scale 1 --size 224 \
--bmodel output/resnet50/resnet50_batch1/compilation.bmodel \
--image_path ./dataset/ILSVRC2012/ILSVRC2012_img_val \
--list_file ./dataset/ILSVRC2012/caffe_val.txt
The last line of program will output top 1 and top 5 precision measured by bmodel.
The following two chapters introduce how to verify performance and precision using tpu_perf. In consideration of multiple models and relatively long execution time of commands, it is suggested to execute command by putting it in terminal manager of screen/tmux, if operation is made via ssh dialogue so as to avoid suspending task once dialogue is finished.
Performance test
First, compile model in『docker tool chain environment:
# Env setup
cd /workspace/tpu-nntc_vx.y.z-<hash>-<date>
source scripts/envsetup.sh
# Install tpu_perf tool
pip3 install --upgrade path/to/tpu_perf-X.Y.Z-py3-none-x86_64.whl
cd /workspace/model-zoo
python3 -m tpu_perf.build --list default_cases.txt --time
tpu_perf all commands can specify a case list file to run, here default_cases.txt is used. The full list of cases can be run by specifying full_cases.txt (may take a long time), or a custom list file. If no list is specified, tpu_perf will traverse the current directory looking for config.yaml files and execute them one by one.
Running test shall be carried out in the running environment beyond docker. It is optional to exit docker environment:
exit
If running the tests in the soc environment, execute the following model test commands on the soc.
As model is executed by root user by default after generation in docker, change the owner of output catalogue to the current user.
cd path/to/sophon/model-zoo
sudo chown -R $(whoami):$(whoami) output
Then install and use tpu_perf tool to run model and generate performance data:
pip3 install --upgrade path/to/tpu_perf-X.Y.Z-py3-none-<arch>.whl
python3 -m tpu_perf.run --list default_cases.txt
Performance data can be obtained from output/stats.csv.
Precision verification
First, prepare dataset and compile model in『docker tool chain environment:
Prepare quantization and test LMDB
If model quantization is carried out via auto_cali , quantify model via image set or LMDB dataset. In fact, if image set is designated, auto_cali will generate LMDB dataset automatically based on pre-processing parameters for the purpose of quantization. As several models use the same image set and pre-processing in course of batch quantization, tpu_perf tool is used here to finish LMDB dataset in batch first before calling auto_cali for quantization.
# Install the tpu_perf tool
cd /workspace/tpu_perf
pip3 install --upgrade path/to/tpu_perf-X.Y.Z-py3-none-x86_64.whl
cd /workspace/model-zoo
python3 -m tpu_perf.make_lmdb --list default_cases.txt
Executing this command generates preprocessed lmdb datasets for quantification and testing based on the configuration of each model.
tpu_perf all commands can specify a case list file to run, here default_cases.txt is used. The full list of cases can be run by specifying full_cases.txt (may take a long time), or a custom list file. If no list is specified, tpu_perf will traverse the current directory looking for config.yaml files and execute them one by one.
The tool configuration uses 200 image sets for quantization by default. If you want to choose other image sets, you can put the images in the dataset/ILSVRC2012/caliset directory, and pass the cali_set field in config.yaml specifies the number of quantization sets. For details, see the preprocessing implementation in the dataset directory.
Script execution time may be long, pls wait patiently.
Model quantization and compilation
# Env setup
cd /workspace/tpu-nntc_vx.y.z-<hash>-<date>
source scripts/envsetup.sh
cd /workspace/model-zoo
python3 -m tpu_perf.build --list default_cases.txt
This command quantizes and compiles the models according to the configuration of each model. If there are many models, the script execution time will be longer, please be patient.
Precision test shall be carried out in the running environment beyond docker. It is optional to exit docker environment:
exit
Precision test
If running the tests in the soc environment, execute the following model test commands on the soc.
As dataset and model are executed by root user by default after generation in docker, change the owner of generated catalogue to the current user.
# Install the tpu_perf tool
pip3 install --upgrade path/to/tpu_perf-X.Y.Z-py3-none-<arch>.whl
cd path/to/sophon/model-zoo
sudo chown -R $(whoami):$(whoami) data
sudo chown -R $(whoami):$(whoami) output
The tests can then be run using the tpu_perf tool:
python3 -m tpu_perf.precision_benchmark --list default_cases.txt
Various types of precision data are available in individual csv files in the output directory.