宝塔服务器面板，一键全能部署及管理，送你10850元礼包，点我领取

tensorflow graphdefs to TensorFlow Lite’s flat buffer format
tf、tflite存储格式不同,数据精度不同.

量化

量化好处自不必说了,减小模型大小,减少内存占用,提升速度,以及某些架构的硬件只支持int8,这时候必须量化.缺点就是模型精度有所下降.

量化也分很多种
tflite量化-风君子博客

不同的硬件设备所要求的量化方式是不同的,Edge TPU上只支持后两种.后两种都是要求有训练数据的.

tflite量化-风君子博客
从上图看,post-training的量化方式对推理速度的提升很有限.

Post-training quantization

Post-training quantization也分很多种.
tflite量化-风君子博客
Edge TPU只能使用Full integer quantization.

Float16 quantization of weights

import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_saved_modelsaved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_types = [tf.lite.constants.FLOAT16]
tflite_quant_model = converter.convert)

没啥说的,很少用.还不如直接fp32省心了,无非是减小点模型大小.

Dynamic range quantization

只能用于CPU加速

import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_saved_modelsaved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_quant_model = converter.convert)

只把weights从fp32–>int8. 推理的时候,把int8转回fp32,用floating-point kernels做计算.
tflite量化-风君子博客
注意:However, the outputs are still stored using floating point, so that the speedup with dynamic-range ops is less than a full fixed-point computation

tflite量化-风君子博客
量化得到的模型只有weights是int8的,activation的部分依然是fp32的,activation输出的量化在推理的时候做,一个graph的不同部分用不同的kernel去处理.

Full integer quantization of weights and activations

you need to measure the dynamic range of activations and inputs by supplying a representative data set
要提供representative data set

模型的weights和activation全部转换为int8. 在cpu上速度提高3-4倍. 支持EDGE TPU上运行.

量化后的模型的input和output仍然是fp32的,对于activation op没有量化版本的实现的,会保留为fp32。

import tensorflow as tf

def representative_dataset_gen):
  for _ in rangenum_calibration_steps):
    # Get sample input data as a numpy array in a method of your choosing.
    yield [input]

converter = tf.lite.TFLiteConverter.from_saved_modelsaved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset_gen
tflite_quant_model = converter.convert)

对于没有量化版本实现的op就不量化了,保留为fp32.
对于Edge TPU这种只支持int8 op的硬件,必须模型里的op都是int8的.

converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8

通过指定converter属性要求必须全部op都转为int8的,否则直接报错.

tflite官方支持的模型仓库

int8<–>fp32

int8和fp32之间的映射关系如何确定,参考下面官方链接
https://www.tensorflow.org/lite/performance/quantization_spec

值得一看

https://zhuanlan.zhihu.com/p/99424468
https://blog.csdn.net/qq_19784349/article/details/82883271
https://murphypei.github.io/blog/2019/11/neural-network-quantization