TensorFlow Serving 可以快速部署 Tensorflow 模型，上线 gRPC 或 REST API。
官方推荐 Docker 部署，也给了训练到部署的完整教程：Servers: TFX for TensorFlow Serving。本文只是遵照教程进行的练习，有助于了解 TensorFlow 训练到部署的整个过程。
准备环境准备好 TensorFlow 环境，导入依赖：
import sys # Confirm that we're using Python 3 assert sys.version_info.major == 3, 'Oops, not running Python 3. Use Runtime > Change runtime type' import tensorflow as tf from tensorflow import keras # Helper libraries import numpy as np import matplotlib.pyplot as plt import os import subprocess print(f'TensorFlow version: {tf.__version__}') print(f'TensorFlow GPU support: {tf.test.is_built_with_gpu_support()}') physical_gpus = tf.config.list_physical_devices('GPU') print(physical_gpus) for gpu in physical_gpus: # memory growth must be set before GPUs have been initialized tf.config.experimental.set_memory_growth(gpu, True) logical_gpus = tf.config.experimental.list_logical_devices('GPU') print(len(physical_gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs") TensorFlow version: 2.4.1 TensorFlow GPU support: True [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')] 1 Physical GPUs, 1 Logical GPUs 创建模型载入 Fashion MNIST 数据集：
fashion_mnist = keras.datasets.fashion_mnist (train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data() # scale the values to 0.0 to 1.0 train_images = train_images / 255.0 test_images = test_images / 255.0 # reshape for feeding into the model train_images = train_images.reshape(train_images.shape[0], 28, 28, 1) test_images = test_images.reshape(test_images.shape[0], 28, 28, 1) class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot'] print('\ntrain_images.shape: {}, of {}'.format(train_images.shape, train_images.dtype)) print('test_images.shape: {}, of {}'.format(test_images.shape, test_images.dtype)) train_images.shape: (60000, 28, 28, 1), of float64 test_images.shape: (10000, 28, 28, 1), of float64用最简单的 CNN 训练模型，
model = keras.Sequential([ keras.layers.Conv2D(input_shape=(28,28,1), filters=8, kernel_size=3, strides=2, activation='relu',), keras.layers.Flatten(), keras.layers.Dense(10,) ]) model.summary() testing = False epochs = 5 model.compile(optimizer='adam', loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True), metrics=[keras.metrics.SparseCategoricalAccuracy()]) model.fit(train_images, train_labels, epochs=epochs) test_loss, test_acc = model.evaluate(test_images, test_labels) print('\nTest accuracy: {}'.format(test_acc)) Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= Conv1 (Conv2D) (None, 13, 13, 8) 80 _________________________________________________________________ flatten (Flatten) (None, 1352) 0 _________________________________________________________________ Dense (Dense) (None, 10) 13530 ================================================================= Total params: 13,610 Trainable params: 13,610 Non-trainable params: 0 _________________________________________________________________ Epoch 1/5 1875/1875 [==============================] - 3s 722us/step - loss: 0.7387 - sparse_categorical_accuracy: 0.7449 Epoch 2/5 1875/1875 [==============================] - 1s 793us/step - loss: 0.4561 - sparse_categorical_accuracy: 0.8408 Epoch 3/5 1875/1875 [==============================] - 1s 720us/step - loss: 0.4097 - sparse_categorical_accuracy: 0.8566 Epoch 4/5 1875/1875 [==============================] - 1s 718us/step - loss: 0.3899 - sparse_categorical_accuracy: 0.8636 Epoch 5/5 1875/1875 [==============================] - 1s 719us/step - loss: 0.3673 - sparse_categorical_accuracy: 0.8701 313/313 [==============================] - 0s 782us/step - loss: 0.3937 - sparse_categorical_accuracy: 0.8630 Test accuracy: 0.8629999756813049 保存模型将模型保存成 SavedModel 格式，路径里加上版本号，以便 TensorFlow Serving 时可选择模型版本。
# Fetch the Keras session and save the model # The signature definition is defined by the input and output tensors, # and stored with the default serving key import tempfile MODEL_DIR = os.path.join(tempfile.gettempdir(), 'tfx') version = 1 export_path = os.path.join(MODEL_DIR, str(version)) print('export_path = {}\n'.format(export_path)) tf.keras.models.save_model( model, export_path, overwrite=True, include_optimizer=True, save_format=None, signatures=None, options=None ) print('\nSaved model:') !ls -l {export_path} export_path = /tmp/tfx/1 INFO:tensorflow:Assets written to: /tmp/tfx/1/assets Saved model: total 88 drwxr-xr-x 2 john john 4096 Apr 13 15:10 assets -rw-rw-r-- 1 john john 78169 Apr 13 15:12 saved_model.pb drwxr-xr-x 2 john john 4096 Apr 13 15:12 variables 查看模型使用 saved_model_cli 工具查看模型的 MetaGraphDefs (the models) 和 SignatureDefs (the methods you can call)，了解信息。
!saved_model_cli show --dir '/tmp/tfx/1' --all 2021-04-13 15:12:29.433576: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs: signature_def['__saved_model_init_op']: The given SavedModel SignatureDef contains the following input(s): The given SavedModel SignatureDef contains the following output(s): outputs['__saved_model_init_op'] tensor_info: dtype: DT_INVALID shape: unknown_rank name: NoOp Method name is: signature_def['serving_default']: The given SavedModel SignatureDef contains the following input(s): inputs['Conv1_input'] tensor_info: dtype: DT_FLOAT shape: (-1, 28, 28, 1) name: serving_default_Conv1_input:0 The given SavedModel SignatureDef contains the following output(s): outputs['Dense'] tensor_info: dtype: DT_FLOAT shape: (-1, 10) name: StatefulPartitionedCall:0 Method name is: tensorflow/serving/predict Defined Functions: Function Name: '__call__' Option #1 Callable with: Argument #1 Conv1_input: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32,) Argument #2 DType: bool Value: False Argument #3 DType: NoneType Value: None Option #2 Callable with: Argument #1 inputs: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32,) Argument #2 DType: bool Value: False Argument #3 DType: NoneType Value: None Option #3 Callable with: Argument #1 inputs: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32,) Argument #2 DType: bool Value: True Argument #3 DType: NoneType Value: None Option #4 Callable with: Argument #1 Conv1_input: TensorSpec(shape=(None, 28, 28, 1), dtype=tf.float32,) Argument #2 DType: bool Value: True Argument #3 DType: NoneType Value: None ... 部署模型 安装 Serving echo "deb [arch=amd64] stable tensorflow-model-server tensorflow-model-server-universal" | sudo tee /etc/apt/sources.list.d/tensorflow-serving.list && \ curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | sudo apt-key add - sudo apt update sudo apt install tensorflow-model-server 开启 Serving开启 TensorFlow Serving ，提供 REST API ：
rest_api_port: REST 请求端口。
model_name: REST 请求 URL ，自定义的名称。