I tried to find a good basic example, but it looks like the documentation and samples are a bit scattered about this topic. So let's start with a basic example: tf.estimator quickstart .
This specific example does not actually export the model, so let it do it (not necessary to use 1):
def serving_input_receiver_fn(): """Build the serving inputs."""
A huge star on this code . An error appears in TensorFlow 1.3 that does not allow you to perform the above export on a “canned” evaluation (for example, DNNClassifier). For a workaround, see "Appendix: Workaround."
The following is the export_dir code (return value at the export stage) to emphasize that it is not "/ path / in / model", but rather is a subdirectory of this directory, whose name is a timestamp.
Use case 1: make a prediction in the same process as training
This is a scientific experience based on the use of sci-kit, and is already shown as a sample. For completeness, you simply call predict on the trained model:
classifier.train(input_fn=train_input_fn, steps=2000) # [...snip...] predictions = list(classifier.predict(input_fn=predict_input_fn)) predicted_classes = [p["classes"] for p in predictions]
Use example 2. Download SavedModel in Python / Java / C ++ and make predictions
Python client
Perhaps the easiest way to use it if you want to make a prediction in Python is SavedModelPredictor . In a Python program that will use SavedModel , we need code like this:
from tensorflow.contrib import predictor predict_fn = predictor.from_saved_model(export_dir) predictions = predict_fn( {"x": [[6.4, 3.2, 4.5, 1.5], [5.8, 3.1, 5.0, 1.7]]}) print(predictions['scores'])
Java client
package dummy; import java.nio.FloatBuffer; import java.util.Arrays; import java.util.List; import org.tensorflow.SavedModelBundle; import org.tensorflow.Session; import org.tensorflow.Tensor; public class Client { public static void main(String[] args) { Session session = SavedModelBundle.load(args[0], "serve").session(); Tensor x = Tensor.create( new long[] {2, 4}, FloatBuffer.wrap( new float[] { 6.4f, 3.2f, 4.5f, 1.5f, 5.8f, 3.1f, 5.0f, 1.7f })); // Doesn't look like Java has a good way to convert the // input/output name ("x", "scores") to their underlying tensor, // so we hard code them ("Placeholder:0", ...). // You can inspect them on the command-line with saved_model_cli: // // $ saved_model_cli show --dir $EXPORT_DIR --tag_set serve --signature_def serving_default final String xName = "Placeholder:0"; final String scoresName = "dnn/head/predictions/probabilities:0"; List<Tensor> outputs = session.runner() .feed(xName, x) .fetch(scoresName) .run(); // Outer dimension is batch size; inner dimension is number of classes float[][] scores = new float[2][3]; outputs.get(0).copyTo(scores); System.out.println(Arrays.deepToString(scores)); } }
C ++ Client
Most likely, you will want to use tensorflow::LoadSavedModel with Session .
#include <unordered_set> #include <utility> #include <vector> #include "tensorflow/cc/saved_model/loader.h" #include "tensorflow/core/framework/tensor.h" #include "tensorflow/core/public/session.h" namespace tf = tensorflow; int main(int argc, char** argv) { const string export_dir = argv[1]; tf::SavedModelBundle bundle; tf::Status load_status = tf::LoadSavedModel( tf::SessionOptions(), tf::RunOptions(), export_dir, {"serve"}, &bundle); if (!load_status.ok()) { std::cout << "Error loading model: " << load_status << std::endl; return -1; } // We should get the signature out of MetaGraphDef, but that a bit // involved. We'll take a shortcut like we did in the Java example. const string x_name = "Placeholder:0"; const string scores_name = "dnn/head/predictions/probabilities:0"; auto x = tf::Tensor(tf::DT_FLOAT, tf::TensorShape({2, 4})); auto matrix = x.matrix<float>(); matrix(0, 0) = 6.4; matrix(0, 1) = 3.2; matrix(0, 2) = 4.5; matrix(0, 3) = 1.5; matrix(0, 1) = 5.8; matrix(0, 2) = 3.1; matrix(0, 3) = 5.0; matrix(0, 4) = 1.7; std::vector<std::pair<string, tf::Tensor>> inputs = {{x_name, x}}; std::vector<tf::Tensor> outputs; tf::Status run_status = bundle.session->Run(inputs, {scores_name}, {}, &outputs); if (!run_status.ok()) { cout << "Error running session: " << run_status << std::endl; return -1; } for (const auto& tensor : outputs) { std::cout << tensor.matrix<float>() << std::endl; } }
Case Study 3: Submitting a Model Using the TensorFlow Service
Exporting models in a manner suitable for servicing the Classification Model requires the input to be tf.Example . Here we can export the model for TensorFlow service:
def serving_input_receiver_fn(): """Build the serving inputs."""
To refer to the documentation of TensorFlow Servers for additional information on how to configure the TensorFlow service, I have provided here only the client code:
Note that the examples key referenced in predict_request.inputs must match the key used in serving_input_receiver_fn during export (see the constructor in ServingInputReceiver in this code).
Appendix: Exporting from Canned Models to TF 1.3
In TensorFlow 1.3, an error appears in which canned models are not correctly exported for use in case 2 (the problem does not exist for "user" ratings). Here is a workaround that wraps the DNNClassifier to make everything work, in particular for the Iris example:
# Build 3 layer DNN with 10, 20, 10 units respectively. class Wrapper(tf.estimator.Estimator): def __init__(self, **kwargs): dnn = tf.estimator.DNNClassifier(**kwargs) def model_fn(mode, features, labels): spec = dnn._call_model_fn(features, labels, mode) export_outputs = None if spec.export_outputs: export_outputs = { "serving_default": tf.estimator.export.PredictOutput( {"scores": spec.export_outputs["serving_default"].scores, "classes": spec.export_outputs["serving_default"].classes})}