Passing serialized TensorFlow Example to TF Serving SavedModel

[rs22]

System information

• OS version/distro: macOS 10.15
• .NET Version (eg., dotnet --info): 3.1.301

Issue

• What did you do?
  I would like to use the PredictionEnginePool (eventually) in combination with a pretrained Tensorflow Model that I exported using the Estimator.export_saved_model function in combination with build_parsing_serving_input_receiver_fn.

  Specifically, I went through this tutorial: https://www.tensorflow.org/tfx/tutorials/transform/census. Below, you can find the Tensorflow Serving signature definition according to saved_model_cli.

• What happened?
  The input_example_tensor input expects a serialized Example message (a binary buffer, not a text string). This does not work using the ML.NET library because it re-encodes the data that I'm providing as the model input.

• What did you expect?
  There should be the option in ML.NET to pass raw binary data as a TFString to the model (maybe as a byte[] or ReadOnlyMemory<byte>?).

Source code / logs

Saved model signature:

$ saved_model_cli show --dir ./my_saved_model --tag_set serve --signature_def serving_default
The given SavedModel SignatureDef contains the following input(s):
  inputs['inputs'] tensor_info:
      dtype: DT_STRING
      shape: (-1)
      name: input_example_tensor:0
The given SavedModel SignatureDef contains the following output(s):
  outputs['classes'] tensor_info:
      dtype: DT_STRING
      shape: (-1, 2)
      name: head/Tile:0
  outputs['scores'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 2)
      name: head/predictions/probabilities:0
Method name is: tensorflow/serving/classify

My code:

class ModelInput {
    [ColumnName("input_example_tensor"), VectorType(1)]
    public string[] InputExampleTensor { get; set; }
}
class ModelPrediction {
    [ColumnName("head/Tile:0"), VectorType(2)]
    public string[] Classes { get; set; }

    [ColumnName("head/predictions/probabilities:0"), VectorType(2)]
    public float[] Prediction { get; set; }
}

var mlContext = new MLContext();

var pipeline = mlContext.Model.LoadTensorFlowModel("my_saved_model")
    .ScoreTensorFlowModel(
        outputColumnNames: new[] { "head/Tile:0", "head/predictions/probabilities:0" },
        inputColumnNames: new[] { "input_example_tensor" }
    );

// Train the model
// Since we are simply using a pre-trained TensorFlow model,
// we can "train" it against an empty dataset
var emptyTrainingSet = mlContext.Data.LoadFromEnumerable(new List<ModelInput>());
var mlModel = pipeline.Fit(emptyTrainingSet);
var engine = mlContext.Model.CreatePredictionEngine<ModelInput, ModelPrediction>(mlModel);

// Example is a Protobuf-Class, generated from example.proto
var example = new Example();
// filling the example with features omitted

var input = new ModelInput {
    InputExampleTensor = new[] { new string(example.ToByteArray().Select(x => (char)x).ToArray()) }
};

var prediction = engine.Predict(input);

Which fails with:

W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at example_parsing_ops.cc:92 : Invalid argument: Could not parse example input, value: '<omitted binary data>'

[harishsk]

Hi @rs22 The CastDataAndReturnAsTensor function you point to above also supports byte tensors.

Can you change your ModelInput class to have InputExampleTensor to be a byte[] instead of string[]?

Can you please try that let me know? And can you please share your my_saved_model file so I can try the repro here?

[rs22]

Thanks for your support with my issue!

I've tried setting the type of InputExampleTensor to byte[] as you suggested, but the Predict call now fails with the following exception:

Unhandled exception. System.ArgumentOutOfRangeException: Schema mismatch for input column 'input_example_tensor': expected String, got Vector<Byte, 1> (Parameter 'inputSchema')
   at Microsoft.ML.Transforms.TensorFlowTransformer.Mapper..ctor(TensorFlowTransformer parent, DataViewSchema inputSchema)

I've attached a cleaned-up version of my pretrained model that you can use for reproducing the problem.

To (temporarily) make the prediction work, you'll need to create a TFExample protobuf message using this generated source file, the Google.Protobuf library and this code:

Func<float, Feature> makeFeature = (float x) =>
{
    var floatList = new FloatList();
    floatList.Value.Add(x);
    return new Feature { FloatList = floatList };
};
var example = new Example {Features = new Features()};
example.Features.Feature.Add("my_feature", makeFeature(0));
var engine = mlContext.Model.CreatePredictionEngine<ModelInput, ModelPrediction>(mlModel);
engine.Predict(new ModelInput { InputExampleTensor = new[] { new string(example.ToByteArray().Select(x => (char)x).ToArray()) } });

... and change the encoding-related line in CastDataAndReturnAsTensor to something like:

bytes[i] = ((ReadOnlyMemory<char>)(object)data[i]).ToArray().Select(x => (byte)x).ToArray();

[jsgonsette]

As far as I understand this issue is related to the usage of a model trained in TensorFlow and exported into some '.pb' file. A recipe showing a concrete example from the export in TensorFlow to the usage of that model in ML.Net would be extremely useful as interaction between both sides seems really tricky, up to now all my attempts have been unsuccessful. However, I'm sure that the possibility to serve a trained model on other platforms like dot.net would be very interesting for a lot of people. But for now, I could not find any helpful information regarding that matter.

[harishsk]

@mstfbl Can you please take a look at this?

[rs22]

@jsgonsette I think you're right: The TensorFlow documentation on these topics is really sparse and I've spent quite some time to figure out what the Estimator-SavedModel expects as an input. I've written down my findings here -- maybe this is helpful to you.

[jsgonsette]

Hello Robert, Many thanks for your investigation work. Your page with your findings sounds great. I'm trying that asap. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Garanti sans virus. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> Le dim. 6 sept. 2020 à 16:17, Robert Schmid <[email protected]> a écrit :

…

@jsgonsette <https://github.com/jsgonsette> I think you're right: The TensorFlow documentation on these topics is really sparse and I've spent quite some time to figure out what the Estimator-SavedModel expects as an input. I've written down my findings here <https://www.cyberwelt.io/using-a-tf-serving-savedmodel-from-net/> -- maybe this is helpful to you. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5336 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIGPSHWLPF7QPCB4NT6ZFWLSEOKXZANCNFSM4PTE5UTQ> .

[frank-dong-ms-zz]

@rs22 Looks like you are trying to load a pb model from ML.NET. Note ML.NET only allows load a frozen tensorflow model as below:
https://devblogs.microsoft.com/cesardelatorre/run-with-ml-net-c-code-a-tensorflow-model-exported-from-azure-cognitive-services-custom-vision/

Is your model frozen or not? Could you please try to frozen your model first if it is not already frozen? This is post on how to convert model into frozen model file:
https://www.dlology.com/blog/how-to-convert-trained-keras-model-to-tensorflow-and-make-prediction/

[rs22]

I wasn't really aware of the concept of frozen saved models, so thanks for this pointer! Maybe it would be good to show a warning/throw an exception when users try to load 'unfrozen' saved models (you can't tell them apart from the .pb file extension).

In my case, there was in fact a variables folder next to the .pb file, but I think that the graph exported by the Estimator is effectively frozen. Anyways, I created a new .pb file using freeze_graph.py:

python /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/tools/freeze_graph.py \
  --input_saved_model_dir ./my_saved_model \
  --output_node_names head/predictions/probabilities,head/Tile \
  --output_graph frozen_model.pb

Using the frozen_model.pb, I am still getting the exception. For your reference I've attached a complete Visual Studio project with the .pb files: tfrepro.zip

[frank-dong-ms-zz]

@rs22 Thanks for providing repro project and model file. This issue is basically an encoding issue.
example class instance is first converted to protobuf byte array as below, then you try to convert this byte array to char array and then string to cooperate with ML.NET, then ML.NET will try to convert the string back to byte array and pass to tensorflow.net:

So the problem here is how to do convert [byte array] -> [string] -> [byte array] properly to keep complete information from byte array. There are some ways to do that but most recommended way is using same encoding to do the conversion, in "CastDataAndReturnAsTensor" method from ML.NET we are using below to do the [string] -> [byte array] conversion as below which means you need to use same encoding when do [byte array] -> [string]:
bytes[i] = Encoding.UTF8.GetBytes(((ReadOnlyMemory<char>)(object)data[i]).ToArray());
so below way to do the [byte array] -> [string] conversion is problematic:
InputExampleTensor = new[] { new string(example.ToByteArray().Select(x => (char)x).ToArray()) }

I tried to use same encoding when convert [byte array] -> [string] using UTF8 encoding as below but the problem here is UTF8 is not working properly when you byte string has byte whose value is larger than 127(example.ToByteArray()[24] is 128 which cause issue that we can't convert back to exact same byte array):
InputExampleTensor = new[] { Encoding.UTF8.GetString(example.ToByteArray()) }

So, one way to fix that issue is we use both reliable encoding on both [byte array] -> [string] and [string] -> [byte array] conversion like below:
InputExampleTensor = new[] { Encoding.Unicode.GetString(example.ToByteArray()) }
and
bytes[i] = Encoding.Unicode.GetBytes(((ReadOnlyMemory<char>)(object)data[i]).ToArray());
I have do some test that this is working as expected.

However, recently we upgraded tensorflow.net version in below PR and some old API to create Tensor as byte[][] is no longer exist so I need some more time to figure out an workable solution: #5404

Will update in this issue if I find an workable solution for this issue in new tensorflow.net version.

[frank-dong-ms-zz]

Opened below PR to ask advice from Tensorflow team: tensorflow/tensorflow#44225

[frank-dong-ms-zz]

@rs22 I found a workable solution is that we using Unicode to encode example byte array and that will require a change in ML.NET side.

[frank-dong-ms-zz]

@rs22 sorry, looks like the default encoding for tensorflow is UTF8 so I can't use Unicode here: https://www.tensorflow.org/tutorials/load_data/unicode

So I think the problem now is that your model is using protobuf encoded byte array of example object as input which is not compatible with UTF8. Since we can't change the encoding and ML.NET don't support byte array directly with variance length, what you can do is use serialized string of example object (use method ToText) as input and you need change your model little bit to do de-serialize on input string.

cc @harishsk see if Harish has other suggestions.

[rs22]

Thanks for your investigations! The reason why I'm trying to use a Protobuf as the model input is to maintain compatibility with TF Serving -- which I would lose when I change the model input signature...

[frank-dong-ms-zz]

@rs22 I see, this looks like new feature request to me now, from our current design and implementation looks like we have no way to tell whether user passed in pure string that can be passed to TF.NET directly or an encoded string that need to be decode to byte array and pass to TF.NET. I will mark this as feature request.