Skip to content

Lockdown Microsoft.ML.TensorFlow public surface #2312

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Feb 1, 2019
Merged

Lockdown Microsoft.ML.TensorFlow public surface #2312

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
e3a8858
make constructors internal
ganik Jan 29, 2019
ab27f34
make internals visible to consuming assembly
ganik Jan 29, 2019
e49b1d8
Merge pull request #1 from dotnet/master
ganik Jan 29, 2019
babb484
internals visible to tests
ganik Jan 29, 2019
d1abbb8
merge
ganik Jan 29, 2019
b14d899
rename Arguments to Options
ganik Jan 31, 2019
d3dfb06
Merge pull request #2 from dotnet/master
ganik Jan 31, 2019
8dcda29
Add comment
ganik Jan 31, 2019
b9a24ff
merge
ganik Jan 31, 2019
d6dbea0
fix test
ganik Feb 1, 2019
3190f63
fix comment
ganik Feb 1, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions src/Microsoft.ML.TensorFlow/AssemblyInfo.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using System.Runtime.CompilerServices;
using Microsoft.ML;

[assembly: InternalsVisibleTo(assemblyName: "Microsoft.ML.Tests" + PublicKey.TestValue)]
[assembly: InternalsVisibleTo(assemblyName: "Microsoft.ML.TensorFlow.StaticPipe" + PublicKey.Value)]
14 changes: 7 additions & 7 deletions src/Microsoft.ML.TensorFlow/TensorflowCatalog.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,23 +45,23 @@ public static TensorFlowEstimator ScoreTensorFlowModel(this TransformsCatalog ca
=> new TensorFlowEstimator(CatalogUtils.GetEnvironment(catalog), outputColumnNames, inputColumnNames, tensorFlowModel);

/// <summary>
/// Score or Retrain a tensorflow model (based on setting of the <see cref="TensorFlowTransformer.Arguments.ReTrain"/>) setting.
/// The model is specified in the <see cref="TensorFlowTransformer.Arguments.ModelLocation"/>.
/// Score or Retrain a tensorflow model (based on setting of the <see cref="TensorFlowTransformer.Options.ReTrain"/>) setting.
/// The model is specified in the <see cref="TensorFlowTransformer.Options.ModelLocation"/>.
Copy link
Contributor

@Ivanidzo4ka Ivanidzo4ka Feb 1, 2019 •
edited by ganik
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Options [](start = 75, length = 7)

Please rerun locally RegenerateEntryPointCatalog and upload changes in core_ep-list.tsv to this PR. #Resolved

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

In reply to: 253190931 [](ancestors = 253190931)

/// </summary>
/// <param name="catalog">The transform's catalog.</param>
/// <param name="args">The <see cref="TensorFlowTransformer.Arguments"/> specifying the inputs and the settings of the <see cref="TensorFlowEstimator"/>.</param>
/// <param name="args">The <see cref="TensorFlowTransformer.Options"/> specifying the inputs and the settings of the <see cref="TensorFlowEstimator"/>.</param>
public static TensorFlowEstimator TensorFlow(this TransformsCatalog catalog,
TensorFlowTransformer.Arguments args)
TensorFlowTransformer.Options args)
=> new TensorFlowEstimator(CatalogUtils.GetEnvironment(catalog), args);

/// <summary>
/// Scores or retrains (based on setting of the <see cref="TensorFlowTransformer.Arguments.ReTrain"/>) a pre-traiend TensorFlow model specified via <paramref name="tensorFlowModel"/>.
/// Scores or retrains (based on setting of the <see cref="TensorFlowTransformer.Options.ReTrain"/>) a pre-traiend TensorFlow model specified via <paramref name="tensorFlowModel"/>.
/// </summary>
/// <param name="catalog">The transform's catalog.</param>
/// <param name="args">The <see cref="TensorFlowTransformer.Arguments"/> specifying the inputs and the settings of the <see cref="TensorFlowEstimator"/>.</param>
/// <param name="args">The <see cref="TensorFlowTransformer.Options"/> specifying the inputs and the settings of the <see cref="TensorFlowEstimator"/>.</param>
/// <param name="tensorFlowModel">The pre-trained TensorFlow model.</param>
public static TensorFlowEstimator TensorFlow(this TransformsCatalog catalog,
TensorFlowTransformer.Arguments args,
TensorFlowTransformer.Options args,
TensorFlowModelInfo tensorFlowModel)
=> new TensorFlowEstimator(CatalogUtils.GetEnvironment(catalog), args, tensorFlowModel);
}
Expand Down
41 changes: 24 additions & 17 deletions src/Microsoft.ML.TensorFlow/TensorflowTransform.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
using Microsoft.ML.Transforms.TensorFlow;

[assembly: LoadableClass(TensorFlowTransformer.Summary, typeof(IDataTransform), typeof(TensorFlowTransformer),
typeof(TensorFlowTransformer.Arguments), typeof(SignatureDataTransform), TensorFlowTransformer.UserName, TensorFlowTransformer.ShortName)]
typeof(TensorFlowTransformer.Options), typeof(SignatureDataTransform), TensorFlowTransformer.UserName, TensorFlowTransformer.ShortName)]

[assembly: LoadableClass(TensorFlowTransformer.Summary, typeof(IDataTransform), typeof(TensorFlowTransformer), null, typeof(SignatureLoadDataTransform),
TensorFlowTransformer.UserName, TensorFlowTransformer.LoaderSignature)]
Expand All @@ -37,7 +37,7 @@ namespace Microsoft.ML.Transforms
/// <include file='doc.xml' path='doc/members/member[@name="TensorflowTransformer"]/*' />
public sealed class TensorFlowTransformer : RowToRowTransformerBase
{
public sealed class Arguments : TransformInputBase
public sealed class Options : TransformInputBase
{
/// <summary>
/// Location of the TensorFlow model.
Expand Down Expand Up @@ -297,7 +297,7 @@ private static TensorFlowTransformer Create(IHostEnvironment env, ModelLoadConte
}

// Factory method for SignatureDataTransform.
internal static IDataTransform Create(IHostEnvironment env, Arguments args, IDataView input)
internal static IDataTransform Create(IHostEnvironment env, Options args, IDataView input)
{
Contracts.CheckValue(env, nameof(env));
env.CheckValue(args, nameof(args));
Expand All @@ -308,12 +308,12 @@ internal static IDataTransform Create(IHostEnvironment env, Arguments args, IDat
return new TensorFlowTransformer(env, args, input).MakeDataTransform(input);
}

internal TensorFlowTransformer(IHostEnvironment env, Arguments args, IDataView input)
internal TensorFlowTransformer(IHostEnvironment env, Options args, IDataView input)
: this(env, args, TensorFlowUtils.LoadTensorFlowModel(env, args.ModelLocation), input)
{
}

internal TensorFlowTransformer(IHostEnvironment env, Arguments args, TensorFlowModelInfo tensorFlowModel, IDataView input)
internal TensorFlowTransformer(IHostEnvironment env, Options args, TensorFlowModelInfo tensorFlowModel, IDataView input)
: this(env, tensorFlowModel.Session, args.OutputColumns, args.InputColumns, TensorFlowUtils.IsSavedModel(env, args.ModelLocation) ? args.ModelLocation : null, false)
{

Expand All @@ -332,7 +332,7 @@ internal TensorFlowTransformer(IHostEnvironment env, Arguments args, TensorFlowM
}
}

private void CheckTrainingParameters(Arguments args)
private void CheckTrainingParameters(Options args)
{
Host.CheckNonWhiteSpace(args.LabelColumn, nameof(args.LabelColumn));
Host.CheckNonWhiteSpace(args.OptimizationOperation, nameof(args.OptimizationOperation));
Expand Down Expand Up @@ -401,7 +401,7 @@ private void CheckTrainingParameters(Arguments args)
return (inputColIndex, isInputVector, tfInputType, tfInputShape);
}

private void TrainCore(Arguments args, IDataView input)
private void TrainCore(Options args, IDataView input)
{
var inputsForTraining = new string[Inputs.Length + 1];
var inputColIndices = new int[inputsForTraining.Length];
Expand Down Expand Up @@ -479,7 +479,7 @@ private void TrainCore(Arguments args, IDataView input)
string[] inputsForTraining,
ITensorValueGetter[] srcTensorGetters,
List<string> fetchList,
Arguments args)
Options args)
{
float loss = 0;
float metric = 0;
Expand Down Expand Up @@ -509,7 +509,7 @@ private void TrainCore(Arguments args, IDataView input)
/// After retraining Session and Graphs are both up-to-date
/// However model on disk is not which is used to serialzed to ML.Net stream
/// </summary>
private void UpdateModelOnDisk(string modelDir, Arguments args)
private void UpdateModelOnDisk(string modelDir, Options args)
{
try
{
Expand Down Expand Up @@ -957,7 +957,7 @@ protected override Schema.DetachedColumn[] GetOutputColumnsCore()
Desc = Summary,
UserName = UserName,
ShortName = ShortName)]
internal static CommonOutputs.TransformOutput TensorFlowScorer(IHostEnvironment env, Arguments input)
internal static CommonOutputs.TransformOutput TensorFlowScorer(IHostEnvironment env, Options input)
{
Contracts.CheckValue(env, nameof(env));
env.CheckValue(input, nameof(input));
Expand Down Expand Up @@ -1080,28 +1080,28 @@ public TFTensor GetBufferedBatchTensor()
public sealed class TensorFlowEstimator : IEstimator<TensorFlowTransformer>
{
private readonly IHost _host;
private readonly TensorFlowTransformer.Arguments _args;
private readonly TensorFlowTransformer.Options _args;
private readonly TensorFlowModelInfo _tensorFlowModel;
private readonly TFDataType[] _tfInputTypes;
private readonly ColumnType[] _outputTypes;
private TensorFlowTransformer _transformer;

public TensorFlowEstimator(IHostEnvironment env, string[] outputColumnNames, string[] inputColumnNames, string modelLocation)
internal TensorFlowEstimator(IHostEnvironment env, string[] outputColumnNames, string[] inputColumnNames, string modelLocation)
: this(env, outputColumnNames, inputColumnNames, TensorFlowUtils.LoadTensorFlowModel(env, modelLocation))
{
}

public TensorFlowEstimator(IHostEnvironment env, string[] outputColumnNames, string[] inputColumnNames, TensorFlowModelInfo tensorFlowModel)
internal TensorFlowEstimator(IHostEnvironment env, string[] outputColumnNames, string[] inputColumnNames, TensorFlowModelInfo tensorFlowModel)
: this(env, CreateArguments(tensorFlowModel, outputColumnNames, inputColumnNames), tensorFlowModel)
{
}

public TensorFlowEstimator(IHostEnvironment env, TensorFlowTransformer.Arguments args)
internal TensorFlowEstimator(IHostEnvironment env, TensorFlowTransformer.Options args)
: this(env, args, TensorFlowUtils.LoadTensorFlowModel(env, args.ModelLocation))
{
}

public TensorFlowEstimator(IHostEnvironment env, TensorFlowTransformer.Arguments args, TensorFlowModelInfo tensorFlowModel)
internal TensorFlowEstimator(IHostEnvironment env, TensorFlowTransformer.Options args, TensorFlowModelInfo tensorFlowModel)
{
_host = Contracts.CheckRef(env, nameof(env)).Register(nameof(TensorFlowEstimator));
_args = args;
Expand All @@ -1112,15 +1112,19 @@ public TensorFlowEstimator(IHostEnvironment env, TensorFlowTransformer.Arguments
_outputTypes = outputTuple.outputTypes;
}

private static TensorFlowTransformer.Arguments CreateArguments(TensorFlowModelInfo tensorFlowModel, string[] outputColumnNames, string[] inputColumnName)
private static TensorFlowTransformer.Options CreateArguments(TensorFlowModelInfo tensorFlowModel, string[] outputColumnNames, string[] inputColumnName)
{
var args = new TensorFlowTransformer.Arguments();
var args = new TensorFlowTransformer.Options();
args.ModelLocation = tensorFlowModel.ModelPath;
args.InputColumns = inputColumnName;
args.OutputColumns = outputColumnNames;
args.ReTrain = false;
return args;
}

/// <summary>
/// Returns the output schema shape of the estimator, if the input schema shape is like the one provided.
/// </summary>
public SchemaShape GetOutputSchema(SchemaShape inputSchema)
{
_host.CheckValue(inputSchema, nameof(inputSchema));
Expand All @@ -1146,6 +1150,9 @@ public SchemaShape GetOutputSchema(SchemaShape inputSchema)
return new SchemaShape(resultDic.Values);
}

/// <summary>
/// Train and return a transformer.
/// </summary>
public TensorFlowTransformer Fit(IDataView input)
{
_host.CheckValue(input, nameof(input));
Expand Down
2 changes: 1 addition & 1 deletion test/BaselineOutput/Common/EntryPoints/core_ep-list.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ Transforms.ScoreColumnSelector Selects only the last score columns and the extra
Transforms.Scorer Turn the predictor model into a transform model Microsoft.ML.EntryPoints.ScoreModel MakeScoringTransform Microsoft.ML.EntryPoints.ScoreModel+ModelInput Microsoft.ML.EntryPoints.ScoreModel+Output
Transforms.Segregator Un-groups vector columns into sequences of rows, inverse of Group transform Microsoft.ML.Transforms.GroupingOperations Ungroup Microsoft.ML.Transforms.UngroupTransform+Arguments Microsoft.ML.EntryPoints.CommonOutputs+TransformOutput
Transforms.SentimentAnalyzer Uses a pretrained sentiment model to score input strings Microsoft.ML.Transforms.Text.TextAnalytics AnalyzeSentiment Microsoft.ML.Transforms.Text.SentimentAnalyzingTransformer+Arguments Microsoft.ML.EntryPoints.CommonOutputs+TransformOutput
Transforms.TensorFlowScorer Transforms the data using the TensorFlow model. Microsoft.ML.Transforms.TensorFlowTransformer TensorFlowScorer Microsoft.ML.Transforms.TensorFlowTransformer+Arguments Microsoft.ML.EntryPoints.CommonOutputs+TransformOutput
Transforms.TensorFlowScorer Transforms the data using the TensorFlow model. Microsoft.ML.Transforms.TensorFlowTransformer TensorFlowScorer Microsoft.ML.Transforms.TensorFlowTransformer+Options Microsoft.ML.EntryPoints.CommonOutputs+TransformOutput
Transforms.TextFeaturizer A transform that turns a collection of text documents into numerical feature vectors. The feature vectors are normalized counts of (word and/or character) ngrams in a given tokenized text. Microsoft.ML.Transforms.Text.TextAnalytics TextTransform Microsoft.ML.Transforms.Text.TextFeaturizingEstimator+Arguments Microsoft.ML.EntryPoints.CommonOutputs+TransformOutput
Transforms.TextToKeyConverter Converts input values (words, numbers, etc.) to index in a dictionary. Microsoft.ML.Transforms.Categorical.Categorical TextToKey Microsoft.ML.Transforms.Conversions.ValueToKeyMappingTransformer+Arguments Microsoft.ML.EntryPoints.CommonOutputs+TransformOutput
Transforms.TrainTestDatasetSplitter Split the dataset into train and test sets Microsoft.ML.EntryPoints.TrainTestSplit Split Microsoft.ML.EntryPoints.TrainTestSplit+Input Microsoft.ML.EntryPoints.TrainTestSplit+Output
Expand Down
4 changes: 2 additions & 2 deletions test/Microsoft.ML.Tests/ScenariosWithDirectInstantiation/TensorflowTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -433,7 +433,7 @@ public void TensorFlowTransformMNISTLRTrainingTest()

var pipe = mlContext.Transforms.Categorical.OneHotEncoding("OneHotLabel", "Label")
.Append(mlContext.Transforms.Normalize(new NormalizingEstimator.MinMaxColumn("Features", "Placeholder")))
.Append(new TensorFlowEstimator(mlContext, new TensorFlowTransformer.Arguments()
.Append(mlContext.Transforms.TensorFlow(new TensorFlowTransformer.Options()
{
ModelLocation = model_location,
InputColumns = new[] { "Features" },
Expand Down Expand Up @@ -547,7 +547,7 @@ private void ExecuteTFTransformMNISTConvTrainingTest(bool shuffle, int? shuffleS
}

var pipe = mlContext.Transforms.CopyColumns(("Features", "Placeholder"))
.Append(new TensorFlowEstimator(mlContext, new TensorFlowTransformer.Arguments()
.Append(new TensorFlowEstimator(mlContext, new TensorFlowTransformer.Options()
{
ModelLocation = modelLocation,
InputColumns = new[] { "Features" },
Expand Down