Create model file V1 scenario tests #2899

rogancarr · 2019-03-09T01:23:55Z

As laid out in #2498 , we need scenarios to cover the Model Files functionality we want fully supported in V1.

This PR adds tests for the following scenarios:

I can train a model and save it as a file. This model includes the learner as well as the transforms
I can use a model file in a completely different process to make predictions
I can easily figure out which NuGets (and versions) I need to score an ML.NET model
I can export ML.NET models to ONNX (limited to the existing internal functionality)

codecov · 2019-03-09T02:12:51Z

Codecov Report

Merging #2899 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #2899      +/-   ##
==========================================
- Coverage   72.53%   72.52%   -0.01%     
==========================================
  Files         805      806       +1     
  Lines      144243   144500     +257     
  Branches    16175    16191      +16     
==========================================
+ Hits       104620   104800     +180     
- Misses      35222    35292      +70     
- Partials     4401     4408       +7

Flag	Coverage Δ
#Debug	`72.52% <100%> (-0.01%)`	⬇️
#production	`68.13% <ø> (-0.02%)`	⬇️
#test	`88.78% <100%> (+0.02%)`	⬆️

Impacted Files	Coverage Δ
...soft.ML.Functional.Tests/Datasets/CommonColumns.cs	`100% <ø> (ø)`	⬆️
test/Microsoft.ML.Functional.Tests/ModelFiles.cs	`96.07% <100%> (ø)`
...crosoft.ML.StaticPipe/EvaluatorStaticExtensions.cs	`86.15% <0%> (-13.85%)`	⬇️
.../Microsoft.ML.Data/Model/ModelOperationsCatalog.cs	`88.42% <0%> (-3.31%)`	⬇️
...ft.ML.Data/Evaluators/BinaryClassifierEvaluator.cs	`77.19% <0%> (-2.09%)`	⬇️
src/Microsoft.ML.Data/TrainCatalog.cs	`82.91% <0%> (-1.28%)`	⬇️
...soft.ML.Data/DataLoadSave/DataOperationsCatalog.cs	`72.92% <0%> (-0.32%)`	⬇️
...StandardTrainers/Standard/LinearModelParameters.cs	`60.05% <0%> (-0.27%)`	⬇️
...soft.ML.Data/DataLoadSave/Text/TextLoaderCursor.cs	`84.7% <0%> (-0.21%)`	⬇️
...soft.ML.Data/DataView/DataViewConstructionUtils.cs	`85.09% <0%> (-0.18%)`	⬇️
... and 21 more

rogancarr · 2019-03-12T01:16:22Z

Waiting for #2858 to be checked in; will incorporate those tests.

sfilipi · 2019-03-14T18:36:11Z

test/Microsoft.ML.Functional.Tests/Datasets/ScoreColumn.cs

+    internal sealed class ScoreColumn
+    {
+        public float Score { get; set; }
+    }


can this go somewhere, not on a separate file? Where the other data models are? There might even be one already. #Resolved

Yes. I've consolidated all the single-column classes into one file. I had been following the Apache REEF style by habit.

In reply to: 265712606 [](ancestors = 265712606)

sfilipi · 2019-03-14T18:36:51Z

test/Microsoft.ML.Functional.Tests/ModelFiles.cs

+        public void DetermineNugetVersionFromModel()
+        {
+            var modelFile = GetDataPath(@"backcompat" + Path.DirectorySeparatorChar + @"keep-model.zip");
+            var versionFileName = @"TrainingInfo\Version.txt"; // Can't find this cross plat.


\ [](start = 48, length = 1)

Path.DirectorySeparatorChar? #Resolved

Strangely enough, since the zip archive was made on Windows, we need to specify a '\' here or the tests will fail on Linux and Mac. I've updated this cryptic comment to explain it better.

In reply to: 265712860 [](ancestors = 265712860)

sfilipi · 2019-03-14T18:37:45Z

test/Microsoft.ML.Functional.Tests/ModelFiles.cs

+        [Fact]
+        public void DetermineNugetVersionFromModel()
+        {
+            var modelFile = GetDataPath(@"backcompat" + Path.DirectorySeparatorChar + @"keep-model.zip");


@ [](start = 40, length = 1)

don't think you need the @

you could also go with interpoaltion:

$"backcompat{Path.DirectorySeparatorChar}keep-model.zip" #Resolved

yaeldekel · 2019-03-22T21:57:17Z

test/Microsoft.ML.Functional.Tests/ModelFiles.cs

+
+            var modelPath = DeleteOutputPath("fitPipelineSaveModelAndPredict.zip");
+            // Save model to a file.
+            using (var file = File.Create(modelPath))


Create [](start = 35, length = 6)

There are overloads that take the file path, would this be an appropriate place to use these? #Resolved

yaeldekel · 2019-03-22T22:04:20Z

test/Microsoft.ML.Functional.Tests/Datasets/CommonColumns.cs

    /// </summary>
    internal sealed class FeatureContributionOutput
    {
        public float[] FeatureContributions { get; set; }
    }

    /// <summary>
-    /// A class to hold the Score column.
+    /// A class to hold a feature column.


feature [](start = 26, length = 7)

Is it a feature column or a score column? #Resolved

yaeldekel · 2019-03-22T22:05:59Z

test/Microsoft.ML.Functional.Tests/ModelFiles.cs

+            // Load model from a file.
+            ITransformer serializedModel;
+            using (var file = File.OpenRead(modelPath))
+                serializedModel = mlContext.Model.Load(file, out var serializedSchema);


serializedSchema [](start = 69, length = 16)

You can verify that this is the same as data.Schema. #Resolved

yaeldekel

shmoradims · 2019-03-22T23:30:07Z

test/Microsoft.ML.Functional.Tests/ModelFiles.cs

@@ -30,6 +32,84 @@ private class InputData
            public float[] Features { get; set; }
        }

+        /// <summary>
+        /// Model Files: The (minimum) nuget version can be found in the model file.


model file [](start = 73, length = 10)

Is this an old model file or a current one? If it's current, shouldn't we create the model on the fly (similar to the other scenario below) instead of reading a static keep-model.zip file?

sfilipi · 2019-03-25T17:20:18Z

test/Microsoft.ML.Functional.Tests/ModelFiles.cs

+        ///  1. I can train a model and save it to a file, including transforms.
+        ///  2. Training and prediction happen in different processes (or even different machines). 
+        ///     The actual test will not run in different processes, but will simulate the idea that the 
+        ///     "communication pipe" is just a serialized model of some form.


not necessary for tests files, but as FYI lists need to be in xml style as well.

sfilipi

Adding tests for model files.

2de024f

Rogan Carr added 2 commits March 8, 2019 20:46

fixing cross-plat directory separators.

90525f9

fixing cross-plat directory separators.

121ee96

sfilipi reviewed Mar 14, 2019

View reviewed changes

Rogan Carr added 7 commits March 14, 2019 15:40

Merge branch 'master' into 2896_model_file_scenarios

e79e8c7

Addressing PR comments.

9fb2873

Merge branch 'master' into 2896_model_file_scenarios

2a02d1d

In progress commit.

9ca8ddc

work in progress

e3b9930

Merge branch 'master' into 2896_model_file_scenarios

bc46b4e

Adding ModelFile tests to Loading, combining into one file.

982a3ba

rogancarr requested review from sfilipi and yaeldekel March 22, 2019 20:52

yaeldekel reviewed Mar 22, 2019

View reviewed changes

yaeldekel approved these changes Mar 22, 2019

View reviewed changes

Addressing PR comments.

5000dbf

shmoradims reviewed Mar 22, 2019

View reviewed changes

Changing model version checker to use a model created on the fly.

44ed95e

sfilipi reviewed Mar 25, 2019

View reviewed changes

sfilipi approved these changes Mar 25, 2019

View reviewed changes

Fixing cross-plat tests.

47518c5

rogancarr merged commit f342403 into dotnet:master Mar 25, 2019

rogancarr deleted the 2896_model_file_scenarios branch March 25, 2019 18:58

ghost locked as resolved and limited conversation to collaborators Mar 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create model file V1 scenario tests #2899

Create model file V1 scenario tests #2899

rogancarr commented Mar 9, 2019

codecov bot commented Mar 9, 2019 •

edited

Loading

rogancarr commented Mar 12, 2019

sfilipi Mar 14, 2019 •

edited by rogancarr

Loading

rogancarr Mar 14, 2019

sfilipi Mar 14, 2019 •

edited by rogancarr

Loading

rogancarr Mar 14, 2019 •

edited

Loading

sfilipi Mar 14, 2019 •

edited by rogancarr

Loading

yaeldekel Mar 22, 2019 •

edited by rogancarr

Loading

yaeldekel Mar 22, 2019 •

edited by rogancarr

Loading

yaeldekel Mar 22, 2019 •

edited by rogancarr

Loading

yaeldekel left a comment

shmoradims Mar 22, 2019

sfilipi Mar 25, 2019

sfilipi left a comment

Create model file V1 scenario tests #2899

Create model file V1 scenario tests #2899

Conversation

rogancarr commented Mar 9, 2019

codecov bot commented Mar 9, 2019 • edited Loading

Codecov Report

rogancarr commented Mar 12, 2019

sfilipi Mar 14, 2019 • edited by rogancarr Loading

Choose a reason for hiding this comment

rogancarr Mar 14, 2019

Choose a reason for hiding this comment

sfilipi Mar 14, 2019 • edited by rogancarr Loading

Choose a reason for hiding this comment

rogancarr Mar 14, 2019 • edited Loading

Choose a reason for hiding this comment

sfilipi Mar 14, 2019 • edited by rogancarr Loading

Choose a reason for hiding this comment

yaeldekel Mar 22, 2019 • edited by rogancarr Loading

Choose a reason for hiding this comment

yaeldekel Mar 22, 2019 • edited by rogancarr Loading

Choose a reason for hiding this comment

yaeldekel Mar 22, 2019 • edited by rogancarr Loading

Choose a reason for hiding this comment

yaeldekel left a comment

Choose a reason for hiding this comment

shmoradims Mar 22, 2019

Choose a reason for hiding this comment

sfilipi Mar 25, 2019

Choose a reason for hiding this comment

sfilipi left a comment

Choose a reason for hiding this comment

codecov bot commented Mar 9, 2019 •

edited

Loading

sfilipi Mar 14, 2019 •

edited by rogancarr

Loading

sfilipi Mar 14, 2019 •

edited by rogancarr

Loading

rogancarr Mar 14, 2019 •

edited

Loading

sfilipi Mar 14, 2019 •

edited by rogancarr

Loading

yaeldekel Mar 22, 2019 •

edited by rogancarr

Loading

yaeldekel Mar 22, 2019 •

edited by rogancarr

Loading

yaeldekel Mar 22, 2019 •

edited by rogancarr

Loading