Update Permutation Feature Importance Samples #3247

rogancarr · 2019-04-08T22:04:55Z

This PR updates the samples for PermutationFeatureImportance and adds samples for MulticlassClassification and Ranking tasks.

Fixes #3242

codecov · 2019-04-08T22:44:58Z

Codecov Report

Merging #3247 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3247      +/-   ##
==========================================
- Coverage   72.62%   72.62%   -0.01%     
==========================================
  Files         807      807              
  Lines      145080   145080              
  Branches    16213    16213              
==========================================
- Hits       105369   105368       -1     
  Misses      35294    35294              
- Partials     4417     4418       +1

Flag	Coverage Δ
#Debug	`72.62% <ø> (-0.01%)`	⬇️
#production	`68.17% <ø> (-0.01%)`	⬇️
#test	`88.93% <ø> (ø)`	⬆️

Impacted Files	Coverage Δ
...ansforms/PermutationFeatureImportanceExtensions.cs	`97.93% <ø> (ø)`	⬆️
...StandardTrainers/Standard/LinearModelParameters.cs	`60.05% <0%> (-0.27%)`	⬇️
...rc/Microsoft.ML.Transforms/CustomMappingCatalog.cs	`100% <0%> (ø)`	⬆️
src/Microsoft.ML.Transforms/ExtensionsCatalog.cs	`57.14% <0%> (ø)`	⬆️

Ivanidzo4ka · 2019-04-09T21:51:53Z

...s/Microsoft.ML.Samples/Dynamic/Trainers/BinaryClassification/PermutationFeatureImportance.cs

+using System.Collections.Generic;
+using System.Linq;
+
+namespace Microsoft.ML.Samples.Dynamic.Trainers.BinaryClassification


I think we tend to have namespace Samples.Dynamic #Resolved

Ivanidzo4ka · 2019-04-09T21:53:16Z

...s/Microsoft.ML.Samples/Dynamic/Trainers/BinaryClassification/PermutationFeatureImportance.cs

+            // Fit the pipeline to the data.
+            var model = pipeline.Fit(data);
+
+            // Compute the permutation metrics for the linear model using the normalized data.


// Compute the permutation metrics for the linear model using the normalized data. [](start = 12, length = 82)

put it before var permutationMetrics =? #Resolved

Ivanidzo4ka · 2019-04-09T21:54:17Z

...s/Microsoft.ML.Samples/Dynamic/Trainers/BinaryClassification/PermutationFeatureImportance.cs

+                    Data.FeatureColumns[i],
+                    linearPredictor.Model.SubModel.Weights[i],
+                    auc[i].Mean,
+                    1.96 * auc[i].StandardError);


1.96 * auc[i].StandardError [](start = 20, length = 27)

const number magic! I'm scared and confused! #Resolved

Magic number to convert the interval to the 95%-percentile.

In reply to: 273720232 [](ancestors = 273720232)

Ivanidzo4ka · 2019-04-09T21:55:41Z

docs/samples/Microsoft.ML.Samples/Dynamic/Trainers/Regression/PermutationFeatureImportance.cs

+            // Expected output:
+            //  Feature    Model Weight Change in RMSE  95% Confidence in the Mean Change in RMSE
+            //  Feature2        9.00        4.009       0.008304
+            //  Feature1        4.48        1.901       0.003351


I would align Expected output and Feature.
One space instead of two.
Same across all files. #ByDesign

Two spaces for printed verbatim output is the style across the Samples project.

In reply to: 273720618 [](ancestors = 273720618)

Ivanidzo4ka

artidoro · 2019-04-09T23:16:25Z

...s/Microsoft.ML.Samples/Dynamic/Trainers/BinaryClassification/PermutationFeatureImportance.cs

+
+            private static double Sigmoid(double x)
+            {
+                return 1.0 / (1.0 + Math.Exp(-1 * x));


If you could remove the braces, since it's just one line, and use =>

artidoro · 2019-04-09T23:20:25Z

...s/Microsoft.ML.Samples/Dynamic/Trainers/BinaryClassification/PermutationFeatureImportance.cs

+            public static readonly string[] FeatureColumns = new string[] { nameof(Feature1), nameof(Feature2) };
+
+            public static IEnumerable<Data> GenerateData(int nExamples = 10000,
+                double bias = 0, double weight1 = 1, double weight2 = 2, int seed = 1)


Could you add a one line explanation of how you are generating the data?

artidoro · 2019-04-09T23:23:57Z

...s/Microsoft.ML.Samples/Dynamic/Trainers/BinaryClassification/PermutationFeatureImportance.cs

+
+            public float Feature1 { get; set; }
+
+            public float Feature2 { get; set; }


Usually the class that defines the data does not do anything else. I like that pattern, because it makes it easy for users to understand what is happening.
I would suggest to move the GenerateData method outside of the Data class.

Similarly for Sigmoid.

In reply to: 273741337 [](ancestors = 273741337)

artidoro

Looks good, please do address the comments about the data generation code.

* Updating PFI Docs. (cherry picked from commit e0c029c)

Rogan Carr added 2 commits April 8, 2019 15:03

Updating PFI Docs.

becf7d6

Adding links to samples

abd6404

rogancarr requested review from sfilipi, artidoro, zeahmed and Ivanidzo4ka April 9, 2019 20:39

Ivanidzo4ka reviewed Apr 9, 2019

View reviewed changes

Rogan Carr added 2 commits April 9, 2019 15:02

Addressing PR comments.

628e812

Merge branch 'master' into 3242_PFI_Docs

5a35404

Ivanidzo4ka approved these changes Apr 9, 2019

View reviewed changes

artidoro reviewed Apr 9, 2019

View reviewed changes

artidoro approved these changes Apr 9, 2019

View reviewed changes

Addressing PR comments.

ee63a9a

rogancarr merged commit e0c029c into dotnet:master Apr 10, 2019

rogancarr deleted the 3242_PFI_Docs branch April 10, 2019 22:44

rogancarr added a commit to rogancarr/machinelearning that referenced this pull request Apr 11, 2019

Update Permutation Feature Importance Samples (dotnet#3247)

9f21c18

* Updating PFI Docs. (cherry picked from commit e0c029c)

ghost locked as resolved and limited conversation to collaborators Mar 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Permutation Feature Importance Samples #3247

Update Permutation Feature Importance Samples #3247

rogancarr commented Apr 8, 2019

codecov bot commented Apr 8, 2019 •

edited

Loading

Ivanidzo4ka Apr 9, 2019 •

edited by artidoro

Loading

Ivanidzo4ka Apr 9, 2019 •

edited by rogancarr

Loading

Ivanidzo4ka Apr 9, 2019 •

edited by rogancarr

Loading

rogancarr Apr 9, 2019

Ivanidzo4ka Apr 9, 2019 •

edited by rogancarr

Loading

rogancarr Apr 9, 2019

Ivanidzo4ka left a comment

artidoro Apr 9, 2019

artidoro Apr 9, 2019

artidoro Apr 9, 2019

artidoro Apr 9, 2019

artidoro left a comment •

edited

Loading


		public float Feature1 { get; set; }

		public float Feature2 { get; set; }

Update Permutation Feature Importance Samples #3247

Update Permutation Feature Importance Samples #3247

Conversation

rogancarr commented Apr 8, 2019

codecov bot commented Apr 8, 2019 • edited Loading

Codecov Report

Ivanidzo4ka Apr 9, 2019 • edited by artidoro Loading

Choose a reason for hiding this comment

Ivanidzo4ka Apr 9, 2019 • edited by rogancarr Loading

Choose a reason for hiding this comment

Ivanidzo4ka Apr 9, 2019 • edited by rogancarr Loading

Choose a reason for hiding this comment

rogancarr Apr 9, 2019

Choose a reason for hiding this comment

Ivanidzo4ka Apr 9, 2019 • edited by rogancarr Loading

Choose a reason for hiding this comment

rogancarr Apr 9, 2019

Choose a reason for hiding this comment

Ivanidzo4ka left a comment

Choose a reason for hiding this comment

artidoro Apr 9, 2019

Choose a reason for hiding this comment

artidoro Apr 9, 2019

Choose a reason for hiding this comment

artidoro Apr 9, 2019

Choose a reason for hiding this comment

artidoro Apr 9, 2019

Choose a reason for hiding this comment

artidoro left a comment • edited Loading

Choose a reason for hiding this comment

codecov bot commented Apr 8, 2019 •

edited

Loading

Ivanidzo4ka Apr 9, 2019 •

edited by artidoro

Loading

Ivanidzo4ka Apr 9, 2019 •

edited by rogancarr

Loading

Ivanidzo4ka Apr 9, 2019 •

edited by rogancarr

Loading

Ivanidzo4ka Apr 9, 2019 •

edited by rogancarr

Loading

artidoro left a comment •

edited

Loading