Samples for FeatureSelection transform estimators #3184

abgoswam · 2019-04-03T00:16:07Z

Towards #1209

The PR makes the following changes

Adds sample for the SelectFeaturesBasedOnCount transform estimator.
Adds sample for the SelectFeaturesBasedOnMutualInformation transform estimator.
Delete old sample.

codecov · 2019-04-03T00:54:24Z

Codecov Report

Merging #3184 into master will increase coverage by 0.06%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3184      +/-   ##
==========================================
+ Coverage   72.54%    72.6%   +0.06%     
==========================================
  Files         807      807              
  Lines      144774   145077     +303     
  Branches    16208    16213       +5     
==========================================
+ Hits       105021   105332     +311     
+ Misses      35339    35326      -13     
- Partials     4414     4419       +5

Flag	Coverage Δ
#Debug	`72.6% <ø> (+0.06%)`	⬆️
#production	`68.14% <ø> (+0.01%)`	⬆️
#test	`88.92% <ø> (+0.09%)`	⬆️

Impacted Files	Coverage Δ
...Microsoft.ML.Transforms/FeatureSelectionCatalog.cs	`60% <ø> (ø)`	⬆️
...c/Microsoft.ML.FastTree/Utils/ThreadTaskManager.cs	`79.48% <0%> (-20.52%)`	⬇️
src/Microsoft.ML.DataView/KeyDataViewType.cs	`74.57% <0%> (-3.76%)`	⬇️
src/Microsoft.ML.Maml/MAML.cs	`24.75% <0%> (-1.46%)`	⬇️
src/Microsoft.ML.Transforms/Text/LdaTransform.cs	`89.26% <0%> (-0.63%)`	⬇️
src/Microsoft.ML.Data/Transforms/ValueMapping.cs	`84.26% <0%> (-0.14%)`	⬇️
test/Microsoft.ML.Tests/ImagesTests.cs	`98.69% <0%> (-0.13%)`	⬇️
src/Microsoft.ML.Transforms/CategoricalCatalog.cs	`68.42% <0%> (ø)`	⬆️
...Microsoft.ML.Tests/Transformers/NormalizerTests.cs	`100% <0%> (ø)`	⬆️
...crosoft.ML.Tests/Transformers/ValueMappingTests.cs	`100% <0%> (ø)`	⬆️
... and 7 more

sfilipi · 2019-04-03T15:41:10Z

src/Microsoft.ML.Transforms/FeatureSelectionCatalog.cs

@@ -24,7 +24,7 @@ public static class FeatureSelectionCatalog
        /// <example>
        /// <format type="text/markdown">
        /// <![CDATA[
-        /// [!code-csharp[SelectFeaturesBasedOnMutualInformation](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/FeatureSelectionTransform.cs?range=1-4,10-121)]
+        /// [!code-csharp[SelectFeaturesBasedOnMutualInformation](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/SelectFeaturesBasedOnMutualInformation.cs)]


Dynamic/S [](start = 118, length = 9)

Dynamic/Transforms/ #Resolved

sfilipi · 2019-04-03T15:41:29Z

src/Microsoft.ML.Transforms/FeatureSelectionCatalog.cs

+        /// <example>
+        /// <format type="text/markdown">
+        /// <![CDATA[
+        /// [!code-csharp[SelectFeaturesBasedOnMutualInformation](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/SelectFeaturesBasedOnMutualInformation.cs)]


Dynamic [](start = 118, length = 7)

Dynamic/Transforms/ #Resolved

sfilipi · 2019-04-03T15:44:58Z

...mples/Microsoft.ML.Samples/Dynamic/Transforms/FeatureSelection/SelectFeaturesBasedOnCount.cs

+        {
+            // Downloading a classification dataset from github.com/dotnet/machinelearning.
+            // It will be stored in the same path as the executable
+            string dataFilePath = SamplesUtils.DatasetUtils.DownloadBreastCancerDataset();


string dataFilePath = SamplesUtils.DatasetUtils.DownloadBreastCancerDataset(); [](start = 9, length = 81)

can it be done with a small in memory dataset? #Resolved

sfilipi · 2019-04-03T15:45:45Z

...oft.ML.Samples/Dynamic/Transforms/FeatureSelection/SelectFeaturesBasedOnMutualInformation.cs

+            pipeline = mlContext.Transforms.FeatureSelection.SelectFeaturesBasedOnMutualInformation(
+                new InputOutputColumnPair[] { new InputOutputColumnPair("GroupB"), new InputOutputColumnPair("GroupC") },
+                labelColumnName: "Label",
+                slotsInOutput:4);


slotsInOutput:4 [](start = 16, length = 15)

one line comment about what this does. #Resolved

comment in line 28 should clarify this

In reply to: 271808888 [](ancestors = 271808888)

sfilipi · 2019-04-03T15:45:59Z

...oft.ML.Samples/Dynamic/Transforms/FeatureSelection/SelectFeaturesBasedOnMutualInformation.cs

+            // 3 7 1
+            // 3 1 1
+
+            // Second, we define the transformations that we apply on the data. Remember that an Estimator does not transform data


Remember [](start = 80, length = 8)

remove #Resolved

sfilipi · 2019-04-03T15:47:58Z

...mples/Microsoft.ML.Samples/Dynamic/Transforms/FeatureSelection/SelectFeaturesBasedOnCount.cs

+        {
+            public float[] GroupB { get; set; }
+
+            public float[] GroupC { get; set; }


space #Resolved

Ivanidzo4ka · 2019-04-03T19:58:46Z

...mples/Microsoft.ML.Samples/Dynamic/Transforms/FeatureSelection/SelectFeaturesBasedOnCount.cs

+            string dataFilePath = SamplesUtils.DatasetUtils.DownloadBreastCancerDataset();
+
+            // Data Preview
+            //    1. Label							0=benign, 1=malignant


[](start = 26, length = 7)

He use tabs! Where is my pitchfork! #Resolved

:) lol

In reply to: 271907898 [](ancestors = 271907898)

Ivanidzo4ka · 2019-04-03T20:09:05Z

...mples/Microsoft.ML.Samples/Dynamic/Transforms/FeatureSelection/SelectFeaturesBasedOnCount.cs

+            // In this example we define a CountFeatureSelectingEstimator, that selects slots in a feature vector that have more non-default 
+            // values than the specified count. This transformation can be used to remove slots with too many missing values.
+            var pipeline = mlContext.Transforms.FeatureSelection.SelectFeaturesBasedOnCount(
+                outputColumnName: "FeaturesSelectedGroupB", inputColumnName: "GroupB", count: 695);


695 [](start = 94, length = 3)

Where this number coming from?
#Resolved

comment in line 29 should clarify this now, also the in-memory example should make it more intuitive

In reply to: 271911491 [](ancestors = 271911491)

Ivanidzo4ka · 2019-04-03T20:12:12Z

...mples/Microsoft.ML.Samples/Dynamic/Transforms/FeatureSelection/SelectFeaturesBasedOnCount.cs

+            // 5 7
+            // 1 2
+            // 1 3
+            // 3 2


with current data it's not obvious at all. Can we switch to some small in memory sample rather than unknown dataset? #Resolved

true. moved over to use a small in-memory dataset instead

In reply to: 271912555 [](ancestors = 271912555)

shmoradims · 2019-04-04T00:40:43Z

...soft.ML.Samples/Dynamic/Transforms/FeatureSelection/SelectFeaturesBasedOnCountMultiColumn.cs

+            // We will use the SelectFeaturesBasedOnCount transform estimator, to retain only those slots which have 
+            // at least 'count' non-default values per slot.
+
+            // Multi column example : This pipeline uses two columns for transformation


This pipeline uses two columns for transformation [](start = 38, length = 49)

i think this is clearer: this pipeline transform two columns using the same options.

just want to make sure it's clear that columns are transformed independently and are not mixed #Resolved

shmoradims

abgoswam · 2019-04-04T21:31:13Z

...oft.ML.Samples/Dynamic/Transforms/FeatureSelection/SelectFeaturesBasedOnMutualInformation.cs

+            // We define a MutualInformationFeatureSelectingEstimator that selects the top k slots in a feature 
+            // vector based on highest mutual information between that slot and a specified label. 
+
+            var pipeline = mlContext.Transforms.FeatureSelection.SelectFeaturesBasedOnMutualInformation(


SelectFeaturesBasedOnMutualInformation [](start = 65, length = 38)

this sample is only for API reference, so small in-memory dataset suffices for this example.

we should have a "tutorial" to show the computation of MI..something along the lines of

https://www.researchgate.net/post/How_can_i_calculate_Mutual_Information_theory_from_a_simple_dataset

+1 @natke was looking into how this transform works.

In reply to: 272375725 [](ancestors = 272375725)

sfilipi · 2019-04-04T22:27:47Z

...mples/Microsoft.ML.Samples/Dynamic/Transforms/FeatureSelection/SelectFeaturesBasedOnCount.cs

+            foreach (var item in convertedData)
+                Console.WriteLine("{0}\t\t\t{1}", string.Join("\t", item.NumericVector), string.Join("\t", item.StringVector));
+            // 4       NaN     6                       A       WA   Male
+            // 4       5       6                       A            Female


[](start = 63, length = 5)

align for just here.. no need to make it match exactly to the output and have it looked tabbing off. #Resolved

or is it a separate column?

It helps to print the headers.

In reply to: 272391190 [](ancestors = 272391190)

the alignment here seems good to me... for text this slot is empty or null ..

not sure if u meant something else

In reply to: 272391190 [](ancestors = 272391190)

sfilipi · 2019-04-04T22:29:19Z

...mples/Microsoft.ML.Samples/Dynamic/Transforms/FeatureSelection/SelectFeaturesBasedOnCount.cs

+
+            // We will use the SelectFeaturesBasedOnCount to retain only those slots which have at least 'count' non-default values per slot.
+
+            // Usage on numeric column.


remove space #Resolved

sfilipi · 2019-04-04T22:30:42Z

...mples/Microsoft.ML.Samples/Dynamic/Transforms/FeatureSelection/SelectFeaturesBasedOnCount.cs

+            // The pipeline can then be trained, using .Fit(), and the resulting transformer can be used to transform data. 
+            var transformedData = pipeline.Fit(data).Transform(data);
+
+            Console.WriteLine("Contents of column 'NumericVector'");


Console.WriteLine("Contents of column 'NumericVector'"); [](start = 11, length = 57)

convert to just comment. #Resolved

sfilipi · 2019-04-04T22:30:56Z

...mples/Microsoft.ML.Samples/Dynamic/Transforms/FeatureSelection/SelectFeaturesBasedOnCount.cs

+                    Console.Write($"{row[i]}\t");
+                Console.WriteLine();
+            }
+            // 4       6


// 4 6 [](start = 12, length = 12)

headers #Resolved

sfilipi · 2019-04-04T22:31:52Z

...mples/Microsoft.ML.Samples/Dynamic/Transforms/FeatureSelection/SelectFeaturesBasedOnCount.cs

+                for (var i = 0; i < row.Length; i++)
+                    Console.Write($"{row[i]}\t");
+                Console.WriteLine();
+            }


would it polish it a bit if you made a little helper for this, since it is being used twice? #Resolved

one of then is a float[] while the other is string[] .. did not want to over-engineer this...

is there anything specific you had in mind, or can we keep it as is ?

In reply to: 272392068 [](ancestors = 272392068)

object :)

In reply to: 272394600 [](ancestors = 272394600,272392068)

sfilipi · 2019-04-04T22:32:43Z

...mples/Microsoft.ML.Samples/Dynamic/Transforms/FeatureSelection/SelectFeaturesBasedOnCount.cs

+
+            Console.WriteLine("Contents of two columns 'NumericVector' and 'StringVector'.");
+            foreach (var item in convertedData)
+                Console.WriteLine("{0}\t\t\t{1}", string.Join("\t", item.NumericVector), string.Join("\t", item.StringVector));


\t [](start = 63, length = 2)

think for vectors we are using comma ',' as the separator. #Resolved

sfilipi · 2019-04-04T22:52:17Z

...mples/Microsoft.ML.Samples/Dynamic/Transforms/FeatureSelection/SelectFeaturesBasedOnCount.cs

+
+            // Usage on text column.
+            pipeline = mlContext.Transforms.FeatureSelection.SelectFeaturesBasedOnCount(
+                outputColumnName: "StringVector", count: 3);


I'd probably append to the previous pipeline, and show the prints once. Compacts the sample. #Resolved

sounds good. will do.

In reply to: 272396211 [](ancestors = 272396211)

sfilipi · 2019-04-04T22:56:42Z

...soft.ML.Samples/Dynamic/Transforms/FeatureSelection/SelectFeaturesBasedOnCountMultiColumn.cs

+
+            Console.WriteLine("Contents of two columns 'NumericVector' and 'StringVector'.");
+            foreach (var item in rawData)
+                Console.WriteLine("{0}\t\t\t{1}", string.Join("\t", item.NumericVector), string.Join("\t", item.StringVector));


\t [](start = 63, length = 2)

same comment, ',' #Resolved

sfilipi · 2019-04-04T22:59:46Z

...oft.ML.Samples/Dynamic/Transforms/FeatureSelection/SelectFeaturesBasedOnMutualInformation.cs

+
+            Console.WriteLine("Contents of column 'NumericVector'");
+            PrintDataColumn(transformedData, "NumericVector");
+            // 4 0


4 [](start = 15, length = 1)

Curious, why is it dropping 6, but keeping 4? It is not obvious to me. Is it because slotsInOutput is 2? A comment about that might help. #Pending

we need to improve this once we have found a better sample. i have noted this in the issue #1209

In reply to: 272397744 [](ancestors = 272397744)

sfilipi

sfilipi · 2019-04-04T23:02:10Z

...les/Dynamic/Transforms/FeatureSelection/SelectFeaturesBasedOnMutualInformationMultiColumn.cs

+
+            Console.WriteLine("Contents of columns 'Label', 'NumericVectorA' and 'NumericVectorB'.");
+            foreach (var item in rawData)
+                Console.WriteLine("{0}\t\t{1}\t\t{2}", item.Label, string.Join(" ", item.NumericVectorA), string.Join(" ", item.NumericVectorB));


[](start = 80, length = 1)

',' #Resolved

zeahmed · 2019-04-04T23:02:40Z

...mples/Microsoft.ML.Samples/Dynamic/Transforms/FeatureSelection/SelectFeaturesBasedOnCount.cs

+            var rawData = GetData();
+            var data = mlContext.Data.LoadFromEnumerable(rawData);
+
+            var convertedData = mlContext.Data.CreateEnumerable<InputData>(data, true);


convertedData [](start = 16, length = 13)

Why do you convert that back to Enumerable? rawData is already there. #Resolved

sfilipi · 2019-04-04T23:05:24Z

...les/Dynamic/Transforms/FeatureSelection/SelectFeaturesBasedOnMutualInformationMultiColumn.cs

+            foreach (var item in convertedData)
+                Console.WriteLine("{0}\t\t{1}", string.Join(" ", item.NumericVectorA), string.Join(" ", item.NumericVectorB));
+
+            // Here, we see SelectFeaturesBasedOnMutualInformation selected 4 slots.


4 slot [](start = 76, length = 6)

the 4 slots that had most in common with the respective value in the Label column, maybe? #Resolved

yes, these 4 slots carried the most MI with Label.... we should have a better tutorial for this though

In reply to: 272398839 [](ancestors = 272398839)

sfilipi

rogancarr

* samples for FeatureSelection transform estimators * fix review comments * fix review comments * review comments * take care of review comments * fix copy paste output error

samples for FeatureSelection transform estimators

7545d5a

abgoswam requested review from Ivanidzo4ka, shmoradims and sfilipi April 3, 2019 00:16

sfilipi mentioned this pull request Apr 3, 2019

API reference - Samples for Transforms #1209

Closed

sfilipi reviewed Apr 3, 2019

View reviewed changes

Ivanidzo4ka reviewed Apr 3, 2019

View reviewed changes

fix review comments

fdcc012

shmoradims reviewed Apr 4, 2019

View reviewed changes

fix review comments

c4533bc

shmoradims approved these changes Apr 4, 2019

View reviewed changes

review comments

d44c6a2

abgoswam requested a review from rogancarr April 4, 2019 21:27

abgoswam commented Apr 4, 2019

View reviewed changes

sfilipi reviewed Apr 4, 2019

View reviewed changes

sfilipi approved these changes Apr 4, 2019

View reviewed changes

sfilipi reviewed Apr 4, 2019

View reviewed changes

zeahmed reviewed Apr 4, 2019

View reviewed changes

sfilipi reviewed Apr 4, 2019

View reviewed changes

sfilipi approved these changes Apr 4, 2019

View reviewed changes

abgoswam added 2 commits April 4, 2019 23:32

take care of review comments

51b8507

fix copy paste output error

cb09d87

rogancarr approved these changes Apr 4, 2019

View reviewed changes

abgoswam merged commit 8130567 into dotnet:master Apr 5, 2019

abgoswam mentioned this pull request Apr 5, 2019

Cherry pick for samples {Image, Categorical, FeatureSelection} #3221

Closed

ghost locked as resolved and limited conversation to collaborators Mar 23, 2022


		// We will use the SelectFeaturesBasedOnCount to retain only those slots which have at least 'count' non-default values per slot.

		// Usage on numeric column.

Samples for FeatureSelection transform estimators #3184

Samples for FeatureSelection transform estimators #3184

Conversation

abgoswam commented Apr 3, 2019

codecov bot commented Apr 3, 2019 • edited Loading

Codecov Report

sfilipi Apr 3, 2019 • edited by abgoswam Loading

Choose a reason for hiding this comment

sfilipi Apr 3, 2019 • edited by abgoswam Loading

Choose a reason for hiding this comment

sfilipi Apr 3, 2019 • edited by abgoswam Loading

Choose a reason for hiding this comment

sfilipi Apr 3, 2019 • edited by abgoswam Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfilipi Apr 3, 2019 • edited by abgoswam Loading

Choose a reason for hiding this comment

sfilipi Apr 3, 2019 • edited by abgoswam Loading

Choose a reason for hiding this comment

Ivanidzo4ka Apr 3, 2019 • edited by abgoswam Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ivanidzo4ka Apr 3, 2019 • edited by abgoswam Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ivanidzo4ka Apr 3, 2019 • edited by abgoswam Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shmoradims Apr 4, 2019 • edited by abgoswam Loading

Choose a reason for hiding this comment

shmoradims left a comment

Choose a reason for hiding this comment

abgoswam Apr 4, 2019 • edited Loading

Choose a reason for hiding this comment

sfilipi Apr 4, 2019 • edited Loading

Choose a reason for hiding this comment

sfilipi Apr 4, 2019 • edited by abgoswam Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfilipi Apr 4, 2019 • edited by abgoswam Loading

Choose a reason for hiding this comment

sfilipi Apr 4, 2019 • edited by abgoswam Loading

Choose a reason for hiding this comment

sfilipi Apr 4, 2019 • edited by abgoswam Loading

Choose a reason for hiding this comment

sfilipi Apr 4, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfilipi Apr 4, 2019 • edited by abgoswam Loading

Choose a reason for hiding this comment

sfilipi Apr 4, 2019 • edited by abgoswam Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfilipi Apr 4, 2019 • edited by abgoswam Loading

Choose a reason for hiding this comment

sfilipi Apr 4, 2019 • edited by abgoswam Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfilipi left a comment

Choose a reason for hiding this comment

sfilipi Apr 4, 2019 • edited by abgoswam Loading

Choose a reason for hiding this comment

zeahmed Apr 4, 2019 • edited by abgoswam Loading

Choose a reason for hiding this comment

sfilipi Apr 4, 2019 • edited by abgoswam Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfilipi left a comment

Choose a reason for hiding this comment

rogancarr left a comment

Choose a reason for hiding this comment

codecov bot commented Apr 3, 2019 •

edited

Loading

sfilipi Apr 3, 2019 •

edited by abgoswam

Loading

sfilipi Apr 3, 2019 •

edited by abgoswam

Loading

sfilipi Apr 3, 2019 •

edited by abgoswam

Loading

sfilipi Apr 3, 2019 •

edited by abgoswam

Loading

sfilipi Apr 3, 2019 •

edited by abgoswam

Loading

sfilipi Apr 3, 2019 •

edited by abgoswam

Loading

Ivanidzo4ka Apr 3, 2019 •

edited by abgoswam

Loading

Ivanidzo4ka Apr 3, 2019 •

edited by abgoswam

Loading

Ivanidzo4ka Apr 3, 2019 •

edited by abgoswam

Loading

shmoradims Apr 4, 2019 •

edited by abgoswam

Loading

abgoswam Apr 4, 2019 •

edited

Loading

sfilipi Apr 4, 2019 •

edited

Loading

sfilipi Apr 4, 2019 •

edited by abgoswam

Loading

sfilipi Apr 4, 2019 •

edited by abgoswam

Loading

sfilipi Apr 4, 2019 •

edited by abgoswam

Loading

sfilipi Apr 4, 2019 •

edited by abgoswam

Loading

sfilipi Apr 4, 2019 •

edited

Loading

sfilipi Apr 4, 2019 •

edited by abgoswam

Loading

sfilipi Apr 4, 2019 •

edited by abgoswam

Loading

sfilipi Apr 4, 2019 •

edited by abgoswam

Loading

sfilipi Apr 4, 2019 •

edited by abgoswam

Loading

sfilipi Apr 4, 2019 •

edited by abgoswam

Loading

zeahmed Apr 4, 2019 •

edited by abgoswam

Loading

sfilipi Apr 4, 2019 •

edited by abgoswam

Loading