Transforms components docs #1321

sfilipi · 2018-10-19T20:38:37Z

Addresses P1 non-Image transforms for #1209.

Adds samples for Nomalizers, Text, Cocat, Term, KeyToVal

…ntData and sample for the text transform.

…al estimators.

Adding samples for the ConcatEstimator, and KeyToValue, Term that will need to get referenced from the respective catalogs when they happen. re-organized the samples based on static-dynamic. Renamed.

sfilipi · 2018-10-19T20:41:20Z

docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValue_Term.cs

+            var default_pipeline = new WordTokenizer(ml, "ReviewReverse", "ReviewReverse")
+                .Append(new TermEstimator(ml, "ReviewReverse" , defaultColumnName));
+
+            // Another pipeline, that customizes the advanced settings of the FeaturizeText transformer.


// Another pip [](start = 11, length = 15)

@justinormont, @TomFinley you guys think I should add more explanations about SortOrder here in the comments? #Resolved

sfilipi · 2018-10-19T20:52:32Z

docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValue_Term.cs

+            // DefaultColumnName column obtained post-transformation.
+            // radiation galaxy universe duck
+            // space galaxy universe radiation
+            // bus pickup universe radiation


universe radiation [](start = 26, length = 18)

why is this here, log an issue post merge.

justinormont · 2018-10-23T06:00:58Z

docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValue_Term.cs

+            var trainData = ml.CreateStreamingDataView(data);
+
+            // Preview of the data.
+            // Review                                    ReviewReverse,                 Label


Suggested change

// Review ReviewReverse, Label

// Review, ReviewReverse, Label

``` #Resolved

justinormont · 2018-10-23T06:08:28Z

docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValue_Term.cs

+            // "animals birds cats dogs fish horse", "radiation galaxy universe duck",     1
+            // "horse birds house fish duck cats",   "space galaxy universe radiation",    0
+            // "car truck driver bus pickup",        "bus pickup",                         1
+            // "car truck driver bus pickup horse",  "car truck",                          0


May want to say the goal of the dataset. Eg: "The goal of the dataset to classify if the review matches ..."

I ask this, mainly as I'm reading the example, I have no idea what the labels represent vs. the data. #Resolved

+1. It's not obvious what the labels mean.

In reply to: 227236398 [](ancestors = 227236398)

justinormont · 2018-10-23T06:11:11Z

docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValue_Term.cs

+            var default_pipeline = new WordTokenizer(ml, "ReviewReverse", "ReviewReverse")
+                .Append(new TermEstimator(ml, "ReviewReverse" , defaultColumnName));
+
+            // Another pipeline, that customizes the advanced settings of the TermEstimator.


May want to say why changing the hyperparameters of the TermEstimator is useful. Why keep only the first 10 ASCIIbetically ordered terms? #Pending

see if you'd phrase it differently?

In reply to: 227236905 [](ancestors = 227236905)

justinormont · 2018-10-23T06:11:40Z

docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValue_Term.cs

+            // making use of default settings.
+            string defaultColumnName = "DefaultKeys";
+            // REVIEW create through the catalog extension
+            var default_pipeline = new WordTokenizer(ml, "ReviewReverse", "ReviewReverse")


Why use WordTokenizer+Term instead of TextTransform? #Resolved

The sample is about TermEstimator.

In reply to: 227237000 [](ancestors = 227237000)

justinormont · 2018-10-23T06:28:17Z

The two added datasets could use an entry in the dataset README.md: https://github.com/dotnet/machinelearning/blob/586533cb80d7e0ea18c02f2e510e1a5cc0e27f41/test/data/README.md
#Resolved

…/machinelearning-1 into transformsComponentsDocs

GalOshri · 2018-10-23T22:49:07Z

docs/samples/Microsoft.ML.Samples/Dynamic/ConcatTransform.cs

+            var transformedData = pipeline.Fit(trainData).Transform(trainData);
+
+            // Getting the data of the newly created column as an IEnumerable of SampleInfertDataWithFeatures.
+            var featuresColumn = transformedData.AsEnumerable<SampleInfertDataWithFeatures>(ml, reuseRowObject: false);


What does reuseRowObject do? Would defaults be fine for these samples? #Resolved

for the enumerable, it determines whether to return the same object on every row, or allocate a new one per row. It is a required param; doesn't have a default.

For the settings of the transforms, i am using both defaults and non-defaults; since the purpose of this snippet is to educate about usage.

In reply to: 227591855 [](ancestors = 227591855)

shmoradims · 2018-10-24T21:58:15Z

docs/samples/Microsoft.ML.Samples/Dynamic/MinMaxNormalizer.cs

+            printHelper("Induced", normalizedColumn);
+
+            // Preview of the data.
+            // Induced


Induced [](start = 15, length = 7)

nit: this doesn't match $"{colName} column obtained post-transformation." #Closed

The col name is "Induced", see line 39.

In reply to: 227974775 [](ancestors = 227974775)

I meant if these comment are supposed to be the output from printHelper. If they are, then the headers don't match:
// Preview of the data.
// Induced

vs

Console.WriteLine($"{colName} column obtained post-transformation.");

Same thing applies to other comments showing data preview.

In reply to: 227982369 [](ancestors = 227982369,227974775)

shmoradims · 2018-10-24T22:05:29Z

docs/samples/Microsoft.ML.Samples/Static/FastTree.cs

+                separator: '\t', hasHeader: true);
+
+            // Read the data, and leave 10% out, so we can use them for testing
+            var data = reader.Read(new MultiFileSource(dataFile));


new MultiFileSource(dataFile) [](start = 35, length = 29)

I think you can use dataFile directly for a single file. #Closed

shmoradims · 2018-10-24T22:06:26Z

docs/samples/Microsoft.ML.Samples/Static/FastTree.cs

+            Console.WriteLine($"L2 - {averagedMetrics.L2}");
+            Console.WriteLine($"LossFunction - {averagedMetrics.LossFn}");
+            Console.WriteLine($"RMS - {averagedMetrics.Rms}");
+            Console.WriteLine($"RSquared - {averagedMetrics.RSquared}");


please add preview #Closed

shmoradims · 2018-10-25T16:32:07Z

docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValue_Term.cs

+            var trainData = ml.CreateStreamingDataView(data);
+
+            // Preview of the topics data; a dataset that contains one column with two set of keys assigned to a body of text
+            // Review and ReviewReverse. The dataset will be used to classify how accurately the keys are assigned to the text.  


ReviewReverse [](start = 26, length = 13)

It's not clear to me what this column means. #Resolved

shmoradims · 2018-10-25T16:36:08Z

docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValue_Term.cs

+            // making use of default settings.
+            string defaultColumnName = "DefaultKeys";
+            // REVIEW create through the catalog extension
+            var default_pipeline = new WordTokenizeEstimator(ml, "ReviewReverse", "ReviewReverse")


ReviewReverse [](start = 83, length = 13)

I think it's better to leave this out, if you want to do the transformation in-place. #Resolved

shmoradims · 2018-10-25T16:39:28Z

docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValue_Term.cs

+            // radiation galaxy universe duck
+            // space galaxy universe radiation
+            // bus pickup universe radiation
+            // car truck universe radiation


universe radiation [](start = 25, length = 18)

this seems to be a bug too. #Closed

right, will investigate post PR.

In reply to: 228249405 [](ancestors = 228249405)

shmoradims · 2018-10-25T17:47:37Z

docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValue_Term.cs

+            // and condition the order in which they get evaluated by changing sort from the default Occurence (order in which they get encountered) 
+            // to value/alphabetically.
+            string customizedColumnName = "CustomizedKeys";
+            var customized_pipeline = new WordTokenizeEstimator(ml, "ReviewReverse", "ReviewReverse")


ReviewReverse [](start = 86, length = 13)

plz remove for in-place transformation like above #Resolved

shmoradims

Ivanidzo4ka · 2018-10-25T18:53:24Z

docs/samples/Microsoft.ML.Samples/Program.cs

-            Trainers.SdcaRegression();
-            Transformers.ConcatEstimator();
+            TransformSamples.MinMaxNormalizer();
+            // TrainersSamples.LightGbmRegression();


// TrainersSamples.LightGbmRegression(); [](start = 12, length = 40)

looks weird. #Resolved

Ivanidzo4ka · 2018-10-25T19:39:00Z

docs/samples/Microsoft.ML.Samples/Static/FastTree.cs

+            string dataFile = SamplesUtils.DatasetUtils.DownloadHousingRegressionDataset();
+
+            // Creating the ML.Net IHostEnvironment object, needed for the pipeline
+            var env = new LocalEnvironment(seed: 0);


var env = new LocalEnvironment(seed: 0); [](start = 11, length = 41)

why for transforms examples we use mlContext, and here we create LocalEnvironment? #Resolved

thanks for spotting it. I updated them.

In reply to: 228308134 [](ancestors = 228308134)

Ivanidzo4ka · 2018-10-25T19:44:21Z

test/data/gplv2/airquality.csv

+"9",8,19,20.1,61,5,9
+"10",NA,194,8.6,69,5,10
+"11",7,NA,6.9,74,5,11
+"12",16,256,9.7,69,5,12


I don't see any usage of this file in PR, not even mentions of air quality.
Is it necessary?
And is it regression dataset? because we have #938 #Pending

planning to keep the samples consistent with our other repo. They use this dataset, so it will get used in the next set of samples. Just added the two we need in one shot .

In reply to: 228309756 [](ancestors = 228309756)

re-arranging lines.

Ivanidzo4ka · 2018-10-26T18:08:42Z

docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValue_Term.cs

+{
+    public partial class TransformSamples
+    {
+        public static void KeyToValue_Term()


KeyToValue_Term [](start = 27, length = 15)

This is standing out, what this "_" mean, and why it cannot be KeyToValueAndTerm or KeyToValueThenTerm?

Zruty0

…/machinelearning-1 into transformsComponentsDocs

sfilipi added 4 commits October 17, 2018 16:36

Placeholder methods for all P1 transforms, InputClass for the Sentime…

95b34ef

…ntData and sample for the text transform.

Merge branch 'master' into transformsComponentsDocs

121c2e5

adding airquality, infert datasets. Added samples for Term and KeyToV…

f6578a6

…al estimators.

Adding tests for the NormalizerCatalog and the TextTransformCatalog.

161c0d3

Adding samples for the ConcatEstimator, and KeyToValue, Term that will need to get referenced from the respective catalogs when they happen. re-organized the samples based on static-dynamic. Renamed.

sfilipi self-assigned this Oct 19, 2018

sfilipi added API Issues pertaining the friendly API documentation Related to documentation of ML.NET labels Oct 19, 2018

sfilipi requested review from Zruty0, shmoradims, zyw400, justinormont, CESARDELATORRE, GalOshri, TomFinley and najeeb-kazmi October 19, 2018 20:39

sfilipi commented Oct 19, 2018

View reviewed changes

sfilipi added 2 commits October 19, 2018 13:48

fixing comments.

f8664d9

fit and finish

8c8483d

sfilipi commented Oct 19, 2018

View reviewed changes

sfilipi mentioned this pull request Oct 19, 2018

API reference - Samples for Transforms #1209

Closed

sfilipi added 2 commits October 19, 2018 14:05

Merge branch 'master' into transformsComponentsDocs

c84f17d

typo

3a5490c

sfilipi mentioned this pull request Oct 22, 2018

Transform documentation needs examples dotnet/docs#8528

Closed

justinormont reviewed Oct 23, 2018

View reviewed changes

sfilipi added 2 commits October 22, 2018 23:52

Merge branch 'master' into transformsComponentsDocs

5274eed

merging from master

909a46a

sfilipi added 5 commits October 23, 2018 11:19

justin's comments

9f84319

spaces are not cool.

80aa06d

Merge branch 'transformsComponentsDocs' of https://github.com/sfilipi…

b7d0a46

…/machinelearning-1 into transformsComponentsDocs

merging from master

108100d

updating the datasets entries.

e60c1e0

GalOshri reviewed Oct 23, 2018

View reviewed changes

shmoradims reviewed Oct 24, 2018

View reviewed changes

addressing comments

1ec6489

shmoradims reviewed Oct 25, 2018

View reviewed changes

PR comments

01a327a

shmoradims reviewed Oct 25, 2018

View reviewed changes

shmoradims self-requested a review October 25, 2018 17:51

shmoradims approved these changes Oct 25, 2018

View reviewed changes

Ivanidzo4ka reviewed Oct 25, 2018

View reviewed changes

sfilipi added 2 commits October 25, 2018 12:46

merging from master

0317a2e

addressingPR comments

eff8504

re-arranging lines.

Ivanidzo4ka reviewed Oct 26, 2018

View reviewed changes

Zruty0 approved these changes Oct 29, 2018

View reviewed changes

sfilipi added 3 commits October 29, 2018 22:01

Merge branch 'master' into transformsComponentsDocs

68a43c1

post merge fixes

309a13e

Merge branch 'transformsComponentsDocs' of https://github.com/sfilipi…

28b698a

…/machinelearning-1 into transformsComponentsDocs

sfilipi merged commit 1bcb79d into dotnet:master Oct 30, 2018

sfilipi deleted the transformsComponentsDocs branch October 30, 2018 20:06

ghost locked as resolved and limited conversation to collaborators Mar 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Transforms components docs #1321

Transforms components docs #1321

sfilipi commented Oct 19, 2018

sfilipi Oct 19, 2018 •

edited

Loading

sfilipi Oct 19, 2018 •

edited

Loading

justinormont Oct 23, 2018 •

edited by sfilipi

Loading

justinormont Oct 23, 2018 •

edited by sfilipi

Loading

shmoradims Oct 25, 2018

justinormont Oct 23, 2018 •

edited by sfilipi

Loading

sfilipi Oct 23, 2018 •

edited

Loading

justinormont Oct 23, 2018 •

edited by sfilipi

Loading

sfilipi Oct 23, 2018

justinormont commented Oct 23, 2018 •

edited by sfilipi

Loading

GalOshri Oct 23, 2018 •

edited by sfilipi

Loading

sfilipi Oct 24, 2018

shmoradims Oct 24, 2018 •

edited

Loading

sfilipi Oct 24, 2018

shmoradims Oct 25, 2018 •

edited

Loading

shmoradims Oct 24, 2018 •

edited

Loading

shmoradims Oct 24, 2018 •

edited

Loading

shmoradims Oct 25, 2018 •

edited by sfilipi

Loading

shmoradims Oct 25, 2018 •

edited by sfilipi

Loading

shmoradims Oct 25, 2018 •

edited

Loading

sfilipi Oct 25, 2018

shmoradims Oct 25, 2018 •

edited by sfilipi

Loading

shmoradims left a comment

Ivanidzo4ka Oct 25, 2018 •

edited by sfilipi

Loading

Ivanidzo4ka Oct 25, 2018 •

edited by sfilipi

Loading

sfilipi Oct 25, 2018

Ivanidzo4ka Oct 25, 2018 •

edited by sfilipi

Loading

sfilipi Oct 25, 2018

Ivanidzo4ka Oct 26, 2018

Zruty0 left a comment

	// Review ReviewReverse, Label
	// Review, ReviewReverse, Label
	``` #Resolved

Transforms components docs #1321

Transforms components docs #1321

Conversation

sfilipi commented Oct 19, 2018

sfilipi Oct 19, 2018 • edited Loading

Choose a reason for hiding this comment

sfilipi Oct 19, 2018 • edited Loading

Choose a reason for hiding this comment

justinormont Oct 23, 2018 • edited by sfilipi Loading

Choose a reason for hiding this comment

justinormont Oct 23, 2018 • edited by sfilipi Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

justinormont Oct 23, 2018 • edited by sfilipi Loading

Choose a reason for hiding this comment

sfilipi Oct 23, 2018 • edited Loading

Choose a reason for hiding this comment

justinormont Oct 23, 2018 • edited by sfilipi Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

justinormont commented Oct 23, 2018 • edited by sfilipi Loading

GalOshri Oct 23, 2018 • edited by sfilipi Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shmoradims Oct 24, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shmoradims Oct 25, 2018 • edited Loading

Choose a reason for hiding this comment

shmoradims Oct 24, 2018 • edited Loading

Choose a reason for hiding this comment

shmoradims Oct 24, 2018 • edited Loading

Choose a reason for hiding this comment

shmoradims Oct 25, 2018 • edited by sfilipi Loading

Choose a reason for hiding this comment

shmoradims Oct 25, 2018 • edited by sfilipi Loading

Choose a reason for hiding this comment

shmoradims Oct 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shmoradims Oct 25, 2018 • edited by sfilipi Loading

Choose a reason for hiding this comment

shmoradims left a comment

Choose a reason for hiding this comment

Ivanidzo4ka Oct 25, 2018 • edited by sfilipi Loading

Choose a reason for hiding this comment

Ivanidzo4ka Oct 25, 2018 • edited by sfilipi Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ivanidzo4ka Oct 25, 2018 • edited by sfilipi Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zruty0 left a comment

Choose a reason for hiding this comment

sfilipi Oct 19, 2018 •

edited

Loading

sfilipi Oct 19, 2018 •

edited

Loading

justinormont Oct 23, 2018 •

edited by sfilipi

Loading

justinormont Oct 23, 2018 •

edited by sfilipi

Loading

justinormont Oct 23, 2018 •

edited by sfilipi

Loading

sfilipi Oct 23, 2018 •

edited

Loading

justinormont Oct 23, 2018 •

edited by sfilipi

Loading

justinormont commented Oct 23, 2018 •

edited by sfilipi

Loading

GalOshri Oct 23, 2018 •

edited by sfilipi

Loading

shmoradims Oct 24, 2018 •

edited

Loading

shmoradims Oct 25, 2018 •

edited

Loading

shmoradims Oct 24, 2018 •

edited

Loading

shmoradims Oct 24, 2018 •

edited

Loading

shmoradims Oct 25, 2018 •

edited by sfilipi

Loading

shmoradims Oct 25, 2018 •

edited by sfilipi

Loading

shmoradims Oct 25, 2018 •

edited

Loading

shmoradims Oct 25, 2018 •

edited by sfilipi

Loading

Ivanidzo4ka Oct 25, 2018 •

edited by sfilipi

Loading

Ivanidzo4ka Oct 25, 2018 •

edited by sfilipi

Loading

Ivanidzo4ka Oct 25, 2018 •

edited by sfilipi

Loading