XML documentation for Normalizer #3432

singlis · 2019-04-19T01:25:52Z

Tracked in #3204

codecov · 2019-04-19T02:03:23Z

Codecov Report

❗ No coverage uploaded for pull request base (master@082ab77). Click here to learn what that means.
The diff coverage is 83.33%.

@@            Coverage Diff            @@
##             master    #3432   +/-   ##
=========================================
  Coverage          ?   68.27%           
=========================================
  Files             ?      638           
  Lines             ?   113948           
  Branches          ?    14256           
=========================================
  Hits              ?    77796           
  Misses            ?    32048           
  Partials          ?     4104

Flag	Coverage Δ
#Debug	`68.27% <83.33%> (?)`
#production	`68.27% <83.33%> (?)`

Impacted Files	Coverage Δ
src/Microsoft.ML.Data/Transforms/Normalizer.cs	`86.03% <ø> (ø)`
src/Microsoft.ML.Transforms/NormalizerCatalog.cs	`84.78% <83.33%> (ø)`

natke · 2019-04-19T03:31:57Z

src/Microsoft.ML.Data/Transforms/Normalizer.cs

+    /// ###  Estimator Characteristics
+    /// |  |  |
+    /// | -- | -- |
+    /// | Does this estimator need to look at the data to train its parameters? | No |


Is this correct? Doesn't it have to look at the data to find the mean, variance, min, max etc? #Resolved

It should be yes, thanks for mentioning. I will correct this.

In reply to: 276894215 [](ancestors = 276894215)

natke · 2019-04-19T03:32:36Z

src/Microsoft.ML.Data/Transforms/Normalizer.cs

+    /// * Min Max - Linear rescale such that minimum and maximum values are mapped between -1 and 1.
+    /// * Mean Variance - Rescale to unit variance and, optionally, zero mean.
+    /// * Log Mean Variance - Rescale to unit variance on the log scale.
+    /// * Binning - Bucketize and then rescale to between -1 and 1.


Is this correct? Isn't it between 0 and 1? #ByDesign

it looks like -1 and 1. See the sample:

machinelearning/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeBinning.cs

Line 58 in ea5c095

// 0.0000, -0.3333, 0.0000, 1.0000

In reply to: 276894276 [](ancestors = 276894276)

shmoradims · 2019-04-19T13:58:24Z

src/Microsoft.ML.Data/Transforms/Normalizer.cs

+    /// | Output column data type | The same as the data type in the input column |
+    ///
+    /// The resulting NormalizingEstimator will normalize the data in one of the following ways based upon how it was created:
+    /// * Min Max - Linear rescale such that minimum and maximum values are mapped between -1 and 1.


between -1 and 1 [](start = 83, length = 16)

suggestion: let's use intervals: [-1,1] or even better with latex: $[-1,1]$

i think it's more clear that way #Resolved

shmoradims · 2019-04-19T14:03:58Z

src/Microsoft.ML.Data/Transforms/Normalizer.cs

+    /// |  |  |
+    /// | -- | -- |
+    /// | Does this estimator need to look at the data to train its parameters? | No |
+    /// | Input column data type | <xref:System.Single> or <xref:System.Double> |


[](start = 79, length = 1)

I think numeric vectors are also valid, aren't they? If so please change here and also the params. you can write it like this:

xref:System.Single, xref:System.Double, or a known-sized vector of those types.

#Resolved

Its only supports single or double:

machinelearning/src/Microsoft.ML.Data/Transforms/Normalizer.cs

Line 271 in ea5c095

if (!col.ItemType.Equals(NumberDataViewType.Single) && !col.ItemType.Equals(NumberDataViewType.Double))

In reply to: 276994571 [](ancestors = 276994571)

i think known-sized vectors of numerics are supported by normalizer. that line you're referencing is also true if the vector contains only numeric values. could you please confirm with Ivan?

I'm certain that I've normalized the feature vector right before trainer

In reply to: 277045796 [](ancestors = 277045796,276994571)

Look on line 298 in this file.
It's either scalar or vector of known-size of float or double

In reply to: 277069180 [](ancestors = 277069180,277045796,276994571)

sorry I read too quickly through Shahab's original comment and thought he was asking if we supported integers. Yes we do support vectors, sorry for the confusion. I am updating.

In reply to: 277070189 [](ancestors = 277070189,277069180,277045796,276994571)

shmoradims · 2019-04-19T14:05:45Z

src/Microsoft.ML.Transforms/NormalizerCatalog.cs

@@ -31,11 +31,12 @@ public static class NormalizationCatalog
        }

        /// <summary>
-        /// It normalizes the data based on the observed minimum and maximum values of the data.
+        /// Create a <see cref="NormalizingEstimator"/>, which normalizes based on the observed minimum and maximum values of the data.
        /// </summary>
        /// <param name="catalog">The transform catalog</param>
        /// <param name="outputColumnName">Name of the column resulting from the transformation of <paramref name="inputColumnName"/>.</param>


. [](start = 133, length = 1)

Add: Output data type will be the same as input data type. #Resolved

Ivanidzo4ka · 2019-04-19T19:22:56Z

src/Microsoft.ML.Data/Transforms/Normalizer.cs

+    /// | Output column data type | The same data type as the input column |
+    ///
+    /// The resulting NormalizingEstimator will normalize the data in one of the following ways based upon how it was created:
+    /// * Min Max - Linear rescale such that the values are mapped to the $[-1,1]$ with the minimum and maximum values being mapped to -1 and 1 respectively.


-1,1] [](start = 76, length = 5)

If we not talking about fixZero option all normalizers try to map to [0,1] range.
FixZero is special beast....
Not sure it's worth to specify range...

Thanks Ivan - so if its 0,1 unless fixZero is set, why not then just document the normal case?

In reply to: 277070876 [](ancestors = 277070876)

I have added further description to address this.

In reply to: 277084973 [](ancestors = 277084973,277070876)

Ivanidzo4ka · 2019-04-19T19:24:08Z

src/Microsoft.ML.Data/Transforms/Normalizer.cs

+    /// * [NormalizeBinning](xref:Microsoft.ML.NormalizationCatalog.NormalizeBinning(Microsoft.ML.TransformsCatalog,System.String,System.String,System.Int64,System.Boolean,System.Int32))
+    /// * [NormalizeSupervisedBinning](xref:Microsoft.ML.NormalizationCatalog.NormalizeSupervisedBinning(Microsoft.ML.TransformsCatalog,System.String,System.String,System.String,System.Int64,System.Boolean,System.Int32,System.Int32))
+    /// ]]>
+    /// </format>


can you move it outside of remarks section and just use seealso?

Unfortunately not as this hits assembly issues. I would have to move code which was deemed lower priority. So for now I did this to at least have some kind of reference.

wschin · 2019-04-19T23:10:56Z

src/Microsoft.ML.Data/Transforms/Normalizer.cs

+    /// |  |  |
+    /// | -- | -- |
+    /// | Does this estimator need to look at the data to train its parameters? | Yes |
+    /// | Input column data type | <xref:System.Single> or<xref:System.Double> or a known-sized vector of those types. |


thank you for catching the space.

In reply to: 277110149 [](ancestors = 277110149)

wschin · 2019-04-19T23:13:35Z

src/Microsoft.ML.Data/Transforms/Normalizer.cs

+    /// * Min Max - A linear rescale that is based upon the minimum and maximum values for each row.
+    /// * Mean Variance - Rescale each row to unit variance and, optionally, zero mean.
+    /// * Log Mean Variance - Rescale each row to unit variance based on a log scale.
+    /// * Binning - Bucketizes the data in each row and performs a linear rescale based on the calculated bins.


The explanation of binning doesn't carry much information. Also, we need equations for all of them. #WontFix

Talked offline and agreed to address later. Having equations for all of our transformer/trainers is a much larger issue to address.

In reply to: 277110413 [](ancestors = 277110413)

shmoradims

singlis added 2 commits April 18, 2019 18:19

XML documentation for Normalizer

e5b9fc1

Tracked in dotnet#3204

- Fixing space

f988be2

singlis added the documentation Related to documentation of ML.NET label Apr 19, 2019

singlis requested review from codemzs, natke, wschin, yaeldekel, artidoro and shmoradims April 19, 2019 01:25

singlis self-assigned this Apr 19, 2019

natke suggested changes Apr 19, 2019

View reviewed changes

shmoradims reviewed Apr 19, 2019

View reviewed changes

singlis added 2 commits April 19, 2019 11:16

- Updating from feedback.

9a90f5b

- Updating description, intervals are now in latex.

e331364

natke approved these changes Apr 19, 2019

View reviewed changes

Ivanidzo4ka reviewed Apr 19, 2019

View reviewed changes

- Addressing further feedback

5eee846

wschin reviewed Apr 19, 2019

View reviewed changes

wschin approved these changes Apr 19, 2019

View reviewed changes

- update from feedback

c3adf94

shmoradims approved these changes Apr 19, 2019

View reviewed changes

singlis merged commit 270df4f into dotnet:master Apr 20, 2019

sfilipi mentioned this pull request Apr 20, 2019

API reference - XML documentation template for transforms #3204

Closed

ghost locked as resolved and limited conversation to collaborators Mar 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

XML documentation for Normalizer #3432

XML documentation for Normalizer #3432

singlis commented Apr 19, 2019

codecov bot commented Apr 19, 2019 •

edited

Loading

natke Apr 19, 2019 •

edited by singlis

Loading

singlis Apr 19, 2019

natke Apr 19, 2019 •

edited by singlis

Loading

singlis Apr 19, 2019

shmoradims Apr 19, 2019 •

edited by singlis

Loading

shmoradims Apr 19, 2019 •

edited by singlis

Loading

singlis Apr 19, 2019

shmoradims Apr 19, 2019

Ivanidzo4ka Apr 19, 2019

singlis Apr 19, 2019

shmoradims Apr 19, 2019 •

edited by singlis

Loading

Ivanidzo4ka Apr 19, 2019

singlis Apr 19, 2019

singlis Apr 19, 2019

Ivanidzo4ka Apr 19, 2019

singlis Apr 19, 2019

wschin Apr 19, 2019 •

edited by singlis

Loading

singlis Apr 19, 2019

wschin Apr 19, 2019 •

edited by singlis

Loading

singlis Apr 19, 2019

shmoradims left a comment

	/// \| Input column data type \| <xref:System.Single> or<xref:System.Double> or a known-sized vector of those types. \|
	/// \| Input column data type \| <xref:System.Single> or <xref:System.Double> or a known-sized vector of those types. \|
	``` #Resolved

XML documentation for Normalizer #3432

XML documentation for Normalizer #3432

Conversation

singlis commented Apr 19, 2019

codecov bot commented Apr 19, 2019 • edited Loading

Codecov Report

natke Apr 19, 2019 • edited by singlis Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

natke Apr 19, 2019 • edited by singlis Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shmoradims Apr 19, 2019 • edited by singlis Loading

Choose a reason for hiding this comment

shmoradims Apr 19, 2019 • edited by singlis Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shmoradims Apr 19, 2019 • edited by singlis Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wschin Apr 19, 2019 • edited by singlis Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wschin Apr 19, 2019 • edited by singlis Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shmoradims left a comment

Choose a reason for hiding this comment

codecov bot commented Apr 19, 2019 •

edited

Loading

natke Apr 19, 2019 •

edited by singlis

Loading

natke Apr 19, 2019 •

edited by singlis

Loading

shmoradims Apr 19, 2019 •

edited by singlis

Loading

shmoradims Apr 19, 2019 •

edited by singlis

Loading

shmoradims Apr 19, 2019 •

edited by singlis

Loading

wschin Apr 19, 2019 •

edited by singlis

Loading

wschin Apr 19, 2019 •

edited by singlis

Loading