Towards #3204 - Conversion's catalog #3394

sfilipi · 2019-04-18T06:45:19Z

Polishes the documentation for the conversions catalog, adhering to the template in #3204

Marking it as not final until I do a second pass at the changes through CodeFlow.

…plate.

codecov · 2019-04-18T07:23:49Z

Codecov Report

Merging #3394 into master will increase coverage by 0.03%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3394      +/-   ##
==========================================
+ Coverage   72.69%   72.73%   +0.03%     
==========================================
  Files         807      807              
  Lines      145172   145206      +34     
  Branches    16225    16230       +5     
==========================================
+ Hits       105533   105615      +82     
+ Misses      35223    35174      -49     
- Partials     4416     4417       +1

Flag	Coverage Δ
#Debug	`72.73% <ø> (+0.03%)`	⬆️
#production	`68.27% <ø> (+0.04%)`	⬆️
#test	`88.98% <ø> (ø)`	⬆️

Impacted Files	Coverage Δ
src/Microsoft.ML.Data/Transforms/TypeConverting.cs	`74.21% <ø> (ø)`	⬆️
src/Microsoft.ML.Data/Transforms/Hashing.cs	`92.32% <ø> (+0.14%)`	⬆️
...t.ML.Data/Transforms/ValueToKeyMappingEstimator.cs	`88.67% <ø> (ø)`	⬆️
src/Microsoft.ML.Data/Transforms/KeyToValue.cs	`79.16% <ø> (-0.65%)`	⬇️
src/Microsoft.ML.Data/Transforms/KeyToVector.cs	`81.81% <ø> (ø)`	⬆️
...ML.Data/Transforms/ConversionsExtensionsCatalog.cs	`64.07% <ø> (ø)`	⬆️
src/Microsoft.ML.Data/Transforms/ValueMapping.cs	`84.26% <ø> (ø)`	⬆️
...ML.Data/Transforms/ValueToKeyMappingTransformer.cs	`78.38% <ø> (ø)`	⬆️
...ML.Transforms/MutualInformationFeatureSelection.cs	`78.58% <0%> (-0.33%)`	⬇️
... and 23 more

sfilipi · 2019-04-18T17:29:40Z

src/Microsoft.ML.Data/Transforms/ConversionsExtensionsCatalog.cs

-        /// <param name="inputColumnName">Name of the column to transform. If set to <see langword="null"/>, the value of the <paramref name="outputColumnName"/> will be used as source.</param>
+        /// <param name="outputColumnName">Name of the column resulting from the transformation of <paramref name="inputColumnName"/>.
+        /// This column's data type will be a vector of <see cref="System.UInt32"/>, or a <see cref="System.UInt32"/> based on whether the input column data types
+        /// are vectors or scalars.</param>


scalars [](start = 27, length = 7)

scalars or just primitives? @eerhardt @TomFinley

sfilipi · 2019-04-18T17:31:03Z

src/Microsoft.ML.Data/Transforms/ConversionsExtensionsCatalog.cs

-        /// <param name="columns">Description of dataset columns and how to process them.</param>
+        /// <remarks>This transform can operate over several columns.</remarks>
+        /// <param name="catalog">The transform's catalog.</param>
+        /// <param name="columns">The pairs of input and output columns.


airs of [](start = 39, length = 7)

remove #Resolved

sfilipi · 2019-04-18T17:33:07Z

src/Microsoft.ML.Data/Transforms/ConversionsExtensionsCatalog.cs

+        /// <param name="columns">The pairs of input and output columns.
+        /// This estimator operates over text, numeric, boolean and <see cref="DataViewRowId"/> data types.
+        /// The new column's data type will be a vector of <see cref="System.UInt32"/>, or a <see cref="System.UInt32"/> based on whether the input column data types
+        /// were vectors or scalars.</param>


were [](start = 12, length = 4)

are #Resolved

sfilipi · 2019-04-18T17:34:53Z

src/Microsoft.ML.Data/Transforms/ConversionsExtensionsCatalog.cs

-        /// <param name="columns">Description of dataset columns and how to process them.</param>
+        /// <remarks>This transform can operate over several columns.</remarks>
+        /// <param name="catalog">The transform's catalog.</param>
+        /// <param name="columns">The pairs of input and output columns.


The pairs of input and output columns. [](start = 34, length = 38)

The input and output columns. #Resolved

sfilipi · 2019-04-18T17:40:01Z

src/Microsoft.ML.Data/Transforms/ConversionsExtensionsCatalog.cs

        [BestFriend]
        internal static HashingEstimator Hash(this TransformsCatalog.ConversionTransforms catalog, params HashingEstimator.ColumnOptions[] columns)
            => new HashingEstimator(CatalogUtils.GetEnvironment(catalog), columns);

        /// <summary>
-        /// Changes column type of the input column.
+        /// Create a <see cref="TypeConvertingEstimator"/>, which convertes the type of the data to the type specified in <paramref name="outputKind"/>.


convertes [](start = 66, length = 9)

converts #Resolved

sfilipi · 2019-04-18T17:42:27Z

src/Microsoft.ML.Data/Transforms/ConversionsExtensionsCatalog.cs

@@ -114,10 +134,15 @@ public static KeyToValueMappingEstimator MapKeyToValue(this TransformsCatalog.Co
            => new KeyToValueMappingEstimator(CatalogUtils.GetEnvironment(catalog), outputColumnName, inputColumnName);

        /// <summary>
-        /// Convert the key types back to their original values.
+        /// Create a <see cref="KeyToValueMappingEstimator"/>, which convert the key types back to their original values.


convert [](start = 69, length = 7)

converts #Resolved

sfilipi · 2019-04-18T17:51:36Z

src/Microsoft.ML.Data/Transforms/ConversionsExtensionsCatalog.cs

@@ -133,21 +158,24 @@ public static KeyToValueMappingEstimator MapKeyToValue(this TransformsCatalog.Co
        }

        /// <summary>
-        /// Maps key types or key values into a floating point vector.
+        /// Create a <see cref="KeyToVectorMappingEstimator"/>, which maps key types or key values into a floating point vector.


which maps key types or key values into a floating point vector. [](start = 64, length = 64)

which maps the value of into a floating point vector representing the value. #Resolved

sfilipi · 2019-04-18T18:04:47Z

src/Microsoft.ML.Data/Transforms/ConversionsExtensionsCatalog.cs

@@ -326,15 +370,17 @@ public static KeyToValueMappingEstimator MapKeyToValue(this TransformsCatalog.Co
        }

        /// <summary>
-        /// <see cref="ValueMappingEstimator"/>
+        /// Create a <see cref="ValueMappingEstimator"/>, which onverts value types into <see cref="KeyDataViewType"/>, loading the keys to use from <paramref name="keyValuePairs"/>.


onverts [](start = 64, length = 7)

converts #Resolved

sfilipi · 2019-04-18T18:12:06Z

src/Microsoft.ML.Data/Transforms/ConversionsExtensionsCatalog.cs

+        /// <remarks>This transform can operate over several columns.</remarks>
+        /// <param name="catalog">The transform's catalog.</param>
+        /// <param name="columns">The pairs of input and output columns.
+        /// This estimator operates over text, numeric, boolean and <see cref="DataViewRowId"/> data types.


over text, numeric, boolean and data types [](start = 36, length = 70)

add Key #Resolved

sfilipi · 2019-04-18T18:18:54Z

src/Microsoft.ML.Data/Transforms/KeyToValue.cs

+    /// |  |  |
+    /// | -- | -- |
+    /// | Does this estimator need to look at the data to train its parameters? | No |
+    /// | Input column data type | KeyDataViewType |


KeyDataViewType [](start = 35, length = 15)

xref #Closed

sfilipi · 2019-04-18T18:19:06Z

src/Microsoft.ML.Data/Transforms/Hashing.cs

+    /// |  |  |
+    /// | -- | -- |
+    /// | Does this estimator need to look at the data to train its parameters? | Yes, if the mapping of the hashes to the values is required. |
+    /// | Input column data type | Vector or primitive of text or [KeyDataViewType](xref:Microsoft.ML.Data.KeyDataViewType) data types.|


or primitive of text or KeyDataViewType data typ [](start = 41, length = 91)

add all the other types #Resolved

artidoro · 2019-04-18T19:49:17Z

src/Microsoft.ML.Data/Transforms/ConversionsExtensionsCatalog.cs

-        /// <param name="outputColumnName">Name of the column resulting from the transformation of <paramref name="inputColumnName"/>.</param>
-        /// <param name="inputColumnName">Name of the column to transform. If set to <see langword="null"/>, the value of the <paramref name="outputColumnName"/> will be used as source.</param>
+        /// <param name="outputColumnName">Name of the column resulting from the transformation of <paramref name="inputColumnName"/>.
+        /// This column's data type will be a vector of <see cref="System.UInt32"/>, or a <see cref="System.UInt32"/> based on whether the input column data types


[](start = 56, length = 27)

Is it more useful if we say that these are keys? As in KeyDataViewType? #Resolved

think it is more correct to leave it to ints. You're right on them being effectively keys though, but as far as the underlying data type, it is UInt32

In reply to: 276808872 [](ancestors = 276808872)

shmoradims · 2019-04-18T21:25:53Z

src/Microsoft.ML.Data/Transforms/ConversionsExtensionsCatalog.cs

+        /// are vectors or scalars.</param>
+        /// <param name="inputColumnName">Name of the column whose data will be hashed.
+        /// If set to <see langword="null"/>, the value of the <paramref name="outputColumnName"/> will be used as source.
+        /// This estimator operates over text, numeric, boolean, <see cref="KeyDataViewType"/> or <see cref="DataViewRowId"/> data types. </param>


[](start = 65, length = 29)

just 'key' or 'key type' #Pending

shmoradims · 2019-04-18T21:34:33Z

src/Microsoft.ML.Data/Transforms/ConversionsExtensionsCatalog.cs

+        /// are vectors or scalars.</param>
+        /// <param name="inputColumnName">Name of the column whose data will be hashed.
+        /// If set to <see langword="null"/>, the value of the <paramref name="outputColumnName"/> will be used as source.
+        /// This estimator operates over text, numeric, boolean, <see cref="KeyDataViewType"/> or <see cref="DataViewRowId"/> data types. </param>


DataViewRowId [](start = 109, length = 13)

do we need to mention DataViewRowId? is it group id? we should ask Tom about a phrase description for it.
#Pending

I don't think it is group id, it is the idv row id, the Uint128

In reply to: 276842426 [](ancestors = 276842426)

shmoradims · 2019-04-18T21:40:03Z

src/Microsoft.ML.Data/Transforms/ConversionsExtensionsCatalog.cs

        [BestFriend]
        internal static HashingEstimator Hash(this TransformsCatalog.ConversionTransforms catalog, params HashingEstimator.ColumnOptions[] columns)
            => new HashingEstimator(CatalogUtils.GetEnvironment(catalog), columns);

        /// <summary>
-        /// Changes column type of the input column.
+        /// Create a <see cref="TypeConvertingEstimator"/>, which converts the type of the data to the type specified in <paramref name="outputKind"/>.


the type of the data [](start = 75, length = 20)

the input column's data type #Resolved

shmoradims · 2019-04-18T21:40:26Z

src/Microsoft.ML.Data/Transforms/ConversionsExtensionsCatalog.cs

        /// </summary>
        /// <param name="catalog">The conversion transform's catalog.</param>
        /// <param name="outputColumnName">Name of the column resulting from the transformation of <paramref name="inputColumnName"/>.</param>
-        /// <param name="inputColumnName">Name of the column to transform. If set to <see langword="null"/>, the value of the <paramref name="outputColumnName"/> will be used as source.</param>
+        /// <param name="inputColumnName">Name of the column to transform. If set to <see langword="null"/>, the value of the <paramref name="outputColumnName"/> will be used as source.
+        /// This transform operates over numeric, boolean, text, <see cref="System.DateTime"/> and <see cref="KeyDataViewType"/> data types.</param>


KeyDataViewType [](start = 110, length = 15)

key #Resolved

shmoradims · 2019-04-18T21:44:22Z

src/Microsoft.ML.Data/Transforms/Hashing.cs

+    /// |  |  |
+    /// | -- | -- |
+    /// | Does this estimator need to look at the data to train its parameters? | Yes, if the mapping of the hashes to the values is required. |
+    /// | Input column data type | Vector or scalars of numeric, boolean, text, [DateTime](xref: System.DateTime) and [KeyDataViewType](xref:Microsoft.ML.Data.KeyDataViewType) data types.|


[DateTime](xref: System.DateTime) [](start = 80, length = 33)

you can just use automatic links: <xref: System.DateTime> to save yourself from having to write the anchor text #Resolved

shmoradims · 2019-04-18T21:45:49Z

src/Microsoft.ML.Data/Transforms/KeyToValue.cs

@@ -502,6 +499,24 @@ public override JToken SavePfa(BoundPfaContext ctx, JToken srcToken)
        }
    }

+    /// <summary>
+    /// Utilizes KeyValues <see cref="AnnotationInfo"/> of the input column, to map keys to the corresponding values.
+    /// Maps zero values of the <see cref="KeyDataViewType"/> are mapped to the <see langword="default"/> value of the output type.


Maps zero values of the are mapped [](start = 8, length = 64)

re-read: Maps .... are mapped to

#Closed

shmoradims · 2019-04-18T21:51:42Z

src/Microsoft.ML.Data/Transforms/KeyToValue.cs

+    /// |  |  |
+    /// | -- | -- |
+    /// | Does this estimator need to look at the data to train its parameters? | No |
+    /// | Input column data type | [KeyDataViewType](xref:Microsoft.ML.Data.KeyDataViewType) |


KeyDataViewType [](start = 36, length = 15)

so here we can use 'key type' as the anchor text. we're using raw types in the params requirement instead of IDV types. in this table we have a change to link to the actual class, which explains what we mean by 'key type'.

we're doing the same for text.
text
#Pending

how about numeric.. if we keep it to that, we'd have to spell out: float, double, int, long, uint, short, ushort.. doesn't seem feasible. How about i leave it to no references for text and numeric?

In reply to: 276847010 [](ancestors = 276847010)

natke · 2019-04-18T22:37:14Z

src/Microsoft.ML.Data/Transforms/Hashing.cs

@@ -1104,9 +1102,23 @@ public override void Process()
    }

    /// <summary>
-    /// Estimator for <see cref="HashingTransformer"/> which can hash either single valued columns or vector columns. For vector columns,
-    /// it hashes each slot separately. It can hash either text values or key values.
+    /// Estimator for <see cref="HashingTransformer"/>. hash either single valued columns or vector columns. For vector columns,


h -> H #Resolved

natke · 2019-04-18T22:50:31Z

src/Microsoft.ML.Data/Transforms/ValueMapping.cs

-    /// <include file='doc.xml' path='doc/members/member[@name="ValueMappingEstimator"]/*' />
+
+    /// <summary>
+    /// Estimator for <see cref="ValueMappingTransformer"/>. It is a 1-1 mapping from one value to another value.


Suggest: "Create a key-value map using the pairs of values in the input data" or something like that? #Resolved

natke · 2019-04-18T22:51:20Z

src/Microsoft.ML.Data/Transforms/ValueToKeyMappingEstimator.cs

@@ -76,7 +102,7 @@ internal abstract class ColumnOptionsBase
        internal sealed class ColumnOptions : ColumnOptionsBase
        {
            /// <summary>
-            /// Describes how the transformer handles one column pair.
+            /// Describes how the transformer handles onumn pair.


typo? #Resolved

sfilipi · 2019-04-18T22:52:10Z

src/Microsoft.ML.Data/Transforms/Hashing.cs

@@ -1104,9 +1102,23 @@ public override void Process()
    }

    /// <summary>
-    /// Estimator for <see cref="HashingTransformer"/> which can hash either single valued columns or vector columns. For vector columns,
-    /// it hashes each slot separately. It can hash either text values or key values.
+    /// Estimator for <see cref="HashingTransformer"/>. hash either single valued columns or vector columns. For vector columns,


h [](start = 56, length = 1)

uppercase #Resolved

Ivanidzo4ka · 2019-04-19T20:05:30Z

src/Microsoft.ML.Data/Transforms/ConversionsExtensionsCatalog.cs

-        /// <param name="outputColumnName">Name of the column resulting from the transformation of <paramref name="inputColumnName"/>.</param>
-        /// <param name="inputColumnName">Name of the column to transform. If set to <see langword="null"/>, the value of the <paramref name="outputColumnName"/> will be used as source.</param>
+        /// <param name="outputColumnName">Name of the column resulting from the transformation of <paramref name="inputColumnName"/>.
+        /// This column's data type will be a vector of <see cref="System.UInt32"/>, or a <see cref="System.UInt32"/> based on whether the input column data types


, or a [](start = 56, length = 61)

Can you see difference between <see cref="System.UInt32"/> and <see cref="System.UInt32"/> ? Because I don't. #Pending

awesome :)

In reply to: 277079450 [](ancestors = 277079450)

actually that is correct: a vector of ints or an int.

I can add scalar.

In reply to: 277099542 [](ancestors = 277099542,277079450)

shmoradims

sfilipi added 2 commits April 15, 2019 09:40

towards adapting the Conversions catalog documentation to the new tem…

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.

GPG key ID: 4AEE18F83AFDEB23
Expired

Verified
Learn about vigilant mode

7afbcce

…plate.

more formatting

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.

GPG key ID: 4AEE18F83AFDEB23
Expired

Verified
Learn about vigilant mode

e724147

sfilipi requested review from natke, shmoradims and singlis April 18, 2019 06:45

sfilipi changed the title ~~Towards #3204 - Conversion's catalog~~ WIP - Towards #3204 - Conversion's catalog Apr 18, 2019

sfilipi commented Apr 18, 2019

View reviewed changes

sfilipi requested review from Ivanidzo4ka and yaeldekel April 18, 2019 17:36

sfilipi commented Apr 18, 2019

View reviewed changes

self-review fixes

6008974

sfilipi changed the title ~~WIP - Towards #3204 - Conversion's catalog~~ Towards #3204 - Conversion's catalog Apr 18, 2019

artidoro reviewed Apr 18, 2019

View reviewed changes

sfilipi mentioned this pull request Apr 18, 2019

API reference - XML documentation template for transforms #3204

Closed

shmoradims reviewed Apr 18, 2019

View reviewed changes

natke approved these changes Apr 18, 2019

View reviewed changes

sfilipi commented Apr 18, 2019

View reviewed changes

review feedback.

9ed812c

Ivanidzo4ka reviewed Apr 19, 2019

View reviewed changes

shmoradims approved these changes Apr 19, 2019

View reviewed changes

addressing feedback

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.

GPG key ID: 4AEE18F83AFDEB23
Expired

Verified
Learn about vigilant mode

9df1090

sfilipi merged commit 1dd66f0 into dotnet:master Apr 20, 2019

sfilipi deleted the transformsDocFormat branch April 20, 2019 00:23

ghost locked as resolved and limited conversation to collaborators Mar 22, 2022

Towards #3204 - Conversion's catalog #3394

Towards #3204 - Conversion's catalog #3394

Conversation

sfilipi commented Apr 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Apr 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sfilipi Apr 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfilipi Apr 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfilipi Apr 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfilipi Apr 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfilipi Apr 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfilipi Apr 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfilipi Apr 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfilipi Apr 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfilipi Apr 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfilipi Apr 18, 2019 • edited by shmoradims Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfilipi Apr 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

artidoro Apr 18, 2019 • edited by sfilipi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfilipi Apr 19, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shmoradims Apr 18, 2019 • edited by sfilipi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shmoradims Apr 18, 2019 • edited by sfilipi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfilipi Apr 19, 2019

Choose a reason for hiding this comment

Uh oh!

shmoradims Apr 18, 2019 • edited by sfilipi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shmoradims Apr 18, 2019 • edited by sfilipi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

sfilipi commented Apr 18, 2019 •

edited

Loading

codecov bot commented Apr 18, 2019 •

edited

Loading

sfilipi Apr 18, 2019 •

edited

Loading

sfilipi Apr 18, 2019 •

edited

Loading

sfilipi Apr 18, 2019 •

edited

Loading

sfilipi Apr 18, 2019 •

edited

Loading

sfilipi Apr 18, 2019 •

edited

Loading

sfilipi Apr 18, 2019 •

edited

Loading

sfilipi Apr 18, 2019 •

edited

Loading

sfilipi Apr 18, 2019 •

edited

Loading

sfilipi Apr 18, 2019 •

edited

Loading

sfilipi Apr 18, 2019 •

edited by shmoradims

Loading

sfilipi Apr 18, 2019 •

edited

Loading

artidoro Apr 18, 2019 •

edited by sfilipi

Loading

sfilipi Apr 19, 2019 •

edited

Loading

shmoradims Apr 18, 2019 •

edited by sfilipi

Loading

shmoradims Apr 18, 2019 •

edited by sfilipi

Loading

shmoradims Apr 18, 2019 •

edited by sfilipi

Loading

shmoradims Apr 18, 2019 •

edited by sfilipi

Loading

shmoradims Apr 18, 2019 •

edited by sfilipi

Loading

shmoradims Apr 18, 2019 •

edited

Loading

shmoradims Apr 18, 2019 •

edited by sfilipi

Loading

natke Apr 18, 2019 •

edited by sfilipi

Loading

natke Apr 18, 2019 •

edited by sfilipi

Loading

natke Apr 18, 2019 •

edited by sfilipi

Loading

sfilipi Apr 18, 2019 •

edited

Loading

Ivanidzo4ka Apr 19, 2019 •

edited by sfilipi

Loading