Categorical estimator #899

Ivanidzo4ka · 2018-09-12T22:02:13Z

Converts Categorical transform to Estimator chain

Ivanidzo4ka · 2018-09-13T00:10:43Z

src/Microsoft.ML.Transforms/CategoricalStaticExtensions.tt

+#>
+        /// </summary>
+        /// <param name="input">The input column.</param>
+        /// <param name="outputKind"></param>


put description #Closed

Ivanidzo4ka · 2018-09-13T00:10:56Z

src/Microsoft.ML.Transforms/CategoricalStaticExtensions.tt

+        /// We are inputting a key type with values, and in that case the dictionary is considered to be built over the
+        /// values of the keys, rather than the keys themselves. This also mean the key-values learned for the output
+        /// will be a subset of the key-values in the input.
+<# }


replace term summary to categorical #Closed

Ivanidzo4ka · 2018-09-13T00:15:38Z

src/Microsoft.ML.Transforms/CategoricalStaticExtensions.tt

+        /// <param name="maxItems">The maximum number of items.</param>
+
+        /// <returns>The key-valued column.</returns>
+        public static <#=isVarVector?"Var":""#>Vector<float> OneHotEncoding<#=inputIsKey?"<T>":""#>(this <#=omitInputArity?"":arityName+"<"#><#=inputIsKey?"Key<T, ":""#><#=typeName#>><#=inputIsKey&&!isScalar?">":""#> input,


isVarVector [](start = 25, length = 11)

I need to handle situation in which VarVector becomes Vector if user choose bagging output.
Only possible solution coming to my mind so far is to have OneHotEncodingBagged and OneHotEncoding methods and expose enumeration without bagging option #Closed

Ivanidzo4ka · 2018-09-13T00:15:53Z

test/Microsoft.ML.Benchmarks/KMeansAndLogisticRegressionBench.cs

-                        new CategoricalTransform.Column { Name = "CatFeatures", Source = "CatFeatures" }
-                    }
-                }, loader);
+                IDataView trans = CategoricalTransform.Create(env, loader, new CategoricalEstimator.ColumnInfo("CatFeatures", "CatFeaures"));


CatFeaures [](start = 127, length = 10)

CatFeatures #Closed

Ivanidzo4ka · 2018-09-13T00:18:24Z

test/Microsoft.ML.Tests/Transformers/CategoricalTests.cs

+        }
+
+
+        [Fact]


lines #Closed

Ivanidzo4ka · 2018-09-13T00:18:58Z

src/Microsoft.ML.Transforms/CategoricalTransform.cs

@@ -14,6 +14,8 @@
 using Microsoft.ML.Runtime.EntryPoints;
 using Microsoft.ML.Runtime.Internal.Utilities;
 using Microsoft.ML.Runtime.Internal.Internallearn;
+using Microsoft.ML.Core.Data;
+using Microsoft.ML.Data.StaticPipe.Runtime;


order #Closed

Ivanidzo4ka · 2018-09-13T00:19:09Z

using Float = System.Single;

can be removed #Closed

Refers to: src/Microsoft.ML.Transforms/CategoricalTransform.cs:5 in ac91db8. [](commit_id = ac91db8, deletion_comment = False)

Ivanidzo4ka · 2018-09-13T00:19:37Z

src/Microsoft.ML.Transforms/CategoricalTransform.cs

+                col.SetTerms(column.Terms);
+                columns.Add(col);
+            }
+            return Create(env, input, columns.ToArray()) as IDataTransform;


as IDataTransform [](start = 56, length = 18)

I don't like it, but I don't see any other options. #Closed

yea we have to do this for now

In reply to: 217228738 [](ancestors = 217228738)

Ivanidzo4ka · 2018-09-13T00:20:04Z

src/Microsoft.ML.Transforms/CategoricalTransform.cs

+        public class ColumnInfo : TermTransform.ColumnInfo
+        {
+            public readonly CategoricalTransform.OutputKind OutputKind;
+            public ColumnInfo(string input, string output, CategoricalTransform.OutputKind outputKind = Defaults.OutKind, int maxNumTerms = TermEstimator.Defaults.MaxNumTerms, TermTransform.SortOrder sort = TermEstimator.Defaults.Sort, string[] term = null)


CategoricalTransform.OutputKind outputKind = Defaults.OutKin [](start = 59, length = 60)

shorter lines #Closed

Ivanidzo4ka · 2018-09-13T00:21:02Z

src/Microsoft.ML.Transforms/CategoricalTransform.cs

                }
+                var col = new KeyToVectorTransform.ColumnInfo(column.Output, column.Output, bag);


KeyToVectorTransform.ColumnInfo(column.Output, column.Output, bag) [](start = 30, length = 66)

can be just tuple #Closed

Ivanidzo4ka · 2018-09-13T00:21:27Z

src/Microsoft.ML.Transforms/CategoricalTransform.cs

+    {
+        // I am not certain I see a good way to cover the distinct types beyond complete enumeration.
+        // Raw generics would allow illegal possible inputs, e.g., Scalar<Bitmap>. So, this is a partial
+        // class, and all the public facing extension methods for each possible type are in a T4 generated result.


remove it #Closed

Ivanidzo4ka · 2018-09-13T00:22:15Z

src/Microsoft.ML.Transforms/CategoricalTransform.cs

+            }
+        }
+
+        private interface IOneHotCol


OneHot [](start = 27, length = 6)

Categorical probably is better internal name #Closed

Ivanidzo4ka · 2018-09-13T00:23:39Z

src/Microsoft.ML.Transforms/CategoricalHashTransform.cs

@@ -13,6 +13,7 @@
 using Microsoft.ML.Runtime.EntryPoints;
 using Microsoft.ML.Runtime.Internal.Utilities;
 using Microsoft.ML.Runtime.Internal.Internallearn;
+using System.Collections.Generic;


order #Closed

Ivanidzo4ka · 2018-09-13T00:24:37Z

src/Microsoft.ML.Transforms/CategoricalHashTransform.cs

                    args);
            }
        }
+
+        public static IDataTransform CreateTransformCore(CategoricalTransform.OutputKind argsOutputKind, OneToOneColumn[] columns,


public [](start = 8, length = 6)

Don't have to be public (and static, but I doubt its worth effort to refactor it if we plan to replace this code with estimators soon) #Resolved

why is it needed now?

In reply to: 217229386 [](ancestors = 217229386)

Well we still have this CategoricalHash transform which put HashTransform instead of TermTransform, so this is why I moved it into CatHash. I would like to kill it, but we gonna do it anyway, but first we need to convert HashTransform to estimators.

In reply to: 217236901 [](ancestors = 217236901,217229386)

Ivanidzo4ka · 2018-09-13T00:25:32Z

src/Microsoft.ML.Data/Transforms/TermEstimator.cs

 using System.Linq;

 namespace Microsoft.ML.Runtime.Data
 {
    public sealed class TermEstimator : IEstimator<TermTransform>
    {
+        public static class Defaults


Defaults [](start = 28, length = 8)

Is this a right way to do? #Closed

sounds good to me

In reply to: 217229486 [](ancestors = 217229486)

Zruty0 · 2018-09-13T01:20:05Z

src/Microsoft.ML.Transforms/CategoricalHashTransform.cs

+                            bag = true;
+                            break;
+                    }
+                    var col = new KeyToVectorTransform.Column();


Column [](start = 55, length = 6)

object initializer? #ByDesign

It's gonna be dead after hash transform become estimator.

In reply to: 217236161 [](ancestors = 217236161)

Zruty0 · 2018-09-13T01:20:29Z

src/Microsoft.ML.Transforms/CategoricalStaticExtensions.cs

+{
+    public static partial class CategoricalStaticExtensions
+    {
+        // Do not edit this file directly. Rather, it is generated out of TermStaticExtensions.tt.


Term [](start = 74, length = 4)

typo #Closed

Zruty0 · 2018-09-13T01:24:15Z

src/Microsoft.ML.Transforms/CategoricalTransform.cs

+            return Create(env, input, columns.ToArray()) as IDataTransform;
+        }
+
+        public static IDataView Create(IHostEnvironment env, IDataView input, params CategoricalEstimator.ColumnInfo[] columns)


Create [](start = 32, length = 6)

that doesn't serve any purpose, does it? Just make the callers do line 149, it's cleaner #Closed

Zruty0 · 2018-09-13T01:25:01Z

src/Microsoft.ML.Transforms/CategoricalTransform.cs

+
+        public SchemaShape GetOutputSchema(SchemaShape inputSchema)
+        {
+            return _estimatorChain.GetOutputSchema(inputSchema);


return [](start = 12, length = 6)

=> #Closed

Zruty0 · 2018-09-13T01:25:29Z

src/Microsoft.ML.Transforms/CategoricalTransform.cs

@@ -301,4 +307,109 @@ public static CommonOutputs.TransformOutput KeyToText(IHostEnvironment env, KeyT
            return new CommonOutputs.TransformOutput { Model = new TransformModel(env, xf, input.Data), OutputData = xf };
        }
    }
+
+    public enum OneHotOutputKind : byte


OneHotOutputKind [](start = 16, length = 16)

why is it at the top level? #Closed

Zruty0 · 2018-09-13T01:27:15Z

test/Microsoft.ML.Benchmarks/KMeansAndLogisticRegressionBench.cs

-                        new CategoricalTransform.Column { Name = "CatFeatures", Source = "CatFeatures" }
-                    }
-                }, loader);
+                IDataView trans = CategoricalTransform.Create(env, loader, new CategoricalEstimator.ColumnInfo("CatFeatures", "CatFeaures"));


CategoricalTransform [](start = 34, length = 20)

can we have a convenience ctor here that takes params string[] columns or something? #Closed

Zruty0

🕐

Zruty0 · 2018-09-13T19:56:23Z

    /// A helper method to create <see cref="CategoricalTransform"/> for public facing API.

it is NOT for public facing API. IDataTransform is to be deprecated. It is not too much work to write line 129 #Resolved

Refers to: src/Microsoft.ML.Transforms/CategoricalTransform.cs:119 in 114fec9. [](commit_id = 114fec9, deletion_comment = False)

Zruty0 · 2018-09-13T19:58:34Z

src/Microsoft.ML.Transforms/CategoricalTransform.cs

+
+        /// <summary>
+        /// The categorical transform operates on text columns. During construction, it passes through the data to build a dictionary of categories.
+        /// It does not do any parsing to the text; for each row it sees, the whole text appearing in the input column is defined as a category.


all of this is implementation, and should not be present #Resolved

Zruty0 · 2018-09-13T19:58:40Z

src/Microsoft.ML.Transforms/CategoricalTransform.cs

+        /// The output of the categorical transform is an indicator vector. Each slot in this vector corresponds to a category in the dictionary
+        /// (thus, its length is the size of the built dictionary). In each row, it contains a 1 in the slot corresponding to the category in that row,
+        /// and 0 in the rest of the slots.
+        /// </summary>


summary [](start = 14, length = 7)

add params description #Resolved

Zruty0

TomFinley · 2018-09-14T00:08:16Z

src/Microsoft.ML.Transforms/CategoricalTransform.cs

+        /// <param name="outputKind">Specify output type of indicator array: Multiarray, array or binary encoded data.</param>
+        /// <param name="order">How Id for each value would be assigined: by occurence or by value.</param>
+        /// <param name="maxItems">Maximum number of ids to keep during data scanning.</param>
+        public static Vector<float> OneHotEncoding(this Scalar<string> input, OneHotOutputKind outputKind = DefOut, KeyValueOrder order = DefSort, int maxItems = DefMax)


OneHotEncoding [](start = 36, length = 14)

It strikes me that the most useful thing that we could get out of this via an onFit delegate would be the mapping, so people can know what the features actually mean. #Closed

This could be the same type that is from the OnTerm transform... then once that is properly implemented both with benefit.

In reply to: 217569940 [](ancestors = 217569940)

I added onFit delegate, let me know is this enough for now.

In reply to: 217570555 [](ancestors = 217570555,217569940)

Good thanks Ivan.

In reply to: 218176803 [](ancestors = 218176803,217570555,217569940)

TomFinley · 2018-09-14T00:10:05Z

src/Microsoft.ML.Transforms/CategoricalTransform.cs

+        /// Converts the categorical value into an indicator array by building a dictionary of categories based on the data and using the id in the dictionary as the index in the array
+        /// </summary>
+        /// <param name="input">Incoming data.</param>
+        /// <param name="outputKind">Specify output type of indicator array: Multiarray, array or binary encoded data.</param>


Multiarray, array [](start = 77, length = 17)

In the scalar case there is no distinction between these two, so presenting this as an option is at least confusing. Is it worthwhile to clarify this? #Closed

TomFinley · 2018-09-14T00:10:30Z

src/Microsoft.ML.Transforms/CategoricalTransform.cs

+        /// </summary>
+        /// <param name="input">Incoming data.</param>
+        /// <param name="outputKind">Specify output type of indicator array: Multiarray, array or binary encoded data.</param>
+        /// <param name="order">How Id for each value would be assigined: by occurence or by value.</param>


occurence [](start = 77, length = 9)

occurrence, not occurence? #Closed

TomFinley · 2018-09-14T00:11:03Z

src/Microsoft.ML.Transforms/Microsoft.ML.Transforms.csproj

@@ -65,4 +65,8 @@
    </EmbeddedResource>
  </ItemGroup>

+  <ItemGroup>
+    <Service Include="{508349b6-6b84-4df5-91f0-309beebad82d}" />
+  </ItemGroup>


Hmmm. What's this for? #Closed

TomFinley · 2018-09-14T00:14:25Z

src/Microsoft.ML.Transforms/CategoricalTransform.cs

-            CategoricalHashTransform.Arguments catHashArgs = null)
+    }
+
+    public sealed class CategoricalEstimator : IEstimator<ITransformer>


IEstimator [](start = 47, length = 24)

Should not the categorical estimator have a more specific type? We have something that we want to publish, do we not? We want something that is (ultimately) capable of exposing whatever ToKey mapping was learnt in the first step. This strikes me as the only real reason why we would actually have the complication of an implemented type in the first place, am I wrong? #Closed

Is this look good for you?

In reply to: 217570810 [](ancestors = 217570810)

Yes thanks.

In reply to: 218176602 [](ancestors = 218176602,217570810)

TomFinley

need pigsty

d68e59b

Ivanidzo4ka self-assigned this Sep 12, 2018

Ivanidzo4ka added the API Issues pertaining the friendly API label Sep 12, 2018

Ivanidzo4ka added this to the 0918 milestone Sep 12, 2018

tests, somewhat pigsty

ac91db8

Ivanidzo4ka commented Sep 13, 2018

View reviewed changes

test/Microsoft.ML.Tests/Transformers/CategoricalTests.cs Outdated

}

[Fact]

Copy link

Contributor Author

Ivanidzo4ka Sep 13, 2018 •

edited

Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lines #Closed

Ivanidzo4ka commented Sep 13, 2018

View reviewed changes

Ivanidzo4ka requested review from TomFinley, Zruty0 and sfilipi September 13, 2018 00:27

Zruty0 reviewed Sep 13, 2018

View reviewed changes

Zruty0 suggested changes Sep 13, 2018

View reviewed changes

Ivan Matantsev added 2 commits September 13, 2018 11:06

More polishing

2570eb9

some debt

8174f3b

Ivanidzo4ka changed the title ~~WIP Categorical estimator~~ Categorical estimator Sep 13, 2018

fix test

114fec9

Zruty0 reviewed Sep 13, 2018

View reviewed changes

Zruty0 approved these changes Sep 13, 2018

View reviewed changes

minor comments

9d51927

TomFinley reviewed Sep 14, 2018

View reviewed changes

Ivan Matantsev added 6 commits September 14, 2018 11:49

Merge branch 'master' into ivanidze/CategoricalEstimator

4711a8d

pre onFit changes

120d3e8

update

cbbf5d0

merge with master

47a00a9

on fit in categorical transform

7bfedf5

one transformer to rule them all

d2d2a88

TomFinley approved these changes Sep 17, 2018

View reviewed changes

Ivan Matantsev added 5 commits September 17, 2018 13:03

trigger build

c645cbd

save CategoricalTransform

486d65a

Merge branch 'master' into ivanidze/CategoricalEstimator

49c5514

merge with master

71c1d45

fix attributes

a397200

Ivanidzo4ka merged commit 5e2ed11 into dotnet:master Sep 18, 2018

sfilipi mentioned this pull request Oct 10, 2018

API reference - Samples for Transforms #1209

Closed

abgoswam mentioned this pull request Jan 29, 2019

Added a test showing example of text classification using TensorFlow in ML.Net #2302

Merged

ghost locked as resolved and limited conversation to collaborators Mar 29, 2022

		}
		var col = new KeyToVectorTransform.ColumnInfo(column.Output, column.Output, bag);

Categorical estimator #899

Categorical estimator #899

Conversation

Ivanidzo4ka commented Sep 12, 2018 • edited Loading

Ivanidzo4ka Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka commented Sep 13, 2018 • edited Loading

Ivanidzo4ka Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ivanidzo4ka Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ivanidzo4ka Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zruty0 Sep 13, 2018 • edited by Ivanidzo4ka Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zruty0 Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Zruty0 Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Zruty0 Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Zruty0 Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Zruty0 Sep 13, 2018 • edited by Ivanidzo4ka Loading

Choose a reason for hiding this comment

Zruty0 left a comment

Choose a reason for hiding this comment

Zruty0 commented Sep 13, 2018 • edited by Ivanidzo4ka Loading

Zruty0 Sep 13, 2018 • edited by Ivanidzo4ka Loading

Choose a reason for hiding this comment

Zruty0 Sep 13, 2018 • edited by Ivanidzo4ka Loading

Choose a reason for hiding this comment

Zruty0 left a comment

Choose a reason for hiding this comment

TomFinley Sep 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomFinley Sep 14, 2018 • edited Loading

Choose a reason for hiding this comment

TomFinley Sep 14, 2018 • edited Loading

Choose a reason for hiding this comment

TomFinley Sep 14, 2018 • edited Loading

Choose a reason for hiding this comment

TomFinley Sep 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomFinley left a comment

Choose a reason for hiding this comment

Ivanidzo4ka commented Sep 12, 2018 •

edited

Loading

Ivanidzo4ka Sep 13, 2018 •

edited

Loading

Ivanidzo4ka Sep 13, 2018 •

edited

Loading

Ivanidzo4ka Sep 13, 2018 •

edited

Loading

Ivanidzo4ka Sep 13, 2018 •

edited

Loading

Ivanidzo4ka Sep 13, 2018 •

edited

Loading

Ivanidzo4ka Sep 13, 2018 •

edited

Loading

Ivanidzo4ka commented Sep 13, 2018 •

edited

Loading

Ivanidzo4ka Sep 13, 2018 •

edited

Loading

Ivanidzo4ka Sep 13, 2018 •

edited

Loading

Ivanidzo4ka Sep 13, 2018 •

edited

Loading

Ivanidzo4ka Sep 13, 2018 •

edited

Loading

Ivanidzo4ka Sep 13, 2018 •

edited

Loading

Ivanidzo4ka Sep 13, 2018 •

edited

Loading

Ivanidzo4ka Sep 13, 2018 •

edited

Loading

Ivanidzo4ka Sep 13, 2018 •

edited

Loading

Zruty0 Sep 13, 2018 •

edited by Ivanidzo4ka

Loading

Zruty0 Sep 13, 2018 •

edited

Loading

Zruty0 Sep 13, 2018 •

edited

Loading

Zruty0 Sep 13, 2018 •

edited

Loading

Zruty0 Sep 13, 2018 •

edited

Loading

Zruty0 Sep 13, 2018 •

edited by Ivanidzo4ka

Loading

Zruty0 commented Sep 13, 2018 •

edited by Ivanidzo4ka

Loading

Zruty0 Sep 13, 2018 •

edited by Ivanidzo4ka

Loading

Zruty0 Sep 13, 2018 •

edited by Ivanidzo4ka

Loading

TomFinley Sep 14, 2018 •

edited

Loading

TomFinley Sep 14, 2018 •

edited

Loading

TomFinley Sep 14, 2018 •

edited

Loading

TomFinley Sep 14, 2018 •

edited

Loading

TomFinley Sep 14, 2018 •

edited

Loading