Add an example of random PCA using in-memory data structure #2780

wschin · 2019-02-27T23:22:25Z

As title. It also shows some benefits of using in-memory data described in #2726.

Fix build Polish example

Fix build

codecov · 2019-02-28T01:58:58Z

Codecov Report

Merging #2780 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #2780      +/-   ##
==========================================
+ Coverage   71.66%   71.67%   +<.01%     
==========================================
  Files         809      809              
  Lines      142378   142416      +38     
  Branches    16119    16120       +1     
==========================================
+ Hits       102031   102072      +41     
+ Misses      35915    35912       -3     
  Partials     4432     4432

Flag	Coverage Δ
#Debug	`71.67% <100%> (ø)`	⬆️
#production	`67.91% <ø> (ø)`	⬆️
#test	`85.86% <100%> (+0.01%)`	⬆️

Impacted Files	Coverage Δ
src/Microsoft.ML.PCA/PCACatalog.cs	`92.3% <ø> (+38.46%)`	⬆️
test/Microsoft.ML.Tests/AnomalyDetectionTests.cs	`100% <100%> (ø)`	⬆️
src/Microsoft.ML.Maml/MAML.cs	`24.75% <0%> (-1.46%)`	⬇️
...StandardLearners/Standard/LinearModelParameters.cs	`60.63% <0%> (-0.27%)`	⬇️
src/Microsoft.ML.PCA/PcaTrainer.cs	`79.94% <0%> (+0.29%)`	⬆️
src/Microsoft.ML.CpuMath/EigenUtils.cs	`87.69% <0%> (+0.39%)`	⬆️

sfilipi · 2019-02-28T05:01:50Z

docs/samples/Microsoft.ML.Samples/Dynamic/Trainers/AnomalyDetection/RandomizedPcaSample.cs

+    {
+        /// <summary>
+        /// Example with 3 feature values.
+        /// </summary>


please don;t use XML style comments in samples, as it becomes harder to read than simple comments. #Resolved

My bad. Thanks.

In reply to: 261047221 [](ancestors = 261047221)

sfilipi · 2019-02-28T05:02:30Z

docs/samples/Microsoft.ML.Samples/Dynamic/Trainers/AnomalyDetection/RandomizedPcaSample.cs

+        /// Class used to capture prediction of <see cref="DataPoint"/> in <see cref="Example"/>.
+        /// </summary>
+        // We disable this warning because complier doesn't realize those fields below are assigned somewhere.
+#pragma warning disable 649


can we disable this somewhere else? csproj etc? #Resolved

Yes, we can do

<PropertyGroup> <TargetFramework>netcoreapp2.1</TargetFramework> <OutputType>Exe</OutputType> + <NoWarn>1701;1702;0649</NoWarn> </PropertyGroup>

In reply to: 261047307 [](ancestors = 261047307)

sfilipi · 2019-02-28T05:03:19Z

docs/samples/Microsoft.ML.Samples/Dynamic/Trainers/AnomalyDetection/RandomizedPcaSample.cs

+                new DataPoint(){ Features= new float[3] {-100, 50, -100} }
+            };
+
+            // Convert native C# class to IDataView, a consumble format to ML.NET functions.


native C# class [](start = 23, length = 15)

the List #Closed

Supper good to make it strongly-typed. No ambiguity anymore.

In reply to: 261047430 [](ancestors = 261047430)

sfilipi · 2019-02-28T05:03:59Z

docs/samples/Microsoft.ML.Samples/Dynamic/Trainers/AnomalyDetection/RandomizedPcaSample.cs

+            // Apply the trained model on the training data.
+            var transformed = model.Transform(data);
+
+            // Read ML.NET predictions into C# class.


C# class [](start = 44, length = 8)

an IEnumerable #Resolved

Thanks!

In reply to: 261047514 [](ancestors = 261047514)

sfilipi · 2019-02-28T05:04:54Z

docs/samples/Microsoft.ML.Samples/Dynamic/Trainers/AnomalyDetection/RandomizedPcaSample.cs

+                    // The i-th sample is predicted as an outlier.
+                    Console.WriteLine("The {0}-th example with features [{1}] is an outlier with a score of being inlier {2}",
+                        i, featuresInText, result.Score);
+            }


please put in comment below what would the output of this look. (The results of those WriteLine) #Closed

Sounds good. Please see iteration 3.

In reply to: 261047636 [](ancestors = 261047636)

sfilipi · 2019-02-28T05:06:35Z

src/Microsoft.ML.PCA/PCACatalog.cs

+        /// <example>
+        /// <format type="text/markdown">
+        /// <![CDATA[
+        ///  [!code-csharp[RPCA](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Trainers/AnomalyDetection/RandomizedPcaSampleWithOptions.cs)]


RandomizedPcaSampleWithOptions [](start = 119, length = 30)

i don't think this exists #Resolved

Added. Thanks!

In reply to: 261047854 [](ancestors = 261047854)

sfilipi · 2019-02-28T05:07:04Z

test/Microsoft.ML.Tests/AnomalyDetectionTests.cs

@@ -48,6 +50,98 @@ public void NoAnomalyTest()
            Assert.Throws<ArgumentOutOfRangeException>(() => ML.AnomalyDetection.Evaluate(transformedData));
        }

+        [Fact]
+        public static void RandomizedPcaInMemory()


RandomizedPcaInMemory [](start = 27, length = 21)

love it :) #WontFix

wschin · 2019-02-28T16:59:41Z

...les/Microsoft.ML.Samples/Dynamic/Trainers/AnomalyDetection/RandomizedPcaSampleWithOptions.cs

+
+namespace Microsoft.ML.Samples.Dynamic.Trainers.AnomalyDetection
+{
+    class RandomizedPcaSampleWithOptions


class [](start = 4, length = 5)

public static #Closed

abgoswam · 2019-02-28T16:59:57Z

test/Microsoft.ML.Tests/AnomalyDetectionTests.cs

+        /// <summary>
+        /// Class used to capture prediction of <see cref="DataPoint"/> in <see cref="ExecutePipelineWithGivenRandomizedPcaTrainer"/>.
+        /// </summary>
+#pragma warning disable 649


am curious ..whats this warning ? #Resolved

Fields assigned in runtime are considered never assigned, which is wrong. #Resolved

abgoswam · 2019-02-28T17:01:19Z

test/Microsoft.ML.Tests/AnomalyDetectionTests.cs

+                new DataPoint(){ Features= new float[3] {-100, 50, -100} }
+            };
+
+            // Convert native C# class to IDataView, a consumble format to ML.NET functions.


native C# class [](start = 23, length = 15)

nit : maybe just saying List would suffice ? #Resolved

abgoswam

abgoswam · 2019-02-28T17:32:44Z

docs/samples/Microsoft.ML.Samples/Microsoft.ML.Samples.csproj

@@ -3,6 +3,7 @@
  <PropertyGroup>
    <TargetFramework>netcoreapp2.1</TargetFramework>
    <OutputType>Exe</OutputType>
+    <NoWarn>1701;1702;0649</NoWarn>


1701;1702;0649 [](start = 4, length = 31)

one concern i have is that users might not be aware of this..so they will get thrown off when they see the warning about "uninitialized field" ...

an alternative is to make them properties in which case we do not need this NoWarn, and also we do not see the "uninitialized field" warning when we use properties.

@sfilipi what do u think ?

Just switched to properties. No need to disable warning anymore. Your suggestion reduces the redundant information user need to know!

In reply to: 261304076 [](ancestors = 261304076)

sfilipi · 2019-02-28T18:56:03Z

docs/samples/Microsoft.ML.Samples/Dynamic/Trainers/AnomalyDetection/RandomizedPcaSample.cs

+            //   The 2 - th example with features[1, 2, 3] is an inlier with a score of being inlier 0.8450122
+            //   The 3 - th example with features[0, 1, 0] is an inlier with a score of being inlier 0.9428905
+            //   The 4 - th example with features[0, 2, 1] is an inlier with a score of being inlier 0.9999999
+            //   The 5 - th example with features[-100, 50, -100] is an outlier with a score of being inlier 0


Nit: typically this goes below the code that generates it.

sfilipi

sfilipi · 2019-02-28T18:57:38Z

...les/Microsoft.ML.Samples/Dynamic/Trainers/AnomalyDetection/RandomizedPcaSampleWithOptions.cs

+            //   The 2 - th example with features[1, 2, 3] is an inlier with a score of being inlier 0.8450122
+            //   The 3 - th example with features[0, 1, 0] is an inlier with a score of being inlier 0.9428905
+            //   The 4 - th example with features[0, 2, 1] is an inlier with a score of being inlier 0.9999999
+            //   The 5 - th example with features[-100, 50, -100] is an outlier with a score of being inlier 0


same nit, put it below the code.

Fixed. Thanks.

wschin added the documentation Related to documentation of ML.NET label Feb 27, 2019

wschin self-assigned this Feb 27, 2019

wschin requested review from abgoswam and rogancarr February 27, 2019 23:22

wschin changed the title ~~Add an example of random PCA~~ Add an example of random PCA using in-memory data structure Feb 27, 2019

wschin force-pushed the rpca-example branch 3 times, most recently from 8409d09 to 48c01d3 Compare February 28, 2019 00:49

wschin added 2 commits February 27, 2019 17:07

Add an example of random PCA

43884ef

Fix build Polish example

Add another example with options

f9f0185

Fix build

wschin force-pushed the rpca-example branch from 48c01d3 to f9f0185 Compare February 28, 2019 01:10

sfilipi reviewed Feb 28, 2019

View reviewed changes

wschin mentioned this pull request Feb 28, 2019

Text loader v.s in-memory data structure in API reference samples #2726

Closed

Address comments

beb5474

wschin requested a review from sfilipi February 28, 2019 16:56

Example class style

61e3d04

wschin commented Feb 28, 2019

View reviewed changes

abgoswam reviewed Feb 28, 2019

View reviewed changes

Propagate comment changes

d1377f4

abgoswam approved these changes Feb 28, 2019

View reviewed changes

abgoswam reviewed Feb 28, 2019

View reviewed changes

Use property instead of field

a1f96e8

sfilipi reviewed Feb 28, 2019

View reviewed changes

sfilipi approved these changes Feb 28, 2019

View reviewed changes

sfilipi reviewed Feb 28, 2019

View reviewed changes

Address comments

933a480

wschin merged commit 6e9023f into dotnet:master Feb 28, 2019

wschin deleted the rpca-example branch February 28, 2019 23:36

ghost locked as resolved and limited conversation to collaborators Mar 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add an example of random PCA using in-memory data structure #2780

Add an example of random PCA using in-memory data structure #2780

wschin commented Feb 27, 2019 •

edited

Loading

codecov bot commented Feb 28, 2019 •

edited

Loading

sfilipi Feb 28, 2019 •

edited by wschin

Loading

wschin Feb 28, 2019

sfilipi Feb 28, 2019 •

edited by wschin

Loading

wschin Feb 28, 2019

sfilipi Feb 28, 2019 •

edited

Loading

wschin Feb 28, 2019

sfilipi Feb 28, 2019 •

edited by wschin

Loading

wschin Feb 28, 2019

sfilipi Feb 28, 2019 •

edited

Loading

wschin Feb 28, 2019

sfilipi Feb 28, 2019 •

edited by wschin

Loading

wschin Feb 28, 2019

sfilipi Feb 28, 2019 •

edited

Loading

wschin Feb 28, 2019 •

edited by sfilipi

Loading

abgoswam Feb 28, 2019 •

edited by wschin

Loading

wschin Feb 28, 2019 •

edited

Loading

abgoswam Feb 28, 2019 •

edited by wschin

Loading

abgoswam left a comment

abgoswam Feb 28, 2019 •

edited

Loading

wschin Feb 28, 2019 •

edited

Loading

sfilipi Feb 28, 2019

wschin Feb 28, 2019

sfilipi left a comment

sfilipi Feb 28, 2019

wschin Feb 28, 2019

Add an example of random PCA using in-memory data structure #2780

Add an example of random PCA using in-memory data structure #2780

Conversation

wschin commented Feb 27, 2019 • edited Loading

codecov bot commented Feb 28, 2019 • edited Loading

Codecov Report

sfilipi Feb 28, 2019 • edited by wschin Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfilipi Feb 28, 2019 • edited by wschin Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfilipi Feb 28, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfilipi Feb 28, 2019 • edited by wschin Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfilipi Feb 28, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfilipi Feb 28, 2019 • edited by wschin Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfilipi Feb 28, 2019 • edited Loading

Choose a reason for hiding this comment

wschin Feb 28, 2019 • edited by sfilipi Loading

Choose a reason for hiding this comment

abgoswam Feb 28, 2019 • edited by wschin Loading

Choose a reason for hiding this comment

wschin Feb 28, 2019 • edited Loading

Choose a reason for hiding this comment

abgoswam Feb 28, 2019 • edited by wschin Loading

Choose a reason for hiding this comment

abgoswam left a comment

Choose a reason for hiding this comment

abgoswam Feb 28, 2019 • edited Loading

Choose a reason for hiding this comment

wschin Feb 28, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfilipi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wschin commented Feb 27, 2019 •

edited

Loading

codecov bot commented Feb 28, 2019 •

edited

Loading

sfilipi Feb 28, 2019 •

edited by wschin

Loading

sfilipi Feb 28, 2019 •

edited by wschin

Loading

sfilipi Feb 28, 2019 •

edited

Loading

sfilipi Feb 28, 2019 •

edited by wschin

Loading

sfilipi Feb 28, 2019 •

edited

Loading

sfilipi Feb 28, 2019 •

edited by wschin

Loading

sfilipi Feb 28, 2019 •

edited

Loading

wschin Feb 28, 2019 •

edited by sfilipi

Loading

abgoswam Feb 28, 2019 •

edited by wschin

Loading

wschin Feb 28, 2019 •

edited

Loading

abgoswam Feb 28, 2019 •

edited by wschin

Loading

abgoswam Feb 28, 2019 •

edited

Loading

wschin Feb 28, 2019 •

edited

Loading