Get rid of value tuples in TrainTest and CrossValidation #2507

Ivanidzo4ka · 2019-02-11T22:06:34Z

Ivanidzo4ka · 2019-02-11T22:07:51Z

src/Microsoft.ML.Data/TrainCatalog.cs

+            }
+        }
+
+        public class CrossValidationResult<T> where T : class


CrossValidationResult [](start = 21, length = 21)

Summary #Closed

Ivanidzo4ka · 2019-02-11T22:09:18Z

src/Microsoft.ML.Data/TrainCatalog.cs

+            /// <summary>
+            /// Metrics for cross validation fold.
+            /// </summary>
+            public readonly T Metrics;


T [](start = 28, length = 1)

Part of me want to create interface and mark with it all Metric classes. Another part of me remembers what we try get rid of all empty interfaces.
Maybe empty base class?
Would like to hear your comments as well.

+1 for some way of making this not generic.

In reply to: 255714622 [](ancestors = 255714622)

Ivanidzo4ka · 2019-02-11T22:10:25Z

src/Microsoft.ML.Data/TrainCatalog.cs

+            /// </summary>
+            public readonly ITransformer Model;
+            /// <summary>
+            /// <see cref="IDataView"/> for scored fold.


for scored fold [](start = 40, length = 15)

Scored test fold? Any one has good ideas? #Resolved

"The scored hold-out set for this fold"

In reply to: 255714995 [](ancestors = 255714995)

ScoredHoldOutSet is literal, but long.

In reply to: 255726140 [](ancestors = 255726140,255714995)

Ivanidzo4ka · 2019-02-11T22:10:43Z

src/Microsoft.ML.Data/TrainCatalog.cs

+            /// </summary>
+            public readonly int Fold;
+
+            public CrossValidationResult(ITransformer model, T metrics, IDataView scores, int fold)


CrossValidationResult [](start = 19, length = 21)

summary or internal, probably internal. #Closed

Ivanidzo4ka · 2019-02-11T22:11:02Z

    public (BinaryClassificationMetrics metrics, ITransformer model, IDataView scoredTestData)[] CrossValidateNonCalibrated(

miss this one. #Closed

Refers to: src/Microsoft.ML.Data/TrainCatalog.cs:344 in 3d9b174. [](commit_id = 3d9b174, deletion_comment = False)

rogancarr · 2019-02-11T22:39:22Z

src/Microsoft.ML.Data/TrainCatalog.cs

+        /// A pair of datasets, for the train and test set.
+        /// </summary>
+        public struct TrainTestData
+        {


Why struct? #Resolved

Why class?
it's a small collection of immutable objects.

In reply to: 255723990 [](ancestors = 255723990)

Ivanidzo4ka · 2019-02-11T22:41:03Z

src/Microsoft.ML.Data/TrainCatalog.cs

+            }
+        }
+        /// <summary>
+        /// Results of running crossvalidation.


crossvalidation [](start = 31, length = 15)

two words #Resolved

hyphenated

In reply to: 255724522 [](ancestors = 255724522)

rogancarr · 2019-02-11T22:43:00Z

src/Microsoft.ML.Data/TrainCatalog.cs

+        /// A pair of datasets, for the train and test set.
+        /// </summary>
+        public struct TrainTestData
+        {


I could see us wanting to expand this in the future to also include a validation set, so it could be prudent to keep the name a bit vague. PartitionedData? #Resolved

What would be summary for PartitionedData?
This one used as output for TrainTestSplit function, if we ever introduce TrainTestValidationSplit I would prefer to create another object for that.

In reply to: 255725077 [](ancestors = 255725077)

Good point!

In reply to: 255725959 [](ancestors = 255725959,255725077)

rogancarr · 2019-02-11T22:44:45Z

src/Microsoft.ML.Data/TrainCatalog.cs

+        }
+
+        /// <summary>
+        /// Results for specific cross validation fold.


cross validation [](start = 33, length = 16)

Hyphenated #Resolved

Ivanidzo4ka · 2019-02-11T22:45:41Z

src/Microsoft.ML.Data/TrainCatalog.cs

+        /// Results of running crossvalidation.
+        /// </summary>
+        /// <typeparam name="T">Type of metric class</typeparam>
+        public class CrossValidationResult<T> where T : class


CrossValidationResult [](start = 21, length = 21)

sealed. #Resolved

rogancarr · 2019-02-11T22:51:36Z

src/Microsoft.ML.Data/TrainCatalog.cs

+            /// <summary>
+            /// Fold number.
+            /// </summary>
+            public readonly int Fold;


public readonly int Fold [](start = 12, length = 24)

Is this necessary, since they will be returned in an (ordered) array? And it'll be confusing if the order of the array doesn't match the fold number. #Resolved

Is it necessary ?- no, is it nice to have? - I think so. As soon as you tear them away from array, and let's say you want to report specific folds and do some stuff on them, you need to pass around fold anyway.

it'll be confusing if the order of the array doesn't match the fold number.
Not the case right now, but I would prefer to know real fold than index in array.

In reply to: 255727531 [](ancestors = 255727531)

rogancarr · 2019-02-11T22:54:57Z

    public static (DataView<T> trainSet, DataView<T> testSet) TrainTestSplit<T>(this TrainCatalogBase catalog,

This should return the TrainTestSplit object too.

Refers to: src/Microsoft.ML.StaticPipe/TrainingStaticExtensions.cs:34 in 0790005. [](commit_id = 0790005, deletion_comment = False)

rogancarr · 2019-02-11T22:55:44Z

    /// <typeparam name="T">The tuple describing the data schema.</typeparam>

Another Tuple we will want to get rid of.

Refers to: src/Microsoft.ML.StaticPipe/TrainingStaticExtensions.cs:23 in 0790005. [](commit_id = 0790005, deletion_comment = False)

rogancarr · 2019-02-11T22:57:01Z

    public static (RegressionMetrics metrics, Transformer<TInShape, TOutShape, TTransformer> model, DataView<TOutShape> scoredTestData)[] CrossValidate<TInShape, TOutShape, TTransformer>(

Tuple to Object

Refers to: src/Microsoft.ML.StaticPipe/TrainingStaticExtensions.cs:76 in 0790005. [](commit_id = 0790005, deletion_comment = False)

rogancarr · 2019-02-11T22:57:30Z

    public static (MultiClassClassifierMetrics metrics, Transformer<TInShape, TOutShape, TTransformer> model, DataView<TOutShape> scoredTestData)[] CrossValidate<TInShape, TOutShape, TTransformer>(

Tuple => Object.

Refers to: src/Microsoft.ML.StaticPipe/TrainingStaticExtensions.cs:134 in 0790005. [](commit_id = 0790005, deletion_comment = False)

rogancarr · 2019-02-11T22:58:02Z

    public static (BinaryClassificationMetrics metrics, Transformer<TInShape, TOutShape, TTransformer> model, DataView<TOutShape> scoredTestData)[] CrossValidateNonCalibrated<TInShape, TOutShape, TTransformer>(

Tuple to Object

Refers to: src/Microsoft.ML.StaticPipe/TrainingStaticExtensions.cs:192 in 0790005. [](commit_id = 0790005, deletion_comment = False)

Ivanidzo4ka · 2019-02-11T22:58:15Z

    /// <typeparam name="T">The tuple describing the data schema.</typeparam>

If I correctly understand @TomFinley, he is fine with tuples in StaticPipe project. Can be wrong. Would prefer to hear words of wisdom from him.

In reply to: 462528351 [](ancestors = 462528351)

Refers to: src/Microsoft.ML.StaticPipe/TrainingStaticExtensions.cs:23 in 0790005. [](commit_id = 0790005, deletion_comment = False)

rogancarr · 2019-02-11T22:58:17Z

    public static (CalibratedBinaryClassificationMetrics metrics, Transformer<TInShape, TOutShape, TTransformer> model, DataView<TOutShape> scoredTestData)[] CrossValidate<TInShape, TOutShape, TTransformer>(

Tuple to Object

Refers to: src/Microsoft.ML.StaticPipe/TrainingStaticExtensions.cs:250 in 0790005. [](commit_id = 0790005, deletion_comment = False)

rogancarr

Overall, looks good! Just a few comments.

Question: Does this compile? There are quite a few samples that use this API that you haven't bumped yet.

codecov · 2019-02-12T01:36:08Z

Codecov Report

Merging #2507 into master will increase coverage by <.01%.
The diff coverage is 76.81%.

@@            Coverage Diff             @@
##           master    #2507      +/-   ##
==========================================
+ Coverage   71.24%   71.24%   +<.01%     
==========================================
  Files         798      798              
  Lines      141231   141252      +21     
  Branches    16112    16112              
==========================================
+ Hits       100623   100641      +18     
- Misses      36142    36145       +3     
  Partials     4466     4466

Flag	Coverage Δ
#Debug	`71.24% <76.81%> (ø)`	⬆️
#production	`67.57% <66.66%> (ø)`	⬆️
#test	`85.35% <100%> (ø)`	⬆️

rogancarr · 2019-02-12T04:04:03Z

    /// <typeparam name="T">The tuple describing the data schema.</typeparam>

Got it. No worries, if that's the case.

In reply to: 462529091 [](ancestors = 462529091,462528351)

Refers to: src/Microsoft.ML.StaticPipe/TrainingStaticExtensions.cs:23 in 0790005. [](commit_id = 0790005, deletion_comment = False)

rogancarr

sfilipi · 2019-02-12T20:08:21Z

src/Microsoft.ML.Data/TrainCatalog.cs

@@ -263,13 +344,14 @@ internal BinaryClassificationTrainers(BinaryClassificationCatalog catalog)
        /// If the <paramref name="stratificationColumn"/> is not provided, the random numbers generated to create it, will use this seed as value.
        /// And if it is not provided, the default value will be used.</param>
        /// <returns>Per-fold results: metrics, models, scored datasets.</returns>


Per-fold results [](start = 21, length = 16)

maybe<see cref="CrossValidationResult"

sfilipi

Get rid of value tuple in TrainTest and CrossValidation

3d9b174

Ivanidzo4ka requested review from rogancarr, sfilipi and TomFinley February 11, 2019 22:06

Ivanidzo4ka commented Feb 11, 2019

View reviewed changes

address small things

0790005

rogancarr reviewed Feb 11, 2019

View reviewed changes

Ivanidzo4ka commented Feb 11, 2019

View reviewed changes

rogancarr reviewed Feb 11, 2019

View reviewed changes

Ivanidzo4ka commented Feb 11, 2019

View reviewed changes

rogancarr reviewed Feb 11, 2019

View reviewed changes

rogancarr suggested changes Feb 11, 2019

View reviewed changes

make it build?

a83b420

rogancarr approved these changes Feb 12, 2019

View reviewed changes

Ivanidzo4ka requested a review from yaeldekel February 12, 2019 05:47

sfilipi reviewed Feb 12, 2019

View reviewed changes

sfilipi approved these changes Feb 12, 2019

View reviewed changes

Ivanidzo4ka merged commit 56607ba into dotnet:master Feb 12, 2019

ghost locked as resolved and limited conversation to collaborators Mar 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Get rid of value tuples in TrainTest and CrossValidation #2507

Get rid of value tuples in TrainTest and CrossValidation #2507

Ivanidzo4ka commented Feb 11, 2019 •

edited

Loading

Ivanidzo4ka Feb 11, 2019 •

edited

Loading

Ivanidzo4ka Feb 11, 2019

rogancarr Feb 11, 2019

Ivanidzo4ka Feb 11, 2019 •

edited by rogancarr

Loading

rogancarr Feb 11, 2019

rogancarr Feb 11, 2019

Ivanidzo4ka Feb 11, 2019 •

edited

Loading

Ivanidzo4ka commented Feb 11, 2019 •

edited

Loading

rogancarr Feb 11, 2019 •

edited

Loading

Ivanidzo4ka Feb 11, 2019

Ivanidzo4ka Feb 11, 2019 •

edited by rogancarr

Loading

rogancarr Feb 11, 2019

rogancarr Feb 11, 2019 •

edited

Loading

Ivanidzo4ka Feb 11, 2019

rogancarr Feb 11, 2019

rogancarr Feb 11, 2019 •

edited

Loading

Ivanidzo4ka Feb 11, 2019 •

edited by rogancarr

Loading

rogancarr Feb 11, 2019 •

edited

Loading

Ivanidzo4ka Feb 11, 2019

rogancarr commented Feb 11, 2019

rogancarr commented Feb 11, 2019

rogancarr commented Feb 11, 2019

rogancarr commented Feb 11, 2019

rogancarr commented Feb 11, 2019

Ivanidzo4ka commented Feb 11, 2019

rogancarr commented Feb 11, 2019

rogancarr left a comment •

edited

Loading

codecov bot commented Feb 12, 2019 •

edited

Loading

rogancarr commented Feb 12, 2019

rogancarr left a comment

sfilipi Feb 12, 2019

sfilipi left a comment

Get rid of value tuples in TrainTest and CrossValidation #2507

Get rid of value tuples in TrainTest and CrossValidation #2507

Conversation

Ivanidzo4ka commented Feb 11, 2019 • edited Loading

Ivanidzo4ka Feb 11, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ivanidzo4ka Feb 11, 2019 • edited by rogancarr Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ivanidzo4ka Feb 11, 2019 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka commented Feb 11, 2019 • edited Loading

rogancarr Feb 11, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ivanidzo4ka Feb 11, 2019 • edited by rogancarr Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rogancarr Feb 11, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rogancarr Feb 11, 2019 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka Feb 11, 2019 • edited by rogancarr Loading

Choose a reason for hiding this comment

rogancarr Feb 11, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rogancarr commented Feb 11, 2019

rogancarr commented Feb 11, 2019

rogancarr commented Feb 11, 2019

rogancarr commented Feb 11, 2019

rogancarr commented Feb 11, 2019

Ivanidzo4ka commented Feb 11, 2019

rogancarr commented Feb 11, 2019

rogancarr left a comment • edited Loading

Choose a reason for hiding this comment

codecov bot commented Feb 12, 2019 • edited Loading

Codecov Report

rogancarr commented Feb 12, 2019

rogancarr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfilipi left a comment

Choose a reason for hiding this comment

Ivanidzo4ka commented Feb 11, 2019 •

edited

Loading

Ivanidzo4ka Feb 11, 2019 •

edited

Loading

Ivanidzo4ka Feb 11, 2019 •

edited by rogancarr

Loading

Ivanidzo4ka Feb 11, 2019 •

edited

Loading

Ivanidzo4ka commented Feb 11, 2019 •

edited

Loading

rogancarr Feb 11, 2019 •

edited

Loading

Ivanidzo4ka Feb 11, 2019 •

edited by rogancarr

Loading

rogancarr Feb 11, 2019 •

edited

Loading

rogancarr Feb 11, 2019 •

edited

Loading

Ivanidzo4ka Feb 11, 2019 •

edited by rogancarr

Loading

rogancarr Feb 11, 2019 •

edited

Loading

rogancarr left a comment •

edited

Loading

codecov bot commented Feb 12, 2019 •

edited

Loading