CollectionDataSource (train on top of memory collection instead of loading data from file) #106

Ivanidzo4ka · 2018-05-09T23:01:26Z

First iteration which allows user to create IDataview on top of IList or Enumerable and infrastructure to add it in pipeline.
address #10 issue

Ivanidzo4ka · 2018-05-09T23:04:49Z

@GalOshri @glebuk I would appreciate your contribution regarding class names. #Resolved

TomFinley · 2018-05-09T23:12:51Z

src/Microsoft.ML/MemoryCollection.cs

+        public void SetInput(IHostEnvironment env, Experiment experiment)
+        {
+            if (_listCollection!=null)
+            {


Ctrl-k-d, also, maybe since these are one-liners we can omit these vebose {s. #Closed

TomFinley · 2018-05-09T23:15:20Z

src/Microsoft.ML/MemoryCollection.cs

+
+        public MemoryCollection(IList<TInput> collection)
+        {
+            //need validation at some point


//need validation at some point [](start = 12, length = 31)

How about we at least validate to ensure that these aren't null, with a Contracts.CheckParamValue? Here and in the other constructor. #Closed

TomFinley · 2018-05-09T23:23:08Z

src/Microsoft.ML.Api/DataViewConstructionUtils.cs

@@ -14,7 +14,7 @@ namespace Microsoft.ML.Runtime.Api
    /// <summary>
    /// A helper class to create data views based on the user-provided types.
    /// </summary>
-    internal static class DataViewConstructionUtils
+    public static class DataViewConstructionUtils


public static class DataViewConstructionUtils [](start = 4, length = 45)

This makes me nervous... If you make this public rather than internal, then it is no longer appropriate to have what are currently Asserts on the public methods. They'll have to be Checks, with suitably verbose error messages, properly called in the context of env, and so on, and so on. This is more work than I think you want to have right now.

Do you need all of the functions in this class? This was originally a supporting utility class for the pre-ML.NET API. You don't need all of it... in fact I think you only need a couple of them. I might choose to either (1) introduce another public class with the methods you need being accessible through that or (2) change this class so that it remains public but everything in it has its access member changed to internal, except the one or two facilities you actually need. #Closed

Oh, I miss these blobs of text so much. (I'm serious right now) #Closed

Someone already did that for me. All I had to do is to look for method reference :)

In reply to: 187203478 [](ancestors = 187203478)

Thanks Ivan, it's nice to be appreciated. :D :D

In reply to: 187203886 [](ancestors = 187203886)

TomFinley · 2018-05-09T23:26:45Z

src/Microsoft.ML.Api/InternalSchemaDefinition.cs

@@ -16,7 +16,7 @@ namespace Microsoft.ML.Runtime.Api
    /// <summary>
    /// An internal class that holds the (already validated) mapping between a custom type and an IDataView schema.
    /// </summary>
-    internal sealed class InternalSchemaDefinition
+    public sealed class InternalSchemaDefinition


public sealed class InternalSchemaDefinition [](start = 4, length = 44)

I have similar concerns about this class being public. If we can avoid it, possibly by moving some of the logic current in MemoryCollection.cs to this assembly, then calling that from the Microsoft.ML project, I'd be delighted. #Closed

Ivanidzo4ka · 2018-05-09T23:27:00Z

using Microsoft.ML.Data;

need header #Resolved

Refers to: src/Microsoft.ML/Runtime/EntryPoints/DataViewReference.cs:1 in b166f05. [](commit_id = b166f05, deletion_comment = False)

Ivanidzo4ka · 2018-05-09T23:27:09Z

using System;

need header #Resolved

Refers to: src/Microsoft.ML/MemoryCollection.cs:1 in b166f05. [](commit_id = b166f05, deletion_comment = False)

TomFinley · 2018-05-09T23:28:33Z

test/Microsoft.ML.Tests/MemoryCollectionTests.cs

+                Experiment experiment = environment.CreateExperiment();
+                ILearningPipelineDataStep output = collection.ApplyStep(null, experiment) as ILearningPipelineDataStep;
+
+                Assert.NotNull(output.Data);


Assert.NotNull(output.Data); [](start = 16, length = 28)

If you really meant the as above, you should first Assert.NonNull on output. If you didn't mean the as above, then you should do a direct () style cast to ILearningPipelineDataStep. The only reason to use an as style cast is if you are entertaining the possibility that it might not implement that interface, in which case, given that this is a test, you should test that. (And if it wasn't a test, you would Contracts.Check* it.) #Closed

Ivanidzo4ka · 2018-05-09T23:32:55Z

src/Microsoft.ML/MemoryCollection.cs

+        private Data.DataViewReference _dataViewEntryPoint;
+        private IDataView _dataView;
+
+        public MemoryCollection(IList<TInput> collection)


MemoryCollection [](start = 15, length = 16)

public constructor required comments. #Resolved

Ivanidzo4ka · 2018-05-09T23:33:50Z

src/Microsoft.ML.Api/DataViewConstructionUtils.cs

@@ -14,7 +14,7 @@ namespace Microsoft.ML.Runtime.Api
    /// <summary>
    /// A helper class to create data views based on the user-provided types.
    /// </summary>
-    internal static class DataViewConstructionUtils
+    public static class DataViewConstructionUtils
    {
        public static IDataView CreateFromList<TRow>(IHostEnvironment env, IList<TRow> data,


IList [](start = 75, length = 5)

Since we in this file, @[email protected] do you have any opinion about IList?
I would personally convert it to List. #Resolved

Is there a reason why you would prefer List here?

So the reason I would prefer a specific implementer of an interface rather than the interface itself is to avoid having to go through the virtual function table. (Which is why it's often better to write Foo<TBar>(TBar biz) where TBar : IBar vs. Foo(IBar biz).) In this specific case though, I don't think it's worth it. You're ultimately going to be dealing with an IEnumerator<TRow> anyway for most operations on this structure, and there it's unfortunately unavoidable, so I care a bit less. But perhaps there's something to this I'm not appreciating.

In reply to: 187205023 [](ancestors = 187205023)

I guess its just personal. I just don't like IList. In same time, IList allow you to pass Arrays, so this solve problem with Array support.

In reply to: 187207615 [](ancestors = 187207615,187205023)

TomFinley · 2018-05-09T23:46:54Z

src/Microsoft.ML/MemoryCollection.cs

+
+        public MemoryCollection(IEnumerable<TInput> collection)
+        {
+            Contracts.CheckParamValue(collection != null, collection, nameof(collection), "Must be non-null");


CheckParamValue(collection != null, collection [](start = 22, length = 46)

I'm sorry, I'm silly... I didn't mean CheckParamValue, I meant CheckValue. That'll simplify this a bit. #Closed

TomFinley · 2018-05-09T23:49:00Z

src/Microsoft.ML/MemoryCollection.cs

+            public MemoryCollectionPipelineStep(Var<IDataView> data)
+            {
+                Data = data;
+                Model = null;


Unnecessary, the default value is null.

That's fine, but I'd go one step further... What I'd do is change the public Var<ITransformModel> Model { get; } below to public Var<ITransformModel> Model => null;, that way you avoid having any backing field whatsoever. (Plus you save three characters, which totally makes it worth it. :D ) #Closed

TomFinley · 2018-05-09T23:54:15Z

test/Microsoft.ML.Tests/MemoryCollectionTests.cs

+        [Fact]
+        public void CanSuccessfullyApplyATransform()
+        {
+            var collection = new MemoryCollection<Input>(new List<Input>() { new Input { Number1 = 1, String1 = "1" } });


MemoryCollection [](start = 33, length = 23)

Just an observation... if we had instead structured this as a static utility method somewhere, then we could avoid having the double-specification of the Input class, as it would have been inferred by the compiler.

So: if we had a input type MyAwesomeInputType, then instead of MemoryCollection<MyAwesomeInputType>(new MyAwesomeInputType[] {...}), we would have MemoryCollection.Create(new MyAwesomeInputType[] {...}) since the compiler could have done the work of inferring the type and whatnot. #Closed

glebuk · 2018-05-10T01:46:26Z

src/Microsoft.ML/MemoryCollection.cs

+    public class MemoryCollection
+    {
+        /// <summary>
+        /// Creates memory collection loader.


loader [](start = 38, length = 6)

loader from IList #Closed

Is that not obvious from the parameter type? Perhaps adding a "from ." or somesuch would be fine (added to both), but even that I'd say would be a bit too needlessly verbose.

In reply to: 187221474 [](ancestors = 187221474)

it's confusing that it says "Collection Loader" while it takes a List . perhaps we should rename it to a ListLoader. and split away from the StreamingEnumerableLoader

In reply to: 187230849 [](ancestors = 187230849,187221474)

User no longer see class, it just get interface, is it better?

In reply to: 187238467 [](ancestors = 187238467,187230849,187221474)

glebuk · 2018-05-10T01:46:56Z

src/Microsoft.ML/MemoryCollection.cs

+        }
+
+        /// <summary>
+        /// Creates memory collection loader.


loader [](start = 38, length = 6)

loader from IEnumerable. #Closed

glebuk · 2018-05-10T01:47:12Z

src/Microsoft.ML/MemoryCollection.cs

+using Microsoft.ML.Runtime.EntryPoints;
+using Microsoft.ML.Runtime.Internal.Utilities;
+
+namespace Microsoft.ML


.ML [](start = 19, length = 3)

ML.Data #Closed

TextLoader is part of just ML, should I change it as well?

In reply to: 187221578 [](ancestors = 187221578)

good question... seems they both should be in data. the argument went about like this - can a user has out of the box experience with just ML namespace.
I guess we can keep it in ML for now.

In reply to: 187221799 [](ancestors = 187221799,187221578)

Please move to Data. My PR will move TextLoader to ML.Data. #Resolved

glebuk · 2018-05-10T01:50:59Z

src/Microsoft.ML/Runtime/EntryPoints/InMemoryDataView.cs

+[assembly: LoadableClass(typeof(void), typeof(InMemoryDataView), null, typeof(SignatureEntryPointModule), "InMemoryDataView")]
+namespace Microsoft.ML.Runtime.EntryPoints
+{
+    public class InMemoryDataView


InMemoryDataView [](start = 17, length = 16)

Where is this being used? I don't see any references to this class in code or tests. #Closed

This is a entrypoint :) Data.DataViewReference it get used in MemoryCollection.cs (At least it entry point wrapper)

In reply to: 187222065 [](ancestors = 187222065)

Ok, silly me as looking for "InMemoryDataView" not "Data.DataViewReference"

In reply to: 187222365 [](ancestors = 187222365,187222065)

glebuk · 2018-05-11T02:43:55Z

Sorry, indeed I meant to add a different Eric!

In reply to: 388041663 [](ancestors = 388041663)

TomFinley · 2018-05-11T17:56:42Z

src/Microsoft.ML/Data/CollectionLoader.cs

+
+namespace Microsoft.ML.Data
+{
+    public class CollectionLoader


public class [](start = 4, length = 12)

Static class #Closed

Ivanidzo4ka · 2018-05-11T22:28:27Z

@dotnet-bot test Windows_NT Release please

TomFinley · 2018-05-11T22:31:37Z

src/Microsoft.ML/Runtime/EntryPoints/InMemoryDataView.cs

+    {
+        public sealed class Input
+        {
+            [Argument(ArgumentType.Required, ShortName = "data", HelpText = "Pointer to IDataView in memory", SortOrder = 1)]


ShortName = "data" [](start = 45, length = 18)

Since shortname is same as longname, you can safely omit. #Resolved

TomFinley

GalOshri · 2018-05-12T19:18:37Z

test/Microsoft.ML.Tests/CollectionDataSourceTests.cs

+                new IrisData { SepalLength = 1f, SepalWidth = 1f ,PetalLength=0.3f, PetalWidth=5.1f, Label=1},
+                new IrisData { SepalLength = 1.2f, SepalWidth = 0.5f ,PetalLength=0.3f, PetalWidth=5.1f, Label=0}
+            };
+            var collection = CollectionDataSource.Create(data);


Would we address the inconsistency in naming/usage between CollectionDataSource and TextLoader in a separate PR? We want it to be clearer these are the two (current) approaches to bringing data into a LearningPipeline.

Ivanidzo4ka · 2018-05-14T18:16:43Z

@glebuk is this review still in blocking state? If yes, what should be process of API review? #Closed

glebuk · 2018-05-14T23:16:16Z

let me review it. Just give me 20 minutes.

In reply to: 388913155 [](ancestors = 388913155)

glebuk · 2018-05-14T23:18:31Z

src/Microsoft.ML/Data/CollectionDataSource.cs

+
+namespace Microsoft.ML.Data
+{
+    public static class CollectionDataSource


CollectionDataSource [](start = 24, length = 20)

Please add a top level XML help - what is this class, where does it use, what's the purpose for its existance? #Closed

glebuk · 2018-05-14T23:48:15Z

done. please look at 2 new comments

In reply to: 388992391 [](ancestors = 388992391,388913155)

glebuk

Ivanidzo4ka · 2018-05-15T00:18:27Z

@dotnet-bot test OSX10.13 Debug please

glebuk

ntaherkhani · 2018-05-23T20:24:45Z

"CollectionDataSource" is not recognized in my project.
"The name 'CollectionDataSource' does not exist in the current context" .
may somebody help me please.
Also, May please put some samples for other types of data loading except using "TextLoader". I need some samples to add data to pipeline from "DataTable".
Thanks.

Ivanidzo4ka · 2018-05-23T20:41:50Z

@ntaherkhani Changes in this PR would be part of 0.2 release. Currently on nuget.org you can find only 0.1 release. You can consume nuget with current master branch from myget: https://dotnet.myget.org/F/dotnet-core/api/v3/index.json
Check "Include prerelease" in VS and search for Microsoft.ML package.

…ading data from file) (dotnet#106) * in memory loader * add test file for memory collection * even in afterlife EntryPointCatalog will chase me down. * Address some comments. * update tests * address more comments. * remove empty param description * hide collectionloader * refactor classes a little. * pesky new lines! * slightly better comments. but only slighty * rename it * make class static * not a loader * remove alias in entrypoint * address comments

in memory loader

f69d659

Ivanidzo4ka requested review from TomFinley and sfilipi May 9, 2018 23:02

add test file for memory collection

55b6e46

Ivanidzo4ka self-assigned this May 9, 2018

Ivanidzo4ka added the enhancement New feature or request label May 9, 2018

TomFinley reviewed May 9, 2018

View reviewed changes

even in afterlife EntryPointCatalog will chase me down.

b166f05

TomFinley reviewed May 9, 2018

View reviewed changes

shauheen added this to the 0518 milestone May 9, 2018

TomFinley reviewed May 9, 2018

View reviewed changes

TomFinley requested a review from eerhardt May 9, 2018 23:24

TomFinley reviewed May 9, 2018

View reviewed changes

Ivanidzo4ka commented May 9, 2018

View reviewed changes

Ivan Matantsev added 2 commits May 9, 2018 16:36

Address some comments.

a1761b1

update tests

12d1b9e

TomFinley reviewed May 9, 2018

View reviewed changes

Ivan Matantsev added 2 commits May 9, 2018 17:21

address more comments.

ebcf448

remove empty param description

110e205

glebuk reviewed May 10, 2018

View reviewed changes

rename it

d78afa3

TomFinley reviewed May 11, 2018

View reviewed changes

Ivan Matantsev added 2 commits May 11, 2018 11:05

make class static

ab86b09

not a loader

ebe6f33

Ivanidzo4ka changed the title ~~In memory loader (train on top of memory collection instead of loading data from file)~~ CollectionDataSource (train on top of memory collection instead of loading data from file) May 11, 2018

TomFinley reviewed May 11, 2018

View reviewed changes

TomFinley approved these changes May 11, 2018

View reviewed changes

remove alias in entrypoint

04ff469

GalOshri reviewed May 12, 2018

View reviewed changes

shauheen removed this from the 0518 milestone May 14, 2018

This was referenced May 14, 2018

Code generate TextLoader API and make it backward compatible with existing TextLoader API. #38

Closed

Support DataTable/DataView #141

Closed

glebuk reviewed May 14, 2018

View reviewed changes

address comments

9698d19

glebuk approved these changes May 15, 2018

View reviewed changes

Ivanidzo4ka merged commit 6d5a41d into dotnet:master May 15, 2018

Ivanidzo4ka deleted the ivanidze/in_memory_loader branch May 15, 2018 00:26

TomFinley mentioned this pull request May 17, 2018

How do you load data from an in-memory data set? #91

Closed

TomFinley mentioned this pull request Jun 1, 2018

Is it possible update model without retraining? #257

Closed

ghost locked as resolved and limited conversation to collaborators Mar 31, 2022

CollectionDataSource (train on top of memory collection instead of loading data from file) #106

CollectionDataSource (train on top of memory collection instead of loading data from file) #106

Conversation

Ivanidzo4ka commented May 9, 2018 • edited Loading

Ivanidzo4ka commented May 9, 2018 • edited by glebuk Loading

TomFinley May 9, 2018 • edited Loading

Choose a reason for hiding this comment

TomFinley May 9, 2018 • edited Loading

Choose a reason for hiding this comment

TomFinley May 9, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka May 9, 2018 • edited by glebuk Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomFinley May 9, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka commented May 9, 2018 • edited Loading

Ivanidzo4ka commented May 9, 2018 • edited Loading

TomFinley May 9, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka May 9, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka May 9, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ivanidzo4ka May 10, 2018 • edited Loading

Choose a reason for hiding this comment

TomFinley May 9, 2018 • edited Loading

Choose a reason for hiding this comment

TomFinley May 9, 2018 • edited Loading

Choose a reason for hiding this comment

TomFinley May 9, 2018 • edited Loading

Choose a reason for hiding this comment

glebuk May 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glebuk May 10, 2018 • edited Loading

Choose a reason for hiding this comment

glebuk May 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codemzs May 11, 2018 • edited by Ivanidzo4ka Loading

Choose a reason for hiding this comment

glebuk May 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glebuk May 10, 2018 • edited Loading

Choose a reason for hiding this comment

glebuk commented May 11, 2018 • edited Loading

TomFinley May 11, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka commented May 11, 2018 • edited Loading

TomFinley May 11, 2018 • edited by Ivanidzo4ka Loading

Choose a reason for hiding this comment

TomFinley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ivanidzo4ka commented May 14, 2018 • edited by glebuk Loading

glebuk commented May 14, 2018

glebuk May 14, 2018 • edited Loading

Choose a reason for hiding this comment

glebuk commented May 14, 2018

glebuk left a comment

Choose a reason for hiding this comment

Ivanidzo4ka commented May 15, 2018

glebuk left a comment

Choose a reason for hiding this comment

ntaherkhani commented May 23, 2018

Ivanidzo4ka commented May 23, 2018 • edited Loading

Ivanidzo4ka commented May 9, 2018 •

edited

Loading

Ivanidzo4ka commented May 9, 2018 •

edited by glebuk

Loading

TomFinley May 9, 2018 •

edited

Loading

TomFinley May 9, 2018 •

edited

Loading

TomFinley May 9, 2018 •

edited

Loading

Ivanidzo4ka May 9, 2018 •

edited by glebuk

Loading

TomFinley May 9, 2018 •

edited

Loading

Ivanidzo4ka commented May 9, 2018 •

edited

Loading

Ivanidzo4ka commented May 9, 2018 •

edited

Loading

TomFinley May 9, 2018 •

edited

Loading

Ivanidzo4ka May 9, 2018 •

edited

Loading

Ivanidzo4ka May 9, 2018 •

edited

Loading

Ivanidzo4ka May 10, 2018 •

edited

Loading

TomFinley May 9, 2018 •

edited

Loading

TomFinley May 9, 2018 •

edited

Loading

TomFinley May 9, 2018 •

edited

Loading

glebuk May 10, 2018 •

edited

Loading

glebuk May 10, 2018 •

edited

Loading

glebuk May 10, 2018 •

edited

Loading

codemzs May 11, 2018 •

edited by Ivanidzo4ka

Loading

glebuk May 10, 2018 •

edited

Loading

glebuk May 10, 2018 •

edited

Loading

glebuk commented May 11, 2018 •

edited

Loading

TomFinley May 11, 2018 •

edited

Loading

Ivanidzo4ka commented May 11, 2018 •

edited

Loading

TomFinley May 11, 2018 •

edited by Ivanidzo4ka

Loading

Ivanidzo4ka commented May 14, 2018 •

edited by glebuk

Loading

glebuk May 14, 2018 •

edited

Loading

Ivanidzo4ka commented May 23, 2018 •

edited

Loading