Extract IDataView into its own assembly and NuGet package

[eerhardt]

IDataView is a very flexible, efficient way of describing tabular data (columns and rows) in a read-only manner.

https://github.com/dotnet/machinelearning/blob/master/docs/code/IDataViewDesignPrinciples.md

At its heart are 2 key concepts:

• Schema (describing the columns)
• Cursoring (how to read the rows of data)

It has other capabilities that I won’t enumerate here, the above link describes them in more detail.

IDataView is very useful as an abstraction for tabular data that will allow users to pass data between two independent libraries.

For example, ML.NET is able to both consume and produce IDataView instances. Say there was a .NET library for Apache Arrow. If the Arrow .NET data type implements IDataView, the Apache Arrow data can be passed directly into ML.NET without having to copy it into a format that ML.NET consumes.

Another example is: say we had a visualization/graphing/plotting library in .NET that could consume data using IDataView. Then we could take data that was produced by ML.NET, or Apache Arrow, and feed it directly into the graphing library. There would be no need to copy, or change the shape of the data at all. And there is no need for this graphing library to know anything about ML.NET or Apache Arrow.

In my mind, we can use IDataView in a similar manner to what OData was promised to be: An exchange format which allows producers and consumers of data to communicate in a standardized way. (Although, OData has more capabilities such as filtering, sorting, updating data, etc which I am not proposing we add to IDataView. I was just using it as an analogy.)

/cc @TomFinley @Zruty0 @markusweimer @danmosemsft @stephentoub

[eerhardt]

In order to extract these types, the following work will need to be performed:

• Remove ISchema
• Rename UInt128
• Finish the ColumnType cleanup work started with IDataView Cleanup: ColumnType cleanup #1533
• IRowCursorConsolidator needs to break its dependency on IChannelProvider. We cannot bring IChannelProvider into the new DataView NuGet/assembly. See also Remove IRowCursorConsolidator #1867

/cc @singlis

[TomFinley]

Ah that reminds me, IRowCursorConsolidator actually ought to go away altogether, pursuant to API review meetings. But I see I forgot to enter an associated issue... OK entered as #1867.

There are some more open questions we might ask. Do we consider things like the existing implementations of ColumnType that we presently enjoy to be inherent in this, or ancillary to it? It is technically ancillary, but in order to be meaningful as any sort of interchange between different frameworks there should be some shared opinion. This would include things like vector valued types, and the like.

One thing I'm somewhat interested in hearing is whether this is useful to other types of applications. We specifically engineered this so that it would be useful for ML applications, and eschewed anything that didn't seem to have a direct application. If this idiom is "uplifted," I wonder if it will prove useful to other things. The danger I see, which is at this moment theoretical, is that we might succeed in making something useless in our quest to make it universally applicable. 😄 But we'll see.

[eerhardt]

Do we consider things like the existing implementations of ColumnType that we presently enjoy to be inherent in this, or ancillary to it? It is technically ancillary, but in order to be meaningful as any sort of interchange between different frameworks there should be some shared opinion. This would include things like vector valued types, and the like.

My current thinking is that ColumnType and its "primitive" implementations will be extracted as well. I think KeyType and VectorType might be a little too "ML.NET"-specific. For example, the VBuffer type (which VectorType references) might not be the best "general vector" representation for .NET. Instead, we might imagine different "dense" or "sparse" vector types that were originally envisioned in #608. And then possibly someone could write an ITransformer from those types into VBuffer for ML.NET to consume. (Note, this whole buffer discussion can be designed/implemented at a later date. The first round of work would just be the primitive types outside of ML.NET, and we can build on that.)

So my current proposal is to leave KeyType and VectorType as extension types of the extractedIDataView - just like how the ImageType class isn't part of ML.Core.