Extract IDataView into its own assembly and NuGet package

[eerhardt]

IDataView is a very flexible, efficient way of describing tabular data (columns and rows) in a read-only manner.

https://github.com/dotnet/machinelearning/blob/master/docs/code/IDataViewDesignPrinciples.md

At its heart are 2 key concepts:

• Schema (describing the columns)
• Cursoring (how to read the rows of data)

It has other capabilities that I won’t enumerate here, the above link describes them in more detail.

IDataView is very useful as an abstraction for tabular data that will allow users to pass data between two independent libraries.

For example, ML.NET is able to both consume and produce IDataView instances. Say there was a .NET library for Apache Arrow. If the Arrow .NET data type implements IDataView, the Apache Arrow data can be passed directly into ML.NET without having to copy it into a format that ML.NET consumes.

Another example is: say we had a visualization/graphing/plotting library in .NET that could consume data using IDataView. Then we could take data that was produced by ML.NET, or Apache Arrow, and feed it directly into the graphing library. There would be no need to copy, or change the shape of the data at all. And there is no need for this graphing library to know anything about ML.NET or Apache Arrow.

In my mind, we can use IDataView in a similar manner to what OData was promised to be: An exchange format which allows producers and consumers of data to communicate in a standardized way. (Although, OData has more capabilities such as filtering, sorting, updating data, etc which I am not proposing we add to IDataView. I was just using it as an analogy.)

/cc @TomFinley @Zruty0 @markusweimer @danmosemsft @stephentoub

[eerhardt]

In order to extract these types, the following work will need to be performed:

• Remove ISchema
• Rename UInt128
• Finish the ColumnType cleanup work started with IDataView Cleanup: ColumnType cleanup #1533
• IRowCursorConsolidator needs to break its dependency on IChannelProvider. We cannot bring IChannelProvider into the new DataView NuGet/assembly. See also Remove IRowCursorConsolidator #1867

/cc @singlis

[TomFinley]

Ah that reminds me, IRowCursorConsolidator actually ought to go away altogether, pursuant to API review meetings. But I see I forgot to enter an associated issue... OK entered as #1867.

There are some more open questions we might ask. Do we consider things like the existing implementations of ColumnType that we presently enjoy to be inherent in this, or ancillary to it? It is technically ancillary, but in order to be meaningful as any sort of interchange between different frameworks there should be some shared opinion. This would include things like vector valued types, and the like.

One thing I'm somewhat interested in hearing is whether this is useful to other types of applications. We specifically engineered this so that it would be useful for ML applications, and eschewed anything that didn't seem to have a direct application. If this idiom is "uplifted," I wonder if it will prove useful to other things. The danger I see, which is at this moment theoretical, is that we might succeed in making something useless in our quest to make it universally applicable. 😄 But we'll see.