Do a column validation during pipeline construction

[ganik]

In code below supply a wrong column (bolded below)
Currently wrong column discovered during Train command
Ideally pipeline.Add() should return an error saying that column doesn't exist.

        var pipeline = new LearningPipeline();
        pipeline.Add(new TextLoader<HousePriceData>(dataPath, header: true, sep: ","));
        pipeline.Add(new ColumnConcatenator(outputColumn: "NumericalFeatures",
            "**SqftLiving1**", "SqftLot", "SqftAbove", "SqftBasement", "Lat", "Long", "SqftLiving15", "SqftLot15"));

[justinormont]

This can be a bit difficult as the columns produced by a transform depends on their parameters. For instance, OutputTokens = true in the TextFeaturizer causes it to produce another column named [YourOriginalOutputColumnName]_TransformedText. This example is even more benign than most as it's a simple boolean decision. The worst would be the LoadTransform, which its parameters tell nothing about what is produced (you would have to inspect the model being loaded to see outputs).

That said, all is doable. We would have to track the normal columns produced by transforms, plus have something in place to do static analysis of the pipeline as it's being constructed, but before it runs. The hard part comes in to play with knowing the rules for a transform and being able to codify its rules in a manor to be consumed at the pipeline construction phase.

One furtherance of this request is to also check column types. For instance, we know PCA must take in a vector of R4, so throw an error at pipeline construction if you pass PCA text instead of a vector of R4.

Perhaps we could codify the happy path, and only throw "missing column" errors when we are certain they are missing. Otherwise wait until the pipeline is executed.

[TomFinley]

The imperfect solution in other contexts is to merely run on a sample of data for the validation and preview mode -- e.g., run on perhaps 1000 rows or something. It works "fine," by which I mean it is as a solution more helpful than unhelpful.

By unhelpful: this has lead to user confusion internally when we expose the results to them, e.g., in a preview. ("I know for a fact my corpus has some hundred million unique terms, but this preview window gives me a feature vector of some tens of thousands. What gives?") Also sometimes a pipeline is correct on all the data, but not on a subset (or vice versa) -- e.g., using term transform on labels for binary/multiclass classification. There are also some transforms where running on 1000 rows may actually take a considerable amount of time. (E.g., anything having to do with images.) It may be better than doing nothing though.

When we've previously discussed how to do this, the only "proper" way we've figured how to do this would be to have some parallel structures to all data transforms that, when instantiated, applied more or less the same validation logic, and produced ISchama-like objects. By ISchema-like I mean to say except instead of actual dimensions or counts on vector types and key value types, they may instead have unbound variables... instead of actual metadata they might have a promise that the metadata would be there in the final product, etc. We've tended to call this parallel infrastructure a "manifest," for some reason. The trouble with that is it might need to be separate logic from the "true" validation (since existing validation over a real ISchema would still need to happen, you would need to have other logic for ISchema-like). Also there are basic simple engineering challenges -- by the time an IDataTransform is instantiated, the ship has already sailed (so to speak). This suggests an intermediate structure, but I'm afraid of reinventing what I think is a structure that would wind up looking like ISchemaBindableMapper. 😝

[zeahmed]

Regarding,
The imperfect solution in other contexts is to merely run on a sample of data for the validation and preview mode

The visual studio debugging option provide a way to inspect the pipeline on sample data. This can used for validation at early stage before actual training begins.

[justinormont]

I very much like @TomFinley's suggestion for a "proper" method.

I've shied away from the running-on-a-sample method for non-GUI uses. It's very useful in a GUI as it shows the current state of the pipeline and the pipeline can be partially run to give you the status of your data/pipeline after each featurization step. But for use in a programmatic setting, this causes a time hit where we first setup the pipeline, instantiate each component, run 1k rows thru it, delete all components (just for the verification step). Then after we verify it works, we would run the full data through. Granted, if the likelyhood of incorrect pipelines is very high, this time is well spent (so it the error is thrown in 5 seconds instead of 20 minutes). This sub-sample method can also have false positives, for instance if the first 1k rows of data only corresponds to a single class, the learner will only receive a single class (and hopefully complain about it); or an otherwise correct pipeline w/ K-Means w/ k=2000, will complain if there are only 1k rows of data (as it can't form its clusters).

Perhaps as an intermediate solution we could have a pipeline checker, like bool success = pipeline.Check<Data, Prediction>(); which a user can optionally call before var model = pipeline.Train<Data, Prediction>();, which runs the pipeline on a sample of data. But being optional, the time hit is only incurred when purposefully called. Another option would be to add a default-off flag to the pipeline which runs the check at the end of each .Add(), which would cause it to throw at the right pipeline component during the pipeline setup.

The method @zeahmed brings up of interactively debugging in Visual Studio is definitely useful though more so when you expect it to fail. Early throwing is useful for the normal case where one's expecting the pipeline to work but instead dies after 20min of time (and you don't notice the failure until 5pm).