Skip to content

Exception when converting PredictedLabel from Key To Value #3090

New issue
New issue
Closed
Closed
Exception when converting PredictedLabel from Key To Value#3090
Assignees
Ivanidzo4ka
Labels
bugSomething isn't working
Milestone
 
0319
@daholste

Description

@daholste
daholste
opened on Mar 26, 2019 · edited by daholste

When trying to convert the output PredictedLabel from key back to value, the exception

System.InvalidOperationException
  HResult=0x80131509
  Message=Metadata KeyValues does not exist
  Source=Microsoft.ML.Core
  StackTrace:
   at Microsoft.ML.Runtime.Contracts.Check(IExceptionContext ctx, Boolean f, String msg) in C:\MLDotNet2\src\Microsoft.ML.Core\Utilities\Contracts.cs:line 497

is thrown.

Code to repro:

var mlContext = new MLContext();

var textLoaderOptions = new TextLoader.Options()
{
	Columns = new[]
	{
		new TextLoader.Column("Label", DataKind.Single, 0),
		new TextLoader.Column("Row", DataKind.Single, 1),
		new TextLoader.Column("Column", DataKind.Single, 2),
	},
	HasHeader = true,
	Separators = new[] { '\t' }
};
var textLoader = mlContext.Data.CreateTextLoader(textLoaderOptions);
var data = textLoader.Load(@"C:\MLDotNet2\test\data\trivial-train.tsv");

var ap = mlContext.BinaryClassification.Trainers.AveragedPerceptron();
var ova = mlContext.MulticlassClassification.Trainers.OneVersusAll(ap);

var pipeline = mlContext.Transforms.Conversion.MapValueToKey("Label")
	.Append(mlContext.Transforms.Concatenate("Features", "Row", "Column"))
	.Append(ova)
	.Append(mlContext.Transforms.Conversion.MapKeyToValue("PredictedLabel"));

var model = pipeline.Fit(data);

Replace C:\MLDotNet2\ with path to ML.NET repo on your local machine

Activity

sfilipi
self-assigned this
on Mar 26, 2019
sfilipi

sfilipi commented on Mar 26, 2019

@sfilipi
sfilipi
on Mar 26, 2019
Member

The problem here is that the Annotations are not carried from the Label column to the PredictedLabel column.

sfilipi
added
bugSomething isn't working
on Mar 26, 2019
sfilipi
removed their assignment
on Mar 26, 2019
Ivanidzo4ka
self-assigned this
on Mar 26, 2019
vinodshanbhag

vinodshanbhag commented on Mar 26, 2019

@vinodshanbhag
vinodshanbhag
on Mar 26, 2019
Member

@Ivanidzo4ka @shauheen
This is a regression from 0.11 and is currently currently blocking automl multiclass scenario.
Can I please request that we prioritize this issue?
Thanks in advance

Ivanidzo4ka
mentioned this on Mar 26, 2019
eerhardt
added this to the 0319 milestone on Mar 27, 2019
eerhardt
added
Bug Bash
on Mar 27, 2019
eerhardt

eerhardt commented on Mar 27, 2019

@eerhardt
eerhardt
on Mar 27, 2019
Member

I also hit this issue during the bug bash in upgrading XamlBrewer.Uwp.MachineLearningSample to the latest ML.NET build.

https://github.com/XamlBrewer/UWP-MachineLearning-Sample/blob/150b11ed4941451e0aa48c7f71500069efa2c2fb/XamlBrewer.Uwp.MachineLearningSample/Models/MulticlassClassification/MulticlassClassificationModel.cs#L19-L30

            _pipeline = MLContext.Transforms.Conversion.MapValueToKey("Label")
                .Append(MLContext.Transforms.Text.FeaturizeText("Features", "Text"))
            // Main algorithm
            // .Append(MLContext.MulticlassClassification.Trainers.StochasticDualCoordinateAscent())
            // or
                .Append(MLContext.MulticlassClassification.Trainers.LogisticRegression())
            // or
            // .Append(MLContext.MulticlassClassification.Trainers.NaiveBayes()) // yields weird metrics...


                
            // Convert the predicted value back into a language.
                .Append(MLContext.Transforms.Conversion.MapKeyToValue("PredictedLabel"));

This doesn't appear to be OVA specific.

eerhardt
added a commit that references this issue on Mar 27, 2019

Update to ML.NET version 0.12.0-preview-27526-3

074f4c9
eerhardt
added
Shiproom-review
on Mar 29, 2019
shauheen
added
Shiproom-approve
and removed
Shiproom-review
on Apr 1, 2019
Ivanidzo4ka
closed this as completedin #3101on Apr 1, 2019
artemiusgreat

artemiusgreat commented on Dec 25, 2019

@artemiusgreat
artemiusgreat
on Dec 25, 2019 · edited by artemiusgreat

Still reproducible in ML.NET 1.4
I have 18 records in IDataView for training.
Executing CrossValidate on dataView.TestSet always raises
System.InvalidOperationException: 'Metadata KeyValues does not exist'.
Performing CrossValidate on entire dataView works fine.
Performing CrossValidate on dataView.TrainSet works fine only if numberOfFolds is greater than 3, otherwise, the same exception.

public dynamic GetCrossMetrics(IEstimator<ITransformer> pipeline, IEstimator<ITransformer> estimator, IDataView dataView)
{
  var inputs = pipeline.Fit(dataView).Transform(dataView);
  var metrics = Predictor.MulticlassClassification.CrossValidate(inputs, pipeline.Append(estimator), numberOfFolds: 2, labelColumnName: "Emotion");

  return metrics;
}

var inputs = GetInputs();                          // Read data from TSV file
var pipeline = GetPipeline();                    // Create "Label" column, concatenate "Features", and NormalizeMinMax
var estimator = GetEstimator();               // MultiClass.LightGbm
var inputSets = Predictor.Data.TrainTestSplit(inputs, testFraction: 0.2);
var model = pipeline.Append(estimator).Fit(inputSets.TrainSet);

var x1 = GetCrossMetrics(pipeline, estimator, inputs);                      // always works
var x2 = GetCrossMetrics(pipeline, estimator, inputSets.TrainSet);   // works only for numberOfFolds > 3
var x3 = GetCrossMetrics(pipeline, estimator, inputSets.TestSet);     // always fails, TestSet contains 2 items
antoniovs1029
mentioned this on Jan 15, 2020
ghost locked as resolved and limited conversation to collaborators on Mar 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @Ivanidzo4ka@shauheen@artemiusgreat@eerhardt@sfilipi

      Issue actions
        Exception when converting PredictedLabel from Key To Value · Issue #3090 · dotnet/machinelearning