Skip to content

Bring back support for ConfusionMatrix in the new API #2009

Closed
@abgoswam

Description

@abgoswam

While updating tests from the old AP to the new API, we found that the new API does not support ConfusionMatrix in the evaluation output.

public MultiClassClassifierMetrics Evaluate(IDataView data, string label, string score, string predictedLabel)
{
Host.CheckValue(data, nameof(data));
Host.CheckNonEmpty(label, nameof(label));
Host.CheckNonEmpty(score, nameof(score));
Host.CheckNonEmpty(predictedLabel, nameof(predictedLabel));
var roles = new RoleMappedData(data, opt: false,
RoleMappedSchema.ColumnRole.Label.Bind(label),
RoleMappedSchema.CreatePair(MetadataUtils.Const.ScoreValueKind.Score, score),
RoleMappedSchema.CreatePair(MetadataUtils.Const.ScoreValueKind.PredictedLabel, predictedLabel));
var resultDict = ((IEvaluator)this).Evaluate(roles);
Host.Assert(resultDict.ContainsKey(MetricKinds.OverallMetrics));
var overall = resultDict[MetricKinds.OverallMetrics];

In the Legacy API, there was support for ConfusionMatrix

public static partial class Evaluate
{
[TlcModule.EntryPoint(Name = "Models.ClassificationEvaluator", Desc = "Evaluates a multi class classification scored dataset.")]
public static CommonOutputs.ClassificationEvaluateOutput MultiClass(IHostEnvironment env, MultiClassMamlEvaluator.Arguments input)
{
Contracts.CheckValue(env, nameof(env));
var host = env.Register("EvaluateMultiClass");
host.CheckValue(input, nameof(input));
EntryPointUtils.CheckInputArgs(host, input);
MatchColumns(host, input, out string label, out string weight, out string name);
IMamlEvaluator evaluator = new MultiClassMamlEvaluator(host, input);
var data = new RoleMappedData(input.Data, label, null, null, weight, name);
var metrics = evaluator.Evaluate(data);
var warnings = ExtractWarnings(host, metrics);
var overallMetrics = ExtractOverallMetrics(host, metrics, evaluator);
var perInstanceMetrics = evaluator.GetPerInstanceMetrics(data);
var confusionMatrix = ExtractConfusionMatrix(host, metrics);

We should add back the checks for ConfusionMatrix in the following tests :

  • TrainAndPredictIrisModelTest
  • TrainAndPredictIrisModelWithStringLabelTest

@codemzs @Ivanidzo4ka

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions