Exception in AutoML Hyperparam Sweeper

[daholste]

System information

• OS version/distro: Windows 10
• .NET Version (eg., dotnet --info): .NET Core 2.1

Source code / logs

System.InvalidOperationException: Value not in correct range
   at Microsoft.ML.Runtime.Contracts.Check(Boolean f, String msg) in C:\MLDotNet\src\Microsoft.ML.Core\Utilities\Contracts.cs:line 496
   at Microsoft.ML.Runtime.Sweeper.FloatValueGenerator.NormalizeValue(IParameterValue value) in C:\MLDotNet\src\Microsoft.ML.Sweeper\Parameters.cs:line 442
   at Microsoft.ML.Runtime.Sweeper.Algorithms.SweeperProbabilityUtils.ParameterSetAsFloatArray(IHost host, IValueGenerator[] sweepParams, ParameterSet ps, Boolean expandCategoricals) in C:\MLDotNet\src\Microsoft.ML.Sweeper\Algorithms\SweeperProbabilityUtils.cs:line 207
   at Microsoft.ML.Runtime.Sweeper.Algorithms.KdoSweeper.SampleChild(ParameterSet parent, Double fitness, Int32 n, IRunResult[] previousRuns, Double rMean, Double rVar, Boolean isMetricMaximizing) in C:\MLDotNet\src\Microsoft.ML.Sweeper\Algorithms\KdoSweeper.cs:line 206
   at Microsoft.ML.Runtime.Sweeper.Algorithms.KdoSweeper.GenerateChildConfigurations(IRunResult[] history, Int32[] parentIndicies, Double[] weights, IRunResult[] previousRuns, Double rMean, Double rVar) in C:\MLDotNet\src\Microsoft.ML.Sweeper\Algorithms\KdoSweeper.cs:line 187
   at Microsoft.ML.Runtime.Sweeper.Algorithms.KdoSweeper.GenerateCandidateConfigurations(Int32 numOfCandidates, IRunResult[] previousRuns) in C:\MLDotNet\src\Microsoft.ML.Sweeper\Algorithms\KdoSweeper.cs:line 176
   at Microsoft.ML.Runtime.Sweeper.Algorithms.KdoSweeper.ProposeSweeps(Int32 maxSweeps, IEnumerable`1 previousRuns) in C:\MLDotNet\src\Microsoft.ML.Sweeper\Algorithms\KdoSweeper.cs:line 153
   at Microsoft.ML.Runtime.PipelineInference.PipelineOptimizerBase.SampleHyperparameters(SuggestedLearner learner, ISweeper sweeper, Boolean isMaximizingMetric, PipelinePattern[] history) in C:\MLDotNet\src\Microsoft.ML.PipelineInference\Interfaces\IPipelineOptimizer.cs:line 143
   at Microsoft.ML.Runtime.PipelineInference.RocketEngine.SampleHyperparameters(SuggestedLearner learner, PipelinePattern[] history) in C:\MLDotNet\src\Microsoft.ML.PipelineInference\AutoMlEngines\RocketEngine.cs:line 128
   at Microsoft.ML.Runtime.PipelineInference.RocketEngine.NextCandidates(PipelinePattern[] history, Int32 numCandidates, Boolean defaultHyperParams, Boolean uniformRandomTransforms) in C:\MLDotNet\src\Microsoft.ML.PipelineInference\AutoMlEngines\RocketEngine.cs:line 309
   at Microsoft.ML.Runtime.PipelineInference.RocketEngine.GetNextCandidates(IEnumerable`1 history, Int32 numCandidates, RoleMappedData dataRoles) in C:\MLDotNet\src\Microsoft.ML.PipelineInference\AutoMlEngines\RocketEngine.cs:line 266
   at Microsoft.ML.Runtime.PipelineInference.AutoInference.AutoMlMlState.MainLearningLoop(Int32 batchSize, Int32 numOfTrainingRows) in C:\MLDotNet\src\Microsoft.ML.PipelineInference\AutoInference.cs:line 213

[daholste]

I've also seen this exception in the same method:

System.ArgumentException: Array was not a two-dimensional array.
   at System.Array.SetValue(Object value, Int32 index1, Int32 index2)
   at Microsoft.ML.Runtime.Sweeper.Algorithms.SweeperProbabilityUtils.Normalize(Double[] weights) in C:\MLDotNet\src\Microsoft.ML.Sweeper\Algorithms\SweeperProbabilityUtils.cs:line 145
   at Microsoft.ML.Runtime.PipelineInference.PipelineOptimizerBase.LearnerHistoryToWeights(PipelinePattern[] history, Boolean isMaximizingMetric) in C:\MLDotNet\src\Microsoft.ML.PipelineInference\Interfaces\IPipelineOptimizer.cs:line 131
   at Microsoft.ML.Runtime.PipelineInference.RocketEngine.GetTopLearners(IEnumerable`1 history) in C:\MLDotNet\src\Microsoft.ML.PipelineInference\AutoMlEngines\RocketEngine.cs:line 211
   at Microsoft.ML.Runtime.PipelineInference.RocketEngine.GetNextCandidates(IEnumerable`1 history, Int32 numCandidates, RoleMappedData dataRoles) in C:\MLDotNet\src\Microsoft.ML.PipelineInference\AutoMlEngines\RocketEngine.cs:line 236
   at Microsoft.ML.Runtime.PipelineInference.AutoInference.AutoMlMlState.MainLearningLoop(Int32 batchSize, Int32 numOfTrainingRows) in C:\MLDotNet\src\Microsoft.ML.PipelineInference\AutoInference.cs:line 213

I'm going to try replacing

machinelearning/src/Microsoft.ML.Sweeper/Algorithms/SweeperProbabilityUtils.cs

Line 144 in e090660

weights.SetValue(1, 0, weights.Length);

with

for(var i = 0; i < weights.Length; i++)
{
   weights[i] = 1;
}

[TomFinley]

Hi @daholste and @rogancarr . As you've found with this, this code in particular is highly unstable and not quite ready for primetime. I think someone has a plan, or at least a desire, to have it continue to be developed, but not of course as part of the ML.NET repository. Our plan is pursuant to #689, as well as out of absolute necessity as we're getting rid of the ML.NET legacy API, is that it will be removed in a couple weeks.

Maybe once it is moved to its new home (wherever that may be), people that actually develop it can offer greater support than we can. Sorry about that!