Skip to content

Update MacroUtils to map trainer kinds to the correct suffix of trainer entry point names #113

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
May 14, 2018
Merged

Update MacroUtils to map trainer kinds to the correct suffix of trainer entry point names #113

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
02f72c9
Update suffix of trainer entry point names by trainer kind.
yaeldMS May 10, 2018
ff6736d
Address PR comments.
yaeldMS May 10, 2018
1d01831
Add unit test.
yaeldMS May 10, 2018
f1bbeda
Merge remote-tracking branch 'upstream/master' into macroutils
yaeldMS May 11, 2018
db8492d
Update C# API
yaeldMS May 11, 2018
4267412
Move unit test to TestAutoInference and fix EntryPointCatalog test.
yaeldMS May 11, 2018
9bb6c89
Trigger build.
yaeldMS May 11, 2018
be06047
Add reference to the test project to make the sweeper entry point vis…
yaeldMS May 11, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions ZBaselines/Common/EntryPoints/core_ep-list.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,13 @@ Models.NaiveCalibrator Apply a Naive calibrator to an input model Microsoft.ML.R
Models.OneVersusAll One-vs-All macro (OVA) Microsoft.ML.Runtime.EntryPoints.OneVersusAllMacro OVA Microsoft.ML.Runtime.EntryPoints.OneVersusAllMacro+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.OneVersusAllMacro+Output]
Models.OvaModelCombiner Combines a sequence of PredictorModels into a single model Microsoft.ML.Runtime.Learners.OvaPredictor CombineOvaModels Microsoft.ML.Runtime.EntryPoints.ModelOperations+CombineOvaPredictorModelsInput Microsoft.ML.Runtime.EntryPoints.ModelOperations+PredictorModelOutput
Models.PAVCalibrator Apply a PAV calibrator to an input model Microsoft.ML.Runtime.Internal.Calibration.Calibrate Pav Microsoft.ML.Runtime.Internal.Calibration.Calibrate+NoArgumentsInput Microsoft.ML.Runtime.EntryPoints.CommonOutputs+CalibratorOutput
Models.PipelineSweeper AutoML pipeline sweeping optimzation macro. Microsoft.ML.Runtime.EntryPoints.PipelineSweeperMacro PipelineSweep Microsoft.ML.Runtime.EntryPoints.PipelineSweeperMacro+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.PipelineSweeperMacro+Output]
Models.PlattCalibrator Apply a Platt calibrator to an input model Microsoft.ML.Runtime.Internal.Calibration.Calibrate Platt Microsoft.ML.Runtime.Internal.Calibration.Calibrate+NoArgumentsInput Microsoft.ML.Runtime.EntryPoints.CommonOutputs+CalibratorOutput
Models.QuantileRegressionEvaluator Evaluates a quantile regression scored dataset. Microsoft.ML.Runtime.Data.Evaluate QuantileRegression Microsoft.ML.Runtime.Data.QuantileRegressionMamlEvaluator+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+CommonEvaluateOutput
Models.RankerEvaluator Evaluates a ranking scored dataset. Microsoft.ML.Runtime.Data.Evaluate Ranking Microsoft.ML.Runtime.Data.RankerMamlEvaluator+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+CommonEvaluateOutput
Models.RegressionEvaluator Evaluates a regression scored dataset. Microsoft.ML.Runtime.Data.Evaluate Regression Microsoft.ML.Runtime.Data.RegressionMamlEvaluator+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+CommonEvaluateOutput
Models.Summarizer Summarize a linear regression predictor. Microsoft.ML.Runtime.EntryPoints.SummarizePredictor Summarize Microsoft.ML.Runtime.EntryPoints.SummarizePredictor+Input Microsoft.ML.Runtime.EntryPoints.CommonOutputs+SummaryOutput
Models.SweepResultExtractor Extracts the sweep result. Microsoft.ML.Runtime.EntryPoints.PipelineSweeperMacro ExtractSweepResult Microsoft.ML.Runtime.EntryPoints.PipelineSweeperMacro+ResultInput Microsoft.ML.Runtime.EntryPoints.PipelineSweeperMacro+Output
Models.TrainTestBinaryEvaluator Train test for binary classification Microsoft.ML.Runtime.EntryPoints.TrainTestBinaryMacro TrainTestBinary Microsoft.ML.Runtime.EntryPoints.TrainTestBinaryMacro+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.TrainTestBinaryMacro+Output]
Models.TrainTestEvaluator General train test for any supported evaluator Microsoft.ML.Runtime.EntryPoints.TrainTestMacro TrainTest Microsoft.ML.Runtime.EntryPoints.TrainTestMacro+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.TrainTestMacro+Output]
Trainers.AveragedPerceptronBinaryClassifier Train a Average perceptron. Microsoft.ML.Runtime.Learners.AveragedPerceptronTrainer TrainBinary Microsoft.ML.Runtime.Learners.AveragedPerceptronTrainer+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+BinaryClassificationOutput
Expand Down
358 changes: 358 additions & 0 deletions ZBaselines/Common/EntryPoints/core_manifest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1723,6 +1723,105 @@
"ITrainerOutput"
]
},
{
"Name": "Models.PipelineSweeper",
"Desc": "AutoML pipeline sweeping optimzation macro.",
"FriendlyName": null,
"ShortName": null,
"Inputs": [
{
"Name": "TrainingData",
"Type": "DataView",
"Desc": "The data to be used for training.",
"Aliases": [
"train"
],
"Required": true,
"SortOrder": 1.0,
"IsNullable": false
},
{
"Name": "TestingData",
"Type": "DataView",
"Desc": "The data to be used for testing.",
"Aliases": [
"test"
],
"Required": true,
"SortOrder": 2.0,
"IsNullable": false
},
{
"Name": "StateArguments",
"Type": {
"Kind": "Component",
"ComponentKind": "AutoMlStateBase"
},
"Desc": "The arguments for creating an AutoMlState component.",
"Aliases": [
"args"
],
"Required": false,
"SortOrder": 3.0,
"IsNullable": false,
"Default": null
},
{
"Name": "State",
"Type": {
"Kind": "C# Object",
"ItemType": "Microsoft.ML.Runtime.EntryPoints.IMlState"
},
"Desc": "The stateful object conducting of the autoML search.",
"Aliases": [
"state"
],
"Required": false,
"SortOrder": 3.0,
"IsNullable": false,
"Default": null
},
{
"Name": "BatchSize",
"Type": "Int",
"Desc": "Number of candidate pipelines to retrieve each round.",
"Aliases": [
"bsize"
],
"Required": true,
"SortOrder": 4.0,
"IsNullable": false,
"Default": 0
},
{
"Name": "CandidateOutputs",
"Type": {
"Kind": "Array",
"ItemType": "DataView"
},
"Desc": "Output datasets from previous iteration of sweep.",
"Required": false,
"SortOrder": 7.0,
"IsNullable": false,
"Default": null
}
],
"Outputs": [
{
"Name": "State",
"Type": {
"Kind": "C# Object",
"ItemType": "Microsoft.ML.Runtime.EntryPoints.IMlState"
},
"Desc": "Stateful autoML object, keeps track of where the search in progress."
},
{
"Name": "Results",
"Type": "DataView",
"Desc": "Results of the sweep, including pipelines (as graph strings), IDs, and metric values."
}
]
},
{
"Name": "Models.PlattCalibrator",
"Desc": "Apply a Platt calibrator to an input model",
Expand Down Expand Up @@ -2209,6 +2308,44 @@
}
]
},
{
"Name": "Models.SweepResultExtractor",
"Desc": "Extracts the sweep result.",
"FriendlyName": null,
"ShortName": null,
"Inputs": [
{
"Name": "State",
"Type": {
"Kind": "C# Object",
"ItemType": "Microsoft.ML.Runtime.EntryPoints.IMlState"
},
"Desc": "The stateful object conducting of the autoML search.",
"Aliases": [
"state"
],
"Required": false,
"SortOrder": 1.0,
"IsNullable": false,
"Default": null
}
],
"Outputs": [
{
"Name": "State",
"Type": {
"Kind": "C# Object",
"ItemType": "Microsoft.ML.Runtime.EntryPoints.IMlState"
},
"Desc": "Stateful autoML object, keeps track of where the search in progress."
},
{
"Name": "Results",
"Type": "DataView",
"Desc": "Results of the sweep, including pipelines (as graph strings), IDs, and metric values."
}
]
},
{
"Name": "Models.TrainTestBinaryEvaluator",
"Desc": "Train test for binary classification",
Expand Down Expand Up @@ -17200,6 +17337,203 @@
}
],
"Components": [
{
"Kind": "AutoMlEngine",
"Components": [
{
"Name": "Defaults",
"Desc": "AutoML engine that returns learners with default settings.",
"FriendlyName": "Defaults Engine",
"Settings": []
},
{
"Name": "Rocket",
"Desc": "AutoML engine that consists of distinct, hierarchical stages of operation.",
"FriendlyName": "Rocket Engine",
"Settings": [
{
"Name": "TopKLearners",
"Type": "Int",
"Desc": "Number of learners to retain for second stage.",
"Aliases": [
"topk"
],
"Required": false,
"SortOrder": 1.0,
"IsNullable": false,
"Default": 2
},
{
"Name": "SecondRoundTrialsPerLearner",
"Type": "Int",
"Desc": "Number of trials for retained second stage learners.",
"Aliases": [
"stage2num"
],
"Required": false,
"SortOrder": 2.0,
"IsNullable": false,
"Default": 5
},
{
"Name": "RandomInitialization",
"Type": "Bool",
"Desc": "Use random initialization only.",
"Aliases": [
"randinit"
],
"Required": false,
"SortOrder": 3.0,
"IsNullable": false,
"Default": false
},
{
"Name": "NumInitializationPipelines",
"Type": "Int",
"Desc": "Number of initilization pipelines, used for random initialization only.",
"Aliases": [
"numinitseeds"
],
"Required": false,
"SortOrder": 4.0,
"IsNullable": false,
"Default": 20
}
]
},
{
"Name": "UniformRandom",
"Desc": "AutoML engine using uniform random sampling.",
"FriendlyName": "Uniform Random Engine",
"Settings": []
}
]
},
{
"Kind": "AutoMlStateBase",
"Components": [
{
"Name": "AutoMlState",
"Desc": "State of an AutoML search and search space.",
"FriendlyName": "AutoML State",
"Aliases": [
"automlst"
],
"Settings": [
{
"Name": "Metric",
"Type": {
"Kind": "Enum",
"Values": [
"Auc",
"AccuracyMicro",
"AccuracyMacro",
"L2",
"F1",
"AuPrc",
"TopKAccuracy",
"Rms",
"LossFn",
"RSquared",
"LogLoss",
"LogLossReduction",
"Ndcg",
"Dcg",
"PositivePrecision",
"PositiveRecall",
"NegativePrecision",
"NegativeRecall",
"DrAtK",
"DrAtPFpr",
"DrAtNumPos",
"NumAnomalies",
"ThreshAtK",
"ThreshAtP",
"ThreshAtNumPos",
"Nmi",
"AvgMinScore",
"Dbi"
]
},
"Desc": "Supported metric for evaluator.",
"Aliases": [
"metric"
],
"Required": true,
"SortOrder": 150.0,
"IsNullable": false,
"Default": "Auc"
},
{
"Name": "Engine",
"Type": {
"Kind": "Component",
"ComponentKind": "AutoMlEngine"
},
"Desc": "AutoML engine (pipeline optimizer) that generates next candidates.",
"Aliases": [
"engine"
],
"Required": true,
"SortOrder": 150.0,
"IsNullable": false
},
{
"Name": "TrainerKind",
"Type": {
"Kind": "Enum",
"Values": [
"SignatureBinaryClassifierTrainer",
"SignatureMultiClassClassifierTrainer",
"SignatureRankerTrainer",
"SignatureRegressorTrainer",
"SignatureMultiOutputRegressorTrainer",
"SignatureAnomalyDetectorTrainer",
"SignatureClusteringTrainer"
]
},
"Desc": "Kind of trainer for task, such as binary classification trainer, multiclass trainer, etc.",
"Aliases": [
"tk"
],
"Required": true,
"SortOrder": 150.0,
"IsNullable": false,
"Default": "SignatureBinaryClassifierTrainer"
},
{
"Name": "TerminatorArgs",
"Type": {
"Kind": "Component",
"ComponentKind": "SearchTerminator"
},
"Desc": "Arguments for creating terminator, which determines when to stop search.",
"Aliases": [
"term"
],
"Required": true,
"SortOrder": 150.0,
"IsNullable": false
},
{
"Name": "RequestedLearners",
"Type": {
"Kind": "Array",
"ItemType": "String"
},
"Desc": "Learner set to sweep over (if available).",
"Aliases": [
"learners"
],
"Required": false,
"SortOrder": 150.0,
"IsNullable": false,
"Default": null
}
]
}
]
},
{
"Kind": "CalibratorTrainer",
"Components": [
Expand Down Expand Up @@ -21441,6 +21775,30 @@
}
]
},
{
"Kind": "SearchTerminator",
"Components": [
{
"Name": "IterationLimited",
"Desc": "Terminators a sweep based on total number of iterations.",
"FriendlyName": "Pipeline Sweep Iteration Terminator",
"Settings": [
{
"Name": "FinalHistoryLength",
"Type": "Int",
"Desc": "Total number of iterations.",
"Aliases": [
"length"
],
"Required": true,
"SortOrder": 150.0,
"IsNullable": false,
"Default": 0
}
]
}
]
},
{
"Kind": "StopWordsRemover",
"Components": [
Expand Down
Loading