|
| 1 | +# Entry Points And Helper Classes |
| 2 | + |
| 3 | +## Overview |
| 4 | + |
| 5 | +Entry points are a way to interface with ML.NET components, by specifying an execution graph of connected inputs and outputs of those components. |
| 6 | +Both the manifest describing available components and their inputs/outputs, and an "experiment" graph description, are expressed in JSON. |
| 7 | +The recommended way of interacting with ML.NET through other, non-.NET programming languages, is by composing, and exchanging pipelines or experiment graphs. |
| 8 | + |
| 9 | +Through the documentation, we also refer to entry points as 'entry points nodes', and that is because they are the nodes of the graph representing the experiment. |
| 10 | +The graph 'variables', the various values of the experiment graph JSON properties, serve to describe the relationship between the entry point nodes. |
| 11 | +The 'variables' are therefore the edges of the DAG (Directed Acyclic Graph). |
| 12 | + |
| 13 | +All of ML.NET entry points are described by their manifest. The manifest is another JSON object that documents and describes the structure of an entry points. |
| 14 | +Manifests are referenced to understand what an entry point does, and how it should be constructed, in a graph. |
| 15 | + |
| 16 | +This document briefly describes the structure of the entry points, the structure of an entry point manifest, and mentions the ML.NET classes that help construct an entry point graph. |
| 17 | + |
| 18 | +## EntryPoint manifest - the definition of an entry point |
| 19 | + |
| 20 | +The components manifest is build by scanning the ML.NET assemblies through reflection and searching for types having the: `SignatureEntryPointModule` signature in their `LoadableClass` assembly attribute definition. |
| 21 | +An example of an entry point manifest object, specifically for the `ColumnTypeConverter` transform, is: |
| 22 | + |
| 23 | +```javascript |
| 24 | +{ |
| 25 | + "Name": "Transforms.ColumnTypeConverter", |
| 26 | + "Desc": "Converts a column to a different type, using standard conversions.", |
| 27 | + "FriendlyName": "Convert Transform", |
| 28 | + "ShortName": "Convert", |
| 29 | + "Inputs": [ |
| 30 | + { "Name": "Column", |
| 31 | + "Type": { |
| 32 | + "Kind": "Array", |
| 33 | + "ItemType": { |
| 34 | + "Kind": "Struct", |
| 35 | + "Fields": [ |
| 36 | + { |
| 37 | + "Name": "ResultType", |
| 38 | + "Type": { |
| 39 | + "Kind": "Enum", |
| 40 | + "Values": [ "I1","I2","U2","I4","U4","I8","U8","R4","Num","R8","TX","Text","TXT","BL","Bool","TimeSpan","TS","DT","DateTime","DZ","DateTimeZone","UG","U16" ] |
| 41 | + }, |
| 42 | + "Desc": "The result type", |
| 43 | + "Aliases": [ "type" ], |
| 44 | + "Required": false, |
| 45 | + "SortOrder": 150, |
| 46 | + "IsNullable": true, |
| 47 | + "Default": null |
| 48 | + }, |
| 49 | + { "Name": "Range", |
| 50 | + "Type": "String", |
| 51 | + "Desc": "For a key column, this defines the range of values", |
| 52 | + "Aliases": [ "key" ], |
| 53 | + "Required": false, |
| 54 | + "SortOrder": 150, |
| 55 | + "IsNullable": false, |
| 56 | + "Default": null |
| 57 | + }, |
| 58 | + { "Name": "Name", |
| 59 | + "Type": "String", |
| 60 | + "Desc": "Name of the new column", |
| 61 | + "Aliases": [ "name" ], |
| 62 | + "Required": false, |
| 63 | + "SortOrder": 150, |
| 64 | + "IsNullable": false, |
| 65 | + "Default": null |
| 66 | + }, |
| 67 | + { "Name": "Source", |
| 68 | + "Type": "String", |
| 69 | + "Desc": "Name of the source column", |
| 70 | + "Aliases": [ "src" ], |
| 71 | + "Required": false, |
| 72 | + "SortOrder": 150, |
| 73 | + "IsNullable": false, |
| 74 | + "Default": null |
| 75 | + } |
| 76 | + ] |
| 77 | + } |
| 78 | + }, |
| 79 | + "Desc": "New column definition(s) (optional form: name:type:src)", |
| 80 | + "Aliases": [ "col" ], |
| 81 | + "Required": true, |
| 82 | + "SortOrder": 1, |
| 83 | + "IsNullable": false |
| 84 | + }, |
| 85 | + { "Name": "Data", |
| 86 | + "Type": "DataView", |
| 87 | + "Desc": "Input dataset", |
| 88 | + "Required": true, |
| 89 | + "SortOrder": 2, |
| 90 | + "IsNullable": false |
| 91 | + }, |
| 92 | + { "Name": "ResultType", |
| 93 | + "Type": { |
| 94 | + "Kind": "Enum", |
| 95 | + "Values": [ "I1","I2","U2","I4","U4","I8","U8","R4","Num","R8","TX","Text","TXT","BL","Bool","TimeSpan","TS","DT","DateTime","DZ","DateTimeZone","UG","U16" ] |
| 96 | + }, |
| 97 | + "Desc": "The result type", |
| 98 | + "Aliases": [ "type" ], |
| 99 | + "Required": false, |
| 100 | + "SortOrder": 2, |
| 101 | + "IsNullable": true, |
| 102 | + "Default": null |
| 103 | + }, |
| 104 | + { "Name": "Range", |
| 105 | + "Type": "String", |
| 106 | + "Desc": "For a key column, this defines the range of values", |
| 107 | + "Aliases": [ "key" ], |
| 108 | + "Required": false, |
| 109 | + "SortOrder": 150, |
| 110 | + "IsNullable": false, |
| 111 | + "Default": null |
| 112 | + } |
| 113 | + ], |
| 114 | + "Outputs": [ |
| 115 | + { |
| 116 | + "Name": "OutputData", |
| 117 | + "Type": "DataView", |
| 118 | + "Desc": "Transformed dataset" |
| 119 | + }, |
| 120 | + { |
| 121 | + "Name": "Model", |
| 122 | + "Type": "TransformModel", |
| 123 | + "Desc": "Transform model" |
| 124 | + } |
| 125 | + ], |
| 126 | + "InputKind": ["ITransformInput" ], |
| 127 | + "OutputKind": [ "ITransformOutput" ] |
| 128 | +} |
| 129 | +``` |
| 130 | + |
| 131 | +The respective entry point, constructed based on this manifest would be: |
| 132 | + |
| 133 | +```javascript |
| 134 | + { |
| 135 | + "Name": "Transforms.ColumnTypeConverter", |
| 136 | + "Inputs": { |
| 137 | + "Column": [{ |
| 138 | + "Name": "Features", |
| 139 | + "Source": "Features" |
| 140 | + }], |
| 141 | + "Data": "$data0", |
| 142 | + "ResultType": "R4" |
| 143 | + }, |
| 144 | + "Outputs": { |
| 145 | + "OutputData": "$Convert_Output", |
| 146 | + "Model": "$Convert_TransformModel" |
| 147 | + } |
| 148 | + } |
| 149 | +``` |
| 150 | + |
| 151 | +## `EntryPointGraph` |
| 152 | + |
| 153 | +This class encapsulates the list of nodes (`EntryPointNode`) and edges |
| 154 | +(`EntryPointVariable` inside a `RunContext`) of the graph. |
| 155 | + |
| 156 | +## `EntryPointNode` |
| 157 | + |
| 158 | +This class represents a node in the graph, and wraps an entry point call. It |
| 159 | +has methods for creating and running entry points. It also has a reference to |
| 160 | +the `RunContext` to allow it to get and set values from `EntryPointVariable`s. |
| 161 | + |
| 162 | +To express the inputs that are set through variables, a set of dictionaries |
| 163 | +are used. The `InputBindingMap` maps an input parameter name to a list of |
| 164 | +`ParameterBinding`s. The `InputMap` maps a `ParameterBinding` to a |
| 165 | +`VariableBinding`. For example, if the JSON looks like this: |
| 166 | + |
| 167 | +```javascript |
| 168 | +'foo': '$bar' |
| 169 | +``` |
| 170 | + |
| 171 | +the `InputBindingMap` will have one entry that maps the string "foo" to a list |
| 172 | +that has only one element, a `SimpleParameterBinding` with the name "foo" and |
| 173 | +the `InputMap` will map the `SimpleParameterBinding` to a |
| 174 | +`SimpleVariableBinding` with the name "bar". For a more complicated example, |
| 175 | +let's say we have this JSON: |
| 176 | + |
| 177 | +```javascript |
| 178 | +'foo': [ '$bar[3]', '$baz'] |
| 179 | +``` |
| 180 | + |
| 181 | +the `InputBindingMap` will have one entry that maps the string "foo" to a list |
| 182 | +that has two elements, an `ArrayIndexParameterBinding` with the name "foo" and |
| 183 | +index 0 and another one with index 1. The `InputMap` will map the first |
| 184 | +`ArrayIndexParameterBinding` to an `ArrayIndexVariableBinding` with name "bar" |
| 185 | +and index 3 and the second `ArrayIndexParameterBinding` to a |
| 186 | +`SimpleVariableBinding` with the name "baz". |
| 187 | + |
| 188 | +For outputs, a node assumes that an output is mapped to a variable, so the |
| 189 | +`OutputMap` is a simple dictionary from string to string. |
| 190 | + |
| 191 | +## `EntryPointVariable` |
| 192 | + |
| 193 | +This class represents an edge in the entry point graph. It has a name, a type |
| 194 | +and a value. Variables can be simple, arrays and/or dictionaries. Currently, |
| 195 | +only data views, file handles, predictor models and transform models are |
| 196 | +allowed as element types for a variable. |
| 197 | + |
| 198 | +## `RunContext` |
| 199 | + |
| 200 | +This class is just a container for all the variables in a graph. |
| 201 | + |
| 202 | +## `VariableBinding` and Derived Classes |
| 203 | + |
| 204 | +The abstract base class represents a "pointer to a (part of a) variable". It |
| 205 | +is used in conjunction with `ParameterBinding`s to specify inputs to an entry |
| 206 | +point node. The `SimpleVariableBinding` is a pointer to an entire variable, |
| 207 | +the `ArrayIndexVariableBinding` is a pointer to a specific index in an array |
| 208 | +variable, and the `DictionaryKeyVariableBinding` is a pointer to a specific |
| 209 | +key in a dictionary variable. |
| 210 | + |
| 211 | +## `ParameterBinding` and Derived Classes |
| 212 | + |
| 213 | +The abstract base class represents a "pointer to a (part of a) parameter". It |
| 214 | +parallels the `VariableBinding` hierarchy and it is used to specify the inputs |
| 215 | +to an entry point node. The `SimpleParameterBinding` is a pointer to a |
| 216 | +non-array, non-dictionary parameter, the `ArrayIndexParameterBinding` is a |
| 217 | +pointer to a specific index of an array parameter and the |
| 218 | +`DictionaryKeyParameterBinding` is a pointer to a specific key of a dictionary |
| 219 | +parameter. |
| 220 | + |
| 221 | +## How to create an entry point for an existing ML.NET component |
| 222 | + |
| 223 | +The steps to take, to create an entry point for an existing ML.NET component, are: |
| 224 | +1. Add the `SignatureEntryPointModule` signature to the `LoadableClass` assembly attribute. |
| 225 | +2. Create a public static method, that: |
| 226 | + a. Takes as input, among others, an object representing the arguments of the component you want to expose. |
| 227 | + b. Initializes and run the components, returning one of the nested classes of `Microsoft.ML.Runtime.EntryPoints.CommonOutputs` |
| 228 | + c. Is annotated with the `TlcModule.EntryPoint` attribute |
| 229 | + |
| 230 | +Based on the type of entry point being created, there are further conventions on the name of the method, for example, the Trainers entry points are typically called: 'TrainMultiClass', 'TrainBinary' etc, based on the task. |
| 231 | +Look at [OnlineGradientDescent](../../src/Microsoft.ML.StandardLearners/Standard/Online/OnlineGradientDescent.cs) for an example of a component and its entry point. |
0 commit comments