Model Execution API

[pyu10055]

There are couple questions on the existing execution API:

1. The current model execution API requires users to provide output buffers before execution, this is not very convenient since this is an extra step for the user and user might not know the shape of the output before hand. Also, for many model this output shape is based on the input shape, it is an extra burden for users to find that out.

2. The current execution is build on the compilation of the full graph, while the execution API does not prevent users to execution the sub-graph of the model, it is not clear why the pre-compilation is needed and should it be internal of the execution, so it can take care of sub-graph execution.

[huningxin]

1. this is not very convenient since this is an extra step for the user and user might not know the shape of the output before hand.

Good catch! One possible solution is allowing user code to query the output shape, such as Android NNAPI @miaowang14.

2. The current execution is build on the compilation of the full graph, while the execution API does not prevent users to execution the sub-graph of the model,

I don't quite get this issue. Could you please elaborate the case that the API allows to compile the full graph while execute a sub-graph?

[gramalingam]

But the output shape may not always be statically known. It would be good if the API allows for the possibility of the output buffer being allocated during execution when its size is known. Can we treat the call to "setOutput" as optional (not required)?

[huningxin]

But the output shape may not always be statically known. It would be good if the API allows for the possibility of the output buffer being allocated during execution when its size is known.

+1. The API may also need to allow specifying the input shape that is not supported currently.

Can we treat the call to "setOutput" as optional (not required)?

Then how to get the execution results? Could you please elaborate the idea?

[gramalingam]

The execution API essentialy needs to allow a way to specify the inputs, optionally specify output buffers, and get back the outputs. I think various possibilities exist for this. In the long run, it may actually be useful to support multiple signatures to support different ways of doing this (mostly as a syntactic convenience).

One option would be to have a return result (for example, a dictionary mapping output-name to its value).

[wchao1115]

One option would be to have a return result (for example, a dictionary mapping output-name to its value).

+1.

I'm also looking at a few related Operand type issues. Maybe we can make one sweeping change.

1. The Operand type should be limited to tensor types i.e. only tensor-float32 type is allowed and not float32. Non-tensor arrays should use sequences, lists, or TypedArray for clarity since a tensor can also be a GPU resource.
2. Quantized tensor as a type should be avoided as it forces all the defined operators to support quantized type, which isn't realistic (OperandType of gemm / matmul return  #84). It was previously agreed that quantized types should be factored out (Remove quantization-specific params from OperandDescriptor #44).
3. Operand should support sync and async download of tensor data to a TypedArray. This should address the first issue above.

[huningxin]

The execution API essentially needs to allow a way to specify the inputs, optionally specify output buffers, and get back the outputs.

+1.

According to current Execution API, the missing functionalities are specifying the input shape, get the output shape and data. So semantically, we may extend the setInput method with an optional shape parameter and add getOutputShape and readOutput methods, such as

 interface Execution {
-  void setInput(DOMString name, ArrayBufferView data);
+  void setInput(DOMString name, ArrayBufferView data, optional sequence<long> shape);
   void setOutput(DOMString name, ArrayBufferView data);
   Promise<void> startCompute();
+  sequence<long> getOutputShape(DOMString name);
+  Promise<void> readOutput(DOMString name, ArrayBufferView buffer);
 };

With that, the sample code for dynamic input/output shape would be

// Create an Execution object for the compiled model.
const execution = compilation.createExecution();

const inputShape = [2, 2, 2, 2];
// Setup the input buffers with value 1.
const inputBuffer1 = new Float32Array(sizeOfShape(inputShape)).fill(1);
const inputBuffer2 = new Float32Array(sizeOfShape(inputShape)).fill(1);

// Specify the input data and its shape.
execution.setInput('input1', inputBuffer1, inputShape);
execution.setInput('input2', inputBuffer2, inputShape);

// Start the asynchronous computation.
await execution.startCompute();

// Get the output shape and allocate the buffer.
const outputShape = execution.getOutputShape('output');
const outputBuffer = new Float32Array(sizeOfShape(outputShape));

// Read the output values into the allocated buffer.
await execution.readOutput('output', outputBuffer);

// The computed result is now in outputBuffer.
console.log(outputBuffer);

One option would be to have a return result (for example, a dictionary mapping output-name to its value).

This could be a syntactic improvement. One difference is above proposal allows to get the value of a single output instead of returning them all.

[huningxin]

I'm also looking at a few related Operand type issues.

@wchao1115 , I agree there are rooms to improve Operand. However, Operand is only used for graph building in current API. Do you propose to use it also for graph execution?

[wchao1115]

@huningxin I understand that Operand as it is defined now acts more like a graph edge and not necessarily a data type like tensor. But in practice the two concepts are inseparable; it would be much simpler for a graph API to think of them as one. Both TensorFlow and PyTorch handle it this way.

Concretely, a few things stand out for me:

1. The operator API already accepts both attribute and tensor arguments, with the former specified as native data types e.g. float, bool, etc. Therefore there is no point at also defining an operand of a float32 data type as a separate notion from an operand of a tensor-float32 data type. An operand with float32 type should just mean a graph edge that would carry a tensor of type float32.
2. Although requiring a CPU buffer view to construct an input operand may make sense, an output operand could be backed by a GPU tensor that is constructed on the fly by the graph execution process. So, requiring the app to always download the output tensor back to the CPU buffer view doesn't make sense, since it may want to pass the GPU resource along to other API that may accept it e.g. with ML models that emit output images, etc. A graph execution may also produce multiple outputs, and the app may choose to ignore some of them.
3. Quantized tensor further clouds the Operand type with more procedural data that doesn't really contribute to the identity of the operand type (as a graph edge). As previously pointed out, it also unnecessarily raises the requirement that all the operator API must support quantized types; a requirement that actually cannot be fulfilled in all cases.

A leaner design in my mind would be one with an operand exposing a method for the app to optionally download its content to an ArrayBuffer only if they want to. This call would fail if it's made before the graph is executed since the output tensor won't be known then. With that API addition, it turns out that execution.setOutput becomes entirely unnecessary since a model would be previously created with the named dictionary of the output operands (see nn.createModel) and that an ArrayBuffer of an output operand is created by the execution and not the construction of the graph.

[kpu]

2. Quantized tensor as a type should be avoided as it forces all the defined operators to support quantized type, which isn't realistic (#84). It was previously agreed that quantized types should be factored out (#44).

In #17 it was "RESOLVED: The specification will reference a subset of the ONNX operations". The ONNX operations include type constraints i.e. https://github.com/onnx/onnx/blob/master/docs/Operators.md#MatMulInteger . Can we just follow them?

[huningxin]

So, requiring the app to always download the output tensor back to the CPU buffer view doesn't make sense, since it may want to pass the GPU resource along to other API that may accept it e.g. with ML models that emit output images, etc.

That's a good point. Besides passing the GPU-backed output tensor to other API, I think the Execution API should be able to accept the output tensor as input. With that, it would enable pipelining multiple graph executions without moving data cross devices.

A leaner design in my mind would be one with an operand exposing a method for the app to optionally download its content to an ArrayBuffer only if they want to. This call would fail if it's made before the graph is executed since the output tensor won't be known then.

My understanding is the input and output tensor of Execution API should be device dependent, say GPU resource. However, Operand is device independent, for example, the nn.input is just a symbolic one that never has values, and nn.constant doesn't know which device to store its values until the compilation.

So I am thinking about adding a Tensor interface that is backed by device memory. The Execution would accept input Tensors and output Tensors. The Tensor interface would have methods for async shape querying and data downloading. As Compilation has target device, we may allow creating Tensor from Compilation. A code sketch would be

// Create an Execution object for the compiled model.
const execution = compilation.createExecution();

const inputShape = [2, 2, 2, 2];
// Setup the input buffers with value 1.
const inputBuffer1 = new Float32Array(sizeOfShape(inputShape)).fill(1);
const inputBuffer2 = new Float32Array(sizeOfShape(inputShape)).fill(1);

// Create the input and output tensors for compilation device.
const inputTensor1 = compilation.device.tensor(float32TensorType, inputBuffer1);
const inputTensor2 = compilation.device.tensor(float32TensorType, inputBuffer2);
const outputTensor = compilation.device.tensor();

// Set input and output tensors to an execution.
execution.setInput('input1', inputTensor1);
execution.setInput('input2', inputTensor2);
execution.setOutput('output', outputTensor);

// Start the computation, no async anymore, as tensor reading back would be async.
execution.startCompute(inputs, outputs);

// Query the output shape, await for shape inference done.
const outputShape = await outputTensor.getDimensions();
const outputBuffer = new Float32Array(sizeOfShape(outputShape));

// Read the data back, await for inference done.
await outputTensor.readInto(outputBuffer);
console.log(outputBuffer);

The execution.setOutput may be still useful, as it would allow to reuse a Tensor from one execution to another.

[gramalingam]

Hi @wchao1115 : one clarification about the attributes and operands: I think that the primary distinction between attributes and operands are that attributes represent values that are known statically, at the time the graph/model is built; on the other hand, operands represent values that typically will not be known statically (though, in some special cases they might be).

So, one key question we should address is how we want to handle scalar values that are known only at runtime (and not statically). Many frameworks do use tensors of rank 0 to represent such scalars. We could do the same. In this case, restricting operand types to be tensor-float332 and not float32 would work. So, this is doable, though it may incur a small performance penalty. But since these scalars are not going to be the performance bottleneck, it should be okay.

[wchao1115]

@gramalingam That's a good point. We haven't fully defined the notion of tensor scalar. From the graph execution standpoint, tensors are device-specific resources, either as inputs or outputs of the process.

[wchao1115]

@huningxin I think this is too complicated since the developers now need to deal with 3 different notions -- operand, tensor, and buffer. Besides, the tensors are now created by compilation, which raises more questions than it answers.

Even though tensors are device-dependent, it doesn't mean that we have to surface it as a type to the users. I think the current notion of nn.input is perfectly fine. If in the future we need an ability to construct an operand out of other device resources e.g. video frame, there is nothing stopping us from defining a nn.inputFromVideoFrame method. We don't need to expose the notion of device-specific resource as a type to the users; we just need to give them a way to work with it. TF.js also does it this way. Their so-called Tensor type in spite of its name is still a conceptual type and not a device-dependent resource type.

Same for the output operand, as long as there is a way to retrieve the computed content in it and put it in a CPU buffer or in a future resource type, there is no need to expose the notion of the device-specific resource type to the users.

Also, if you look at how nn.createModel is defined, we already require through this API that the users specify the dictionary of all output operands, even before compilation. So all that we need is to expose a way to get the content out of it and we won't even need setOutput anymore. We also don't need to pass the output operands to startCompute, as it's already defined in the model.

[huningxin]

Also, if you look at how nn.createModel is defined, we already require through this API that the users specify the dictionary of all output operands, even before compilation. So all that we need is to expose a way to get the content out of it and we won't even need setOutput anymore. We also don't need to pass the output operands to startCompute, as it's already defined in the model.

Users may compile the same Model to multiple Compilations with different CompilationOptions and target different underlying devices. In this case, there is only one output Operand associated with the Model. This Operand might not be able to represent the different device resources. For example

const model = await nn.createModel([{name: 'output', operand: output}]);

// e.g. target iGPU
const compilation1 = await model.createCompilation({powerPreference: 'low-power'});
// e.g. target dGPU
const compilation2 = await model.createCompilation({powerPreference: 'high-performance'});

// Create Execution objects for the compiled models.
const execution1 = await compilation1.createExecution();
const execution2 = await compilation2.createExecution();

execution1.startCompute();
execution2.startCompute();

// Which device to read back, iGPU or dGPU?
await output.readInto(buffer);