-
Notifications
You must be signed in to change notification settings - Fork 159
Description
(Full credit to @sandersdan for the following analysis and design. I'm just turning his doc into a bug.)
The VideoFrame.planes
API allows reading back individual planes, but it has drawbacks:
- The implementation may map and unmap the underlying frame for each plane, which can be slow when all planes are read.
- It’s overly complicated to read all planes (partners have filed bugs showing its easy to get wrong)
- Compared to its complexity, it doesn’t actually offer much control.
This issue proposes a simpler API for copying all planes at once.
Concepts
Region
A region is a {top, left, width, height}
dictionary. VideoFrame
should be updated to use this structure:
- Add
codedRegion
, which is always{top: 0, left: 0, width: codedWidth, height: codedHeight}
.- We could deprecate
codedWidth
andcodedHeight
, but this only seems to complicate constructingVideoFrames
.
- We could deprecate
- Add
visibleRegion
, replacing thecrop
parameters.
The default region is visibleRegion
.
elementSize
The number of bytes in a sample. Each Plane
has an elementSize
attribute.
Stride
The number of bytes from the start of one row to the start of the next row, must be at least elementSize * region.width
.
The default stride is exactly elementSize * region.width
.
Layout
A layout
is a list of {offset, stride}
dictionaries, one for each plane. offset
is the location in a buffer that a plane starts.
The default offset
is tightly packed (each plane immediately follows the previous), and either all offsets should be provided or none should be.
API
// Returns the size in bytes of a buffer that can hold a copy of |region|.
VideoFrame.allocationSize({region});
Plane.allocationSize({region, stride});
// Copies all planes into an output buffer |dst|.
VideoFrame.readInto(dst, {region, layout}); // returns |layout|
Plane.readInto(dst, {region, stride}); // returns |stride|
// Returns a new frame containing a copy of a region. Holding the new frame for an
// extended period of time will not stall decoding.
VideoFrame.copy({region});
Changes to the planes API
Plane
- Add codedRegion, visibleRegion.
- Add elementSize.
- Rename columns to codedWidth and rows to codedHeight.
- Deprecate length. Apps should call allocationSize().
- Deprecate stride. It’s not relevant anymore.
PlaneInit
- Deprecate rows.
- Add offset.
Examples
Copy visibleRegion to a new buffer (packed)
let buf = new ArrayBuffer(frame.allocationSize());
let layout = frame.readInto(buf);
// For 1080p I420, |layout| = [{offset: 0, stride: 1920},
// {offset: 2073600, stride: 960},
// {offset: 2592000, stride: 960}]
Copy codedRegion to a WASM frame pool
let region = frame.codedRegion;
let layout = wasmApp.allocateYUV(region.width, region.height);
frame.readInto(wasmApp.memory, {region, layout});