Skip to content

Simplifying and optimizing VideoFrame readback  #157

@chcunningham

Description

@chcunningham

(Full credit to @sandersdan for the following analysis and design. I'm just turning his doc into a bug.)

The VideoFrame.planes API allows reading back individual planes, but it has drawbacks:

  • The implementation may map and unmap the underlying frame for each plane, which can be slow when all planes are read.
  • It’s overly complicated to read all planes (partners have filed bugs showing its easy to get wrong)
  • Compared to its complexity, it doesn’t actually offer much control.

This issue proposes a simpler API for copying all planes at once.

Concepts

Region

A region is a {top, left, width, height} dictionary. VideoFrame should be updated to use this structure:

  • Add codedRegion, which is always {top: 0, left: 0, width: codedWidth, height: codedHeight}.
    • We could deprecate codedWidth and codedHeight, but this only seems to complicate constructing VideoFrames.
  • Add visibleRegion, replacing the crop parameters.

The default region is visibleRegion.

elementSize

The number of bytes in a sample. Each Plane has an elementSize attribute.

Stride

The number of bytes from the start of one row to the start of the next row, must be at least elementSize * region.width.

The default stride is exactly elementSize * region.width.

Layout

A layout is a list of {offset, stride} dictionaries, one for each plane. offset is the location in a buffer that a plane starts.

The default offset is tightly packed (each plane immediately follows the previous), and either all offsets should be provided or none should be.

API

// Returns the size in bytes of a buffer that can hold a copy of |region|.
VideoFrame.allocationSize({region});
Plane.allocationSize({region, stride});

// Copies all planes into an output buffer |dst|.
VideoFrame.readInto(dst, {region, layout});  // returns |layout|
Plane.readInto(dst, {region, stride});       // returns |stride|

// Returns a new frame containing a copy of a region. Holding the new frame for an
// extended period of time will not stall decoding.
VideoFrame.copy({region});

Changes to the planes API

  • Plane
    • Add codedRegion, visibleRegion.
    • Add elementSize.
    • Rename columns to codedWidth and rows to codedHeight.
    • Deprecate length. Apps should call allocationSize().
    • Deprecate stride. It’s not relevant anymore.
  • PlaneInit
    • Deprecate rows.
    • Add offset.

Examples

Copy visibleRegion to a new buffer (packed)

let buf = new ArrayBuffer(frame.allocationSize());
let layout = frame.readInto(buf);

// For 1080p I420, |layout| = [{offset: 0, stride: 1920},
//                             {offset: 2073600, stride: 960},
//                             {offset: 2592000, stride: 960}]

Copy codedRegion to a WASM frame pool

let region = frame.codedRegion;
let layout = wasmApp.allocateYUV(region.width, region.height);
frame.readInto(wasmApp.memory, {region, layout});

Metadata

Metadata

Assignees

No one assigned

    Labels

    breakingInterface changes that would break current usage (producing errors or undesired behavior).

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions