Decoding WebM files with alpha

[kareljuricka]

Do webcodec decoding operation support alpha channels in webm files?

I'm using https://github.com/Yahweasel/mkvdemuxjs demuxer for getting chunks and webcodes VideoDecoder for getting VideoFrames. But when I draw frame on canvas, alpha layer from input webm file is missing.

I'm not sure if it's problem of demuxer or of webcodecs..
I noticed that encoder does support alpha: 'keep' option, I looked for something similar for decoder but with no luck

[sandersdan]

WebCodecs supports alpha frames, but I don't think any codecs that support alpha are available in any implementation.

Taking VP9 as an example, the alpha is conventionally encoded as a separate stream, and the decoder takes the alpha stream as side data on each packet. WebCodecs doesn't have side data on EncodedVideoChunk, so the only option is to decode the two streams separately and then combine them in JS.

Chrome's current implementation of VideoEncoder will reject any configuration with alpha: "keep".

[yisibl]

Hi @sandersdan, can Chrome encode VP9/VP8 video with alpha now?

[sandersdan]

No, not yet. There is currently no timeline for this feature in WebCodecs.

[StaZhu]

HEVC with Alpha decoding on macOS has recently been implemented in the latest Chrome Canary 108.

It supports WebCodec API well, meanwhile can preserve its alpha layer.

A simple demo here FYI.

[rcunning]

@sandersdan any pointers on how to encode the alpha channel as a separate stream and combine in JS? I have the need to alpha video + webcodecs that will work correctly on Win+Mac in Chrome.

[dalecurtis]

To combine in JS you'll need the opaque pixel data and alpha data in CPU memory. You can then merge them into a I420A frame or RGB convert into RGBA/BGRA.

There's no way to zip hardware decoded opaque data with hardware/software decoded alpha data with the current API, so you'll need to use copyTo() to retrieve one or both if hardware decoding is used.

[rcunning]

To combine in JS you'll need the opaque pixel data and alpha data in CPU memory. You can then merge them into a I420A frame or RGB convert into RGBA/BGRA.

There's no way to zip hardware decoded opaque data with hardware/software decoded alpha data with the current API, so you'll need to use copyTo() to retrieve one or both if hardware decoding is used.

Thanks for responding so quickly @dalecurtis . So it looks pretty easy to extract out the alpha channel with ffmpeg using the alphaextract filter. And also easy to decode both the regular file + alpha file - modifying the webcodec decode sample (https://github.com/w3c/webcodecs/tree/main/samples/video-decode-display). However a couple things:

1. Assuming I can copy these data out correctly, can you elaborate on how I would merge them to RGBA in JS? Is there any sample code you know of that might give me a starting point? Eventually I want put the RGBA frame into a pixi.js texture (i.e. PIXI.Texture.fromBuffer), although it seems like there would be a way to combine the two in webgl/pixi instead of in JS.

2. to use copyTo I need the frame's allocation size, but some files do not include a format in their VideoFrames:

let buffer = new Uint8Array(frame.allocationSize());
frame.copyTo(buffer).then(layout => {
       ...
})

which gives me Uncaught DOMException: Failed to execute 'allocationSize' on 'VideoFrame': Operation is not supported when format is null.
I can work around this by specifying the pix_fmt to yuv420p at encode time, but I am unclear why this fails.

Thank again for your help

[dalecurtis]

More recent versions of Chrome should have less cases where the format is null. When it's null you need to go through ImageBitmap or canvas to read back the pixel data. If the data you readback is in RGBx or BGRx format you just need to replace each alpha byte with the corresponding one from your alpha stream.

If your draw primitives can source the alpha plane from elsewhere you don't even need to zip into RGBA.

[rcunning]

@dalecurtis sorry to bother you again, but I am back on this task and having an issue with the data coming back from copyTo. It appears to be YUV420p by its size, and its y values seem correct. For a 640x360 frame, layout returned shows [{ "offset": 0, "stride": 640 }, { "offset": 230400, "stride": 640 }] with a total size of 345600 -- I would expect the layout to have 3 elements showing u and v as separate offsets to match YUV420p. In any case, I am having trouble converting these data to RGBA because the u and v appear to be not laid out the way I would expect. e.g. a simple rgba JS conversion would pull these out like:

for (let i = 0; i < width * height; i++) {
    const y = yuv[i];
    const u = yuv[(i >> 2) + (width * height)];
    const v = yuv[(i >> 2) + (width * height * 5) / 4];
...

So my images are coming out with b/w looking correct, but the color incorrect. Is there something I am missing here?

e.g.

[dalecurtis]

Given that you only have two returned layouts, it seems like it's NV12 instead of I420?

[rcunning]

Given that you only have two returned layouts, it seems like it's NV12 instead of I420?

omg, yes. Can't believe I did not notice that. Thank you again.

[rcunning]

The issue is closed, but I thought @dalecurtis might want to see the webcodecs-powered alpha video feature which finally was enabled for general use in Scenery this week:

https://twitter.com/rcunning/status/1679993435003015168

[orange4glace]

@rcunning Could you explain how did you make h265 alpha channel video? I tried with FFMPEG + libx265 but it failed.

[rcunning]

@rcunning Could you explain how did you make h265 alpha channel video? I tried with FFMPEG + libx265 but it failed.

I did not use h265! I created an .mp4 with two h264 video tracks:

• yuv420p for the colors
• yuv420p for the alpha channel (so looks like a gray-scale video)

Then at decode+render time I combined each paired VideoFrame rendered textures to get a single RGBA texture for display.

[orange4glace]

I did not use h265! I created an .mp4 with two h264 video tracks:

Thanks for your reply @rcunning !

So basically you have two VideoDecoder, one for colors and another for alpha channel?

[rcunning]

So basically you have two VideoDecoder, one for colors and another for alpha channel?

Yep.