Complex coding modes (simulcast, SVC, Opus FEC, etc.)

[steveanton]

This is a tracking issue for designing a solution for complex coding modes supported by various audio/video codecs like simulcast, SVC, and FEC.

[pthatcherg]

In particular, we need good SVC support and Bernard mentioned that the "dependsOn" currently in https://github.com/pthatcherg/web-codecs/blob/master/webidl.txt isn't sufficient.

[markafoltz]

Are these coding modes a stable part of the codec specifications (i.e. unlikely to change in the future) or something that applications are able to customize using lower level encoders?

[pthatcherg]

I believe they are stable (OPUS and VP9 certainly are), but it sounds like AV1 is very flexible so we need to figure out what modes people would want to use and which we can support.

[aboba]

At this point, the AV1 bitstream specification (including the list of scalability modes in Section 6.7.5) should be considered stable. The AV1 RTP payload specification is still work-in-progress.

K-SVC modes (which are a hybrid of SVC and simulcast) may not be expressable via "depends on" semantics. The L4T5_KEY_SHIFT mode is shown below:

[aboba]

The problem of SVC configuration has been addressed via the scalabilityMode concept defined in WebRTC-SVC and metadata has been added to identify the layer a frame corresponds to. Together, these additions allow the application to change the scalabilityMode, or to stop or re-start sending a layer. At this point, the only major feature not addressed is to allow target bitrates to be configured for each layer, as suggested by @Orphis in w3c/webrtc-svc#14 .

[aboba]

#187 addresses SVC. Simulcast currently is handled by creating separate encoders. In principle things like Opus FEC can be handled via codec configuration. So we seem to have addressed most of this, with the exception of things like FEC or RED (which can be handled in WASM)?

[chcunningham]

POR:

• SVC will be configurable (see Support for SVC scalability modes #40)
• FEC will either be done by the app, or where codec-specific options exist, we will create code-specific config knobs
• Simulcast: out of scope, just spin up multiple encoders (or better yet, use SVC)

[chcunningham]

I don't think there's anything actionable remaining in this issue. SVC is tracked in #40. Simulcast in #44. And we've determined FEC is out of scope. Happy to have new issues filed for codec specific FEC features where demanded. Closing for now. Please reopen if I've overlooked anything.

[murillo128]

Shouldn't we allow to configure opus in-band fec?

https://www.opus-codec.org/docs/html_api/group__encoderctls.html#ga5b67dc832aa46c1c2f35752c46380545

[chcunningham]

That SGTM. I intended to leave a path for that with this comment

Happy to have new issues filed for codec specific FEC features where demanded.

I've filed #244