What about simulcast?

[alvestrand]

If the sender is a simulcast sender, what should be the behavior of the streams?
One per RTP stream, or one for the whole sender? If the latter, where do we know what encoding the frame belongs to?

[murillo128]

I think it would be better to return an Array of streams with a 1-to-1 match to the sendEncodings of the RTPSender

[fippo]

well, its buggy right now

I think we need a way to tell apart different cryptographic contexts, similar to srtp which uses the ssrc. We might use the rid or mid instead.
Whether to put them into an array or use the additionalData... no preference.

[murillo128]

the solution should be aligned to what web codec does for simulcast support:

w3c/webcodecs#44

[guidou]

FWIW, we're currently exposing the SSRC in frames.
See https://cs.chromium.org/chromium/src/third_party/blink/renderer/modules/peerconnection/rtc_encoded_video_frame.idl?type=cs&q=rtc_encoded_video_frame.idl&sq=package:chromium&g=0&l=23

The field might get moved to the metadata field once we get rid of additionalData.

[alvestrand]

The current model seems to be that on the sender, all frames of all streams get exposed to the same transform, and the metadata needs to be very clear about which simulcast stream they belong to.

I think that's an OK model (but does emphasize the importance of correctly defined metadata).

[guidou]

We have adopted the model of exposing the ssrc in the metadata so the layer can be known when processing.
So far, it seems to have worked well.