Coding Tool configuration

[aboba]

In codecs such as AV1 there are screen content coding tools that can offer very effective compression for appropriate content.

The question is how these tools can be configured to be used. Screen content coding cannot be on by default since it isn't generally useful. Dynamically recognizing when screen content coding would be helpful seems like it would be complex.

The Content-Hints specification is one way to activate screen content coding (or other encoding optimizations), but the hint is currently not passed along when a MediaStreamTrack is used as input to create a stream of VideoFrames.

Another possibility is to add an attribute to configuration.

[sandersdan]

This sounds like something I would expect to be a codec-specific configuration parameter.

We also do not currently have a way to request 4:4:4 and RGB encoding, which applies more generally to other codecs.

[chcunningham]

Another possibility is to add an attribute to configuration.

Editors call: We favor this option for its simplicity.

I grabbed this screenshot of AV1 coding tools from @marco99zz's RTC@Scale talk. @aboba are these what you had in mind? @marco99zz, can you help me prioritize this list in terms of what should be added to WebCodecs.

[aboba]

We have added support for common encoding controls used with screen capture.

PR 662 (merged) turns on Palette mode & Intra BC tools for AV1.
PR 659 (merged) enables per-frame QP for AV1.

Being able to control other coding tools would also be useful.

[marco99zz]

On Wed, Apr 6, 2022 at 12:43 PM chcunningham ***@***.***> wrote: Another possibility is to add an attribute to configuration. Editors call: We favor this option for its simplicity. I grabbed this screenshot of AV1 coding tools from @marco99zz <https://github.com/marco99zz>'s ***@***.*** talk. @aboba <https://github.com/aboba> are these what you had in mind? @marco99zz <https://github.com/marco99zz>, can you help me prioritize this list in terms of what should be added to WebCodecs.

Sure I can help you prioritize the list for WebCodecs.

…

[image: unnamed] <https://user-images.githubusercontent.com/12039355/162057176-57173bfa-146b-4bcb-9a0d-9abd90b3d00c.png> — Reply to this email directly, view it on GitHub <#478 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACDDKFCCARKFBUQWJ4WRDODVDXSPNANCNFSM5SFYLLEA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Djuffin]

We went with per-codec configuration (see #646 )
But I'm thinking that we might want a generic content type hint in the VideoEncoderConfig.
At least OpenH264 and libvpx support such hints.

[aboba]

Within the Screen Content Community Group (SCCG) we had a discussion of what libwebrtc does to optimize screen sharing. It varies quite a bit depending on the codec, but doesn't take Content Hints into account. For video, content hints include "motion", "detail" and "text".

For example, if the track came from a screen share, SCREEN_CONTENT_REAL_TIME is set in Open H.264.

This kind of optimization is implementation-specific, and not defined in the codec specification, so it's not something that we can add to a WebCodecs Codec registry. Would it make sense for WebCodecs to consume "Content Hints" to make this kind of adjustment? Note that this may also be an issue for audio, where "speech" and "music" content-hints are defined.

Maybe this is a subject that we should discuss in the MEDIA WG?

[chrisn]

Discussion from Media WG meeting on 13 Jun 2023: minutes and slides.