Revert "Add screen content coding tools as an encoder configuration"

[Djuffin]

av1.forceScreenContentTools was made practically obsolete by contentHint, which does the same but in a codec-agnostic way.

It hasn't been implemented by any UA, and it's safe to remove from the spec.

This reverts commits:
b35c6a3
29f1a8a

Related issue: #646

[Djuffin]

cc @fippo

[aboba]

The reason for including per-codec SCC configuration settings was to be able to detect if they were supported. Our data shows that only a fraction of AV1 hardware acceleration implementations support screen content coding.

Using contentHint, is there a way to determine whether the desired hint can be applied? For example, if a config contains hwAcceleration="prefer-hardware" and contentHint: "text", and an attempt to configure a tool (e.g. palette or block copy) fails, does anything returned by isConfigSupported indicate an inability to apply the hint? For example, in the returned configuration, contentHint: "text" could be omitted if the hint couldn't be applied.

[fippo]

You beat me to it and reverts are better ;-)

[padenot]

@aboba's point is valid though. Quoting the spec:

The User Agent MUST NOT refuse the configuration if it doesn’t support this content hint. See isConfigSupported().

[Djuffin]

Yes, I agree that, apart from inspecting the bitstream, developers won't have a way to check if screen content coding is actually in effect. But I don't think that it's going to be a key factor in their choice of the encoding configuration.

In other words, it's unlikely that someone will say "well, if I can't have SCC on AV1 I'll rather use VP9 or H.264".

[aboba]

Since software implementations of AV1 encode/decode support SCC, the problem arises in situations where a (non-compliant) AV1 hardware encoder or decoder lacks SCC support. In the case of a hardware encoder lacking SCC support, the result will be sending at a higher bitrate, but probably still lower than other codecs, so that there's no reason to switch codecs.

However, if a hardware decoder doesn't support SCC, the result could be a decode failure. In that situation, it would make sense to turn off SCC on the encoder. But that requires knowledge of whether the (hw) encoder is encoding with SCC and also an indication that the decoder error was due to lack of SCC support.

[fippo]

Such a fallback scenario might be handled by the decoder retrying the same frame with software though rather than making changes on the encoding side? (and such a fallback is measurable which ultimatively should lead to the bug being identified and fixed)

[taste1981]

For AV1 decoder SCC must be supported as part of conformance (unlike H.265 which contains SCC support in the form of a profile), so it is not likely for decoder to raise such "error".

There are platforms that supports AV1 encoding, but does not allow you to turn on screen content tools. I think the knob here still makes sense to me.

[Djuffin]

Given that we already expose SCC via contentHint, I'm not convinced we really need an explicit knob here. @aboba if I remember correctly you wanted to check how many decoders fail to decode content using SCC. Do you still think that we shouldn't use contentHint for it? BTW it's already shipped in Chromium

[aboba]

@Djuffin SCC support is a conformance requirement for AV1 decoders, and so far we have not encountered AV1 decoders that do not support SCC. WebCodecs is supported in Safari Tech Preview, and isConfigSupported() indicates support for AV1 hardware accelerated decode in newer devices such as the iPhone Pro 15 as well as M3-based MacBooks. Support for AV1 hardware accelerated encode is not available on iOS or Android mobile devices (or notebooks, for that matter) so I can't speak to the prevalence of SCC support on AV1 hardware encoders. However, SCC is supported in AV1 software encoders across multiple browsers (Chrome, Edge, Safari TP, Firefox).

[aboba]

The question is whether the application can benefit from knowing that SCC encoding isn't supported (e.g. if a hardware encoder does not support SCC). If the application would switch to a software encoder to use SCC, then there is a potential benefit. But if hardware acceleration is required (e.g. for power management reasons) then the application will need to optimize along the lines of what a ContentHint would provide.