-spec: WEBCODECS; urlPrefix: https://w3c.github.io/webcodecs/# - type: attribute - text: AudioDecoderConfig.description; url: dom-audiodecoderconfig-description - text: AudioDecoderConfig.sampleRate; url: dom-audiodecoderconfig-samplerate - text: AudioDecoderConfig.numberOfChannels; url: dom-audiodecoderconfig-numberofchannels - type: dfn - for: EncodedAudioChunkType; text: key; url: dom-encodedaudiochunktype-key - for: EncodedAudioChunk; text: [[internal data]]; url: dom-encodedaudiochunk-internal-data-slot - for: EncodedAudioChunk; text: [[type]]; url: dom-encodedaudiochunk-type-slot - type: interface - text: EncodedAudioChunk; url: encodedaudiochunk - type: dictionary - text: AudioDecoderConfig; url: dictdef-audiodecoderconfig - text: AudioEncoderConfig; url: dictdef-audioencoderconfig --
{
"iso14496-3": {
@@ -76,32 +59,32 @@
================================================
If the bitstream is in {{AacBitstreamFormat/adts}} format, the
-[=EncodedAudioChunk/[[internal data]]=] of {{EncodedAudioChunk}}s are expected
+{{EncodedAudioChunk/[[internal data]]}} of {{EncodedAudioChunk}}s are expected
to be an ADTS frame, as described in section 1.A.3.2 of [[iso14496-3]].
If the bitstream is in {{AacBitstreamFormat/aac}} format, the
-[=EncodedAudioChunk/[[internal data]]=] of {{EncodedAudioChunk}}s are expected
+{{EncodedAudioChunk/[[internal data]]}} of {{EncodedAudioChunk}}s are expected
to be a raw AAC frame (syntax element `raw_data_block()`), as described in
section 4.4.2.1 of [[iso14496-3]].
AudioDecoderConfig description {#audiodecoderconfig-description}
================================================================
-If {{AudioDecoderConfig.description}} is present, it is assumed to a
+If {{AudioDecoderConfig/description}} is present, it is assumed to a
`AudioSpecificConfig` as defined in [[iso14496-3]] section 1.6.2.1, Table 1.15,
and the bitstream is assumed to be in {{AacBitstreamFormat/aac}}.
-If the {{AudioDecoderConfig.description}} is not present, the bitstream is
+If the {{AudioDecoderConfig/description}} is not present, the bitstream is
assumed to be in {{AacBitstreamFormat/adts}} format.
-The {{AudioDecoderConfig.sampleRate}} and {{AudioDecoderConfig.numberOfChannels}}
+The {{AudioDecoderConfig/sampleRate}} and {{AudioDecoderConfig/numberOfChannels}}
members are ignored.
EncodedAudioChunk type {#encodedaudiochunk-type}
================================================
-The [=EncodedAudioChunk/[[type]]=] for an {{EncodedAudioChunk}} containing
-AAC is always "[=EncodedAudioChunkType/key=]".
+The {{EncodedAudioChunk/[[type]]}} for an {{EncodedAudioChunk}} containing
+AAC is always "{{EncodedAudioChunkType/key}}".
NOTE: Once the initialization has succeeded, any AAC packet can be decoded at
any time without error, but this might not result in the expected audio
@@ -161,17 +144,22 @@
-spec: WEBCODECS; urlPrefix: https://w3c.github.io/webcodecs/# - type: attribute - text: AudioDecoderConfig.description; url: dom-audiodecoderconfig-description - type: dfn - for: EncodedAudioChunkType; text: key; url: dom-encodedaudiochunktype-key - for: EncodedAudioChunk; text: [[internal data]]; url: dom-encodedaudiochunk-internal-data-slot - for: EncodedAudioChunk; text: [[type]]; url: dom-encodedaudiochunk-type-slot - type: interface - text: EncodedAudioChunk; url: encodedaudiochunk - type: dictionary - text: AudioDecoderConfig; url: dictdef-audiodecoderconfig - text: AudioEncoderConfig; url: dictdef-audioencoderconfig --
{
"ITU-G.711": {
@@ -66,25 +51,30 @@
EncodedAudioChunk data {#encodedaudiochunk-data}
================================================
-{{EncodedAudioChunk}} [=EncodedAudioChunk/[[internal data]]=] is expected to be
+{{EncodedAudioChunk}} {{EncodedAudioChunk/[[internal data]]}} is expected to be
a sequence of bytes of arbitrary length, where each byte is an A-law
encoded PCM sample as defined by Tables 1a and 1b in [[ITU-G.711]].
AudioDecoderConfig description {#audiodecoderconfig-description}
================================================================
-An {{AudioDecoderConfig.description}} is expected to be omitted from the
+The {{AudioDecoderConfig/description}} is expected to be omitted from the
{{AudioDecoderConfig}}.
EncodedAudioChunk type {#encodedaudiochunk-type}
================================================
-The [=EncodedAudioChunk/[[type]]=] for an {{EncodedAudioChunk}} containing
-A-law PCM is always "[=EncodedAudioChunkType/key=]".
+The {{EncodedAudioChunk/[[type]]}} for an {{EncodedAudioChunk}} containing
+A-law PCM is always "{{EncodedAudioChunkType/key}}".
+
+Privacy Considerations {#privacy-considerations}
+==========================================================================
+
+Please refer to the section [[WEBCODECS#privacy-considerations|Privacy
+Considerations]] in [[WEBCODECS]].
-Privacy and Security Considerations {#privacy-and-security-considerations}
+Security Considerations {#security-considerations}
==========================================================================
-Please refer to the [[WEBCODECS#privacy-considerations|Privacy Considerations]]
-and [[WEBCODECS#security-considerations|Security Considerations]] sections in
-[[WEBCODECS]].
+Please refer to the section [[WEBCODECS#security-considerations|Security
+Considerations]] in [[WEBCODECS]].
diff --git a/av1_codec_registration.src.html b/av1_codec_registration.src.html
index 83b0299a..7b3be04f 100644
--- a/av1_codec_registration.src.html
+++ b/av1_codec_registration.src.html
@@ -7,16 +7,18 @@
Group: mediawg
ED: https://w3c.github.io/webcodecs/av1_codec_registration.html
TR: https://www.w3.org/TR/webcodecs-av1-codec-registration/
-Editor: Chris Cunningham, w3cid 114832, Google Inc. https://www.google.com/
Editor: Paul Adenot, w3cid 62410, Mozilla https://www.mozilla.org/
Editor: Bernard Aboba, w3cid 65611, Microsoft Corporation https://www.microsoft.com/
+Former Editor: Chris Cunningham, w3cid 114832, Google Inc. https://www.google.com/
Abstract: This registration is entered into the [[webcodecs-codec-registry]].
It describes, for AV1, the (1) fully qualified codec strings,
(2) the codec-specific {{EncodedVideoChunk}}
- [=EncodedVideoChunk/[[internal data]]=] bytes, (3) the
- {{VideoDecoderConfig.description}} bytes, and (4) the values of
- {{EncodedVideoChunk}} [=EncodedVideoChunk/[[type]]=].
+ {{EncodedVideoChunk/[[internal data]]}} bytes, (3) the codec-specific
+ extensions to {{VideoEncoderConfig}}, (4) the
+ {{VideoDecoderConfig/description|VideoDecoderConfig.description}} bytes,
+ (5) the values of {{EncodedVideoChunk}} {{EncodedVideoChunk/[[type]]}}, and
+ (6) the codec-specific extensions to {{VideoEncoderEncodeOptions}}.
The registration is not intended to include any information on whether a
codec format is encumbered by intellectual property claims. Implementers and
@@ -32,24 +34,6 @@
!Version History: https://github.com/w3c/webcodecs/commits
--spec: WEBCODECS; urlPrefix: https://w3c.github.io/webcodecs/# - type: attribute - text: EncodedVideoChunkMetadata.decoderConfig; url: dom-encodedvideochunkmetadata-decoderconfig - for: EncodedVideoChunkType; text: key; url: dom-encodedvideochunktype-key - text: VideoDecoderConfig.description; url: dom-videodecoderconfig-description - type: dfn - for: EncodedVideoChunk; text: [[internal data]]; url: dom-encodedvideochunk-internal-data-slot - for: EncodedVideoChunk; text: [[type]]; url: dom-encodedvideochunk-type-slot - for: VideoEncoder; text: [[output callback]]; url: dom-videoencoder-output-callback-slot - type: interface - text: EncodedVideoChunk; url: encodedvideochunk - text: VideoEncoder; url: videoencoder - type: dictionary - text: VideoEncoderConfig; url: dictdef-videoencoderconfig - text: VideoDecoderConfig; url: dictdef-videodecoderconfig --
{
"AV1": {
@@ -74,26 +58,108 @@
EncodedVideoChunk data {#encodedvideochunk-data}
================================================
-{{EncodedVideoChunk}} [=EncodedVideoChunk/[[internal data]]=] is expected to be
+{{EncodedVideoChunk}} {{EncodedVideoChunk/[[internal data]]}} is expected to be
data compliant to the "low-overhead bitstream format" as described in Section 5
of [[AV1]].
VideoDecoderConfig description {#videodecoderconfig-description}
================================================================
-{{VideoDecoderConfig.description}} is not used for this codec.
+{{VideoDecoderConfig/description}} is not used for this codec.
EncodedVideoChunk type {#encodedvideochunk-type}
================================================
-If an {{EncodedVideoChunk}}'s [=EncodedVideoChunk/[[type]]=] is
+If an {{EncodedVideoChunk}}'s {{EncodedVideoChunk/[[type]]}} is
{{EncodedVideoChunkType/key}}, then the {{EncodedVideoChunk}} is expected to
contain a frame with a `frame_type` of `KEY_FRAME` as defined in Section
6.8.2 of [[AV1]].
-Privacy and Security Considerations {#privacy-and-security-considerations}
+VideoEncoderConfig extensions {#videoencoderconfig-extensions}
+==============================================================
+
+
+
+partial dictionary VideoEncoderConfig {
+ AV1EncoderConfig av1;
+};
+
+
+
+++ ++dictionary AV1EncoderConfig { + boolean forceScreenContentTools = false; +}; + +
++ ++partial dictionary VideoEncoderEncodeOptions { + VideoEncoderEncodeOptionsForAv1 av1; +}; + +
++ ++dictionary VideoEncoderEncodeOptionsForAv1 { + unsigned short? quantizer; +}; + +
-spec: WEBCODECS; urlPrefix: https://w3c.github.io/webcodecs/# - type: attribute - text: EncodedVideoChunkMetadata.decoderConfig; url: dom-encodedvideochunkmetadata-decoderconfig - for: EncodedVideoChunkType; text: key; url: dom-encodedvideochunktype-key - text: VideoDecoderConfig.description; url: dom-videodecoderconfig-description - type: dfn - for: EncodedVideoChunk; text: [[internal data]]; url: dom-encodedvideochunk-internal-data-slot - for: EncodedVideoChunk; text: [[type]]; url: dom-encodedvideochunk-type-slot - for: VideoEncoder; text: [[output callback]]; url: dom-videoencoder-output-callback-slot - type: interface - text: EncodedVideoChunk; url: encodedvideochunk - text: VideoEncoder; url: videoencoder - type: dictionary - text: VideoEncoderConfig; url: dictdef-videoencoderconfig - text: VideoDecoderConfig; url: dictdef-videodecoderconfig --
{
"ITU-T-REC-H.264": {
@@ -78,26 +61,26 @@
EncodedVideoChunk data {#encodedvideochunk-data}
================================================
-{{EncodedVideoChunk}} [=EncodedVideoChunk/[[internal data]]=] is expected to be
+{{EncodedVideoChunk}} {{EncodedVideoChunk/[[internal data]]}} is expected to be
an access unit as defined in [[ITU-T-REC-H.264]] section 7.4.1.2.
NOTE: An access unit contains exactly one primary coded picture.
If the bitstream is in {{AvcBitstreamFormat/avc}} format,
-[=EncodedVideoChunk/[[internal data]]=] is assumed to be in canonical format, as
+{{EncodedVideoChunk/[[internal data]]}} is assumed to be in canonical format, as
defined in [[iso14496-15]] section 5.3.2.
If the bitstream is in {{AvcBitstreamFormat/annexb}} format,
-[=EncodedVideoChunk/[[internal data]]=] is assumed to be in in Annex B format,
+{{EncodedVideoChunk/[[internal data]]}} is assumed to be in in Annex B format,
as defined in [[ITU-T-REC-H.264]] Annex B.
-NOTE: Since [=EncodedVideoChunk/[[internal data]]=] is inherently byte-aligned,
+NOTE: Since {{EncodedVideoChunk/[[internal data]]}} is inherently byte-aligned,
implementations are not required to recover byte-alignment.
VideoDecoderConfig description {#videodecoderconfig-description}
================================================================
-If the {{VideoDecoderConfig.description}} is present, it is assumed to be an
+If the {{VideoDecoderConfig/description}} is present, it is assumed to be an
`AVCDecoderConfigurationRecord`, as defined by [[iso14496-15]], section
5.3.3.1, and the bitstream is assumed to be in {{AvcBitstreamFormat/avc}}
format.
@@ -105,7 +88,7 @@
NOTE: This format is commonly used in .MP4 files, where the player generally
has random access to the media data.
-If the {{VideoDecoderConfig.description}} is not present, the bitstream is
+If the {{VideoDecoderConfig/description}} is not present, the bitstream is
assumed to be in {{AvcBitstreamFormat/annexb}} format.
NOTE: "annexb" format is described in greater detail by [[ITU-T-REC-H.264]],
@@ -116,16 +99,17 @@
EncodedVideoChunk type {#encodedvideochunk-type}
================================================
-If an {{EncodedVideoChunk}}'s [=EncodedVideoChunk/[[type]]=] is
+If an {{EncodedVideoChunk}}'s {{EncodedVideoChunk/[[type]]}} is
{{EncodedVideoChunkType/key}}, and the bitstream is in
{{AvcBitstreamFormat/avc}} format, then the {{EncodedVideoChunk}} is expected to
contain a primary coded picture that is an instantaneous decoding refresh (IDR)
picture.
NOTE: If the bitstream is in {{AvcBitstreamFormat/avc}} format, parameter sets
- necessary for decoding are included in {{VideoDecoderConfig.description}}.
+ necessary for decoding are included in
+ {{VideoDecoderConfig/description|VideoDecoderConfig.description}}.
-If an {{EncodedVideoChunk}}'s [=EncodedVideoChunk/[[type]]=] is
+If an {{EncodedVideoChunk}}'s {{EncodedVideoChunk/[[type]]}} is
{{EncodedVideoChunkType/key}}, and the bitstream is in
{{AvcBitstreamFormat/annexb}} format, then the {{EncodedVideoChunk}} is expected
to contain both a primary coded picture that is an instantaneous decoding
@@ -179,12 +163,8 @@
-The {{AvcBitstreamFormat}} determines the location of AVC Sequence Parameter
-Set (SPS) and Picture Parameter Set (PPS) data, and mechanisms for packaging
-the bitstream.
-
-SPS and PPS are described in greater detail in sections G.3.41 and G.3.55 of
-[[ITU-T-REC-H.264]].
+The {{AvcBitstreamFormat}} determines the location of AVC parameter sets, and
+mechanisms for packaging the bitstream.
++ ++partial dictionary VideoEncoderEncodeOptions { + VideoEncoderEncodeOptionsForAvc avc; +}; + +
++ ++dictionary VideoEncoderEncodeOptionsForAvc { + unsigned short? quantizer; +}; + +
-spec: WEBCODECS; urlPrefix: https://w3c.github.io/webcodecs/# - type: attribute - text: AudioDecoderConfig.description; url: dom-audiodecoderconfig-description - text: VideoDecoderConfig.description; url: dom-videodecoderconfig-description - type: dfn - for: EncodedVideoChunk; text: [[type]]; url: dom-encodedaudiochunk-type-slot - type: interface - text: EncodedAudioChunk; url: encodedaudiochunk - text: EncodedVideoChunk; url: encodedvideochunk - type: dictionary - text: AudioDecoderConfig; url: dictdef-audiodecoderconfig - text: AudioEncoderConfig; url: dictdef-audioencoderconfig - text: VideoDecoderConfig; url: dictdef-videodecoderconfig - text: VideoEncoderConfig; url: dictdef-videoencoderconfig -- Organization {#organization} ============================ @@ -78,17 +61,25 @@ 2. {{EncodedAudioChunk}} or {{EncodedVideoChunk}} internal data 3. {{AudioDecoderConfig}} or {{VideoDecoderConfig}} description bytes 4. Expectations for {{EncodedAudioChunk}} or {{EncodedVideoChunk}} - [=EncodedVideoChunk/[[type]]=] -3. Where applicable, a registration specification may include a section + {{EncodedVideoChunk/[[type]]}} +4. Where applicable, a registration specification may include a section describing extensions to {{VideoEncoderConfig}} or {{AudioEncoderConfig}} dictionaries. -4. Candidate entries must be announced by filing an issue in the +5. Candidate entries must be announced by filing an issue in the [WebCodecs GitHub issue tracker](https://github.com/w3c/webcodecs/issues/) so they can be discussed and evaluated for compliance before being added to - the registry. If registry editors reach consensus to accept the candidate, - a pull request should be drafted (either by editors or by the party - requesting the candidate registration) to register the candidate. The - registry editors will review and merge the pull request. + the registry. The Media Working Group may seek expertise from outside the + Working Group as part of its evaluation, e.g., from the codec specification + editors or relevant standards group. If the Working Group reaches consensus + to accept the candidate, a pull request should be drafted (either by editors + or by the party requesting the candidate registration) to register the + candidate. The registry editors will review and merge the pull request. +6. Existing entries cannot be deleted or deprecated. They may be changed after + being published through the same process as candidate entries. Possible + changes include expansion of the codec string to better qualify the codec, + adjustments to the codec name string, and modification of the link to the + public specification. + Audio Codec Registry {#audio-codec-registry} ============================================ @@ -112,7 +103,7 @@ [[WEBCODECS-MP3-CODEC-REGISTRATION]]
-spec: WEBCODECS; urlPrefix: https://w3c.github.io/webcodecs/# - type: attribute - text: AudioDecoderConfig.description; url: dom-audiodecoderconfig-description - text: AudioDecoderConfig.sampleRate; url: dom-audiodecoderconfig-samplerate - text: AudioDecoderConfig.numberOfChannels; url: dom-audiodecoderconfig-numberofchannels - type: dfn - for: EncodedAudioChunkType; text: key; url: dom-encodedaudiochunktype-key - for: EncodedAudioChunk; text: [[internal data]]; url: dom-encodedaudiochunk-internal-data-slot - for: EncodedAudioChunk; text: [[type]]; url: dom-encodedaudiochunk-type-slot - type: interface - text: EncodedAudioChunk; url: encodedaudiochunk - type: dictionary - text: AudioDecoderConfig; url: dictdef-audiodecoderconfig --
{
"FLAC": {
- "href": "https://xiph.org/flac/format.html",
- "title": "FLAC - format",
- "publisher": "Xiph.Org Foundation"
+ "href": "https://datatracker.ietf.org/doc/draft-ietf-cellar-flac",
+ "title": "Free Lossless Audio Codec",
+ "publisher": "IETF"
}
}
@@ -66,21 +51,21 @@
EncodedAudioChunk data {#encodedaudiochunk-data}
================================================
-{{EncodedAudioChunk}} [=EncodedAudioChunk/[[internal data]]=] is expected to be
+{{EncodedAudioChunk}} {{EncodedAudioChunk/[[internal data]]}} is expected to be
a "FRAME" as described in [[FLAC]].
AudioDecoderConfig description {#audiodecoderconfig-description}
================================================================
-{{AudioDecoderConfig.description}} is required, and has to be the following:
+{{AudioDecoderConfig/description}} is required, and has to be the following:
- The bytes `0x66 0x4C 0x61 0x43` ("`fLaC`" in ASCII)
-- A `METADATA_BLOCK` of type `STEAMINFO` as described in [[FLAC]]
-- Optionaly other `METADATA_BLOCK`, that are not used by the specification
+- A `metadata block` (called the STREAMINFO block) as described in section 7 of [[FLAC]]
+- Optionaly other metadata blocks, that are not used by the specification
-The {{AudioDecoderConfig.sampleRate}} and {{AudioDecoderConfig.numberOfChannels}}
+The {{AudioDecoderConfig/sampleRate}} and {{AudioDecoderConfig/numberOfChannels}}
members are overridden by what the decoder finds in the
-{{AudioDecoderConfig.description}}.
+{{AudioDecoderConfig/description}}.
NOTE: This corresponds to the beginning of a FLAC bitstream, before the audio
frames.
@@ -88,15 +73,73 @@
EncodedAudioChunk type {#encodedaudiochunk-type}
================================================
-The [=EncodedAudioChunk/[[type]]=] for an {{EncodedAudioChunk}} containing
-FLAC is always "[=EncodedAudioChunkType/key=]".
+The {{EncodedAudioChunk/[[type]]}} for an {{EncodedAudioChunk}} containing
+FLAC is always "{{EncodedAudioChunkType/key}}".
NOTE: Once the initialization has succeeded, any FLAC packet can be decoded at
any time without error, but this might not result in the expected audio output.
-Privacy and Security Considerations {#privacy-and-security-considerations}
+AudioEncoderConfig extensions {#audioencoderconfig-extensions}
+=============================================================
+
+++ ++partial dictionary AudioEncoderConfig { + FlacEncoderConfig flac; +}; + +
++ +To check if an {{FlacEncoderConfig}} is valid, run these steps: +1. If {{FlacEncoderConfig/blockSize}} is not a valid block size, + which is described section 5.1 of [[FLAC]], return `false`. +1. If {{FlacEncoderConfig/compressLevel}} is specified and not within the range of + `0` (fastest, least compression) and `8` (slowest, most compression) inclusively, return `false`. +2. Return `true`. + ++dictionary FlacEncoderConfig { + [EnforceRange] unsigned long blockSize = 0; + [EnforceRange] unsigned long compressLevel = 5; +}; + +
+Title: HEVC (H.265) WebCodecs Registration
+Repository: w3c/webcodecs
+Status: NOTE-ED
+Shortname: webcodecs-hevc-codec-registration
+Level: none
+Group: mediawg
+ED: https://w3c.github.io/webcodecs/hevc_codec_registration.html
+TR: https://www.w3.org/TR/webcodecs-hevc-codec-registration/
+Editor: Paul Adenot, w3cid 62410, Mozilla https://www.mozilla.org/
+Editor: Bernard Aboba, w3cid 65611, Microsoft Corporation https://www.microsoft.com/
+Former Editor: Chris Cunningham, w3cid 114832, Google Inc. https://www.google.com/
+
+Abstract: This registration is entered into the [[webcodecs-codec-registry]].
+ It describes, for HEVC (H.265), the (1) fully qualified codec strings,
+ (2) the codec-specific {{EncodedVideoChunk}}
+ {{EncodedVideoChunk/[[internal data]]}} bytes, (3) the
+ {{VideoDecoderConfig/description|VideoDecoderConfig.description}} bytes,
+ (4) the values of {{EncodedVideoChunk}} {{EncodedVideoChunk/[[type]]}}, and
+ (5) the codec-specific extensions to {{VideoEncoderConfig}}
+
+ The registration is not intended to include any information on whether a
+ codec format is encumbered by intellectual property claims. Implementers and
+ authors are advised to seek appropriate legal counsel in this matter if they
+ intend to implement or use a specific codec format. Implementers of
+ WebCodecs are not required to support the HEVC / H.265 codec.
+
+ This registration is non-normative.
+
+Markup Shorthands:css no, markdown yes, dfn yes
+!Participate: Git Repository.
+!Participate: File an issue.
+!Version History: https://github.com/w3c/webcodecs/commits
+
+
+
+
+{
+ "ITU-T-REC-H.265": {
+ "href": "https://www.itu.int/rec/T-REC-H.265",
+ "title": "H.265 : High efficiency video coding",
+ "publisher": "ITU",
+ "date": "August 2021"
+ },
+ "iso14496-15": {
+ "href": "https://www.iso.org/standard/74429.html",
+ "title": "ISO/IEC 14496-15:2019 Information technology — Coding of audio-visual objects — Part 15: Carriage of network abstraction layer (NAL) unit structured video in the ISO base media file format",
+ "publisher": "ISO",
+ "date": "September 2019"
+ }
+}
+
+
+Fully qualified codec strings {#fully-qualified-codec-strings}
+==============================================================
+
+The codec string begins with the prefix "hev1." or "hvc1.", with a suffix of
+four dot-separated fields as described in Section E.3 of [[iso14496-15]].
+
+EncodedVideoChunk data {#encodedvideochunk-data}
+================================================
+
+{{EncodedVideoChunk}} {{EncodedVideoChunk/[[internal data]]}} is expected to be
+an access unit as defined in [[ITU-T-REC-H.265]] section 7.4.2.4.
+
+NOTE: An access unit contains exactly one base layer coded picture.
+
+If the bitstream is in {{HevcBitstreamFormat/hevc}} format,
+{{EncodedVideoChunk/[[internal data]]}} is assumed to be in canonical format, as
+defined in [[iso14496-15]] section 8.3.2.
+
+If the bitstream is in {{HevcBitstreamFormat/annexb}} format,
+{{EncodedVideoChunk/[[internal data]]}} is assumed to be in Annex B format,
+as defined in [[ITU-T-REC-H.265]] Annex B.
+
+NOTE: Since {{EncodedVideoChunk/[[internal data]]}} is inherently byte-aligned,
+ implementations are not required to recover byte-alignment.
+
+VideoDecoderConfig description {#videodecoderconfig-description}
+================================================================
+
+If the {{VideoDecoderConfig/description}} is present, it is assumed to be an
+`HEVCDecoderConfigurationRecord`, as defined by [[iso14496-15]], section
+8.3.3.1, and the bitstream is assumed to be in {{HevcBitstreamFormat/hevc}}
+format.
+
+NOTE: This format is commonly used in .MP4 files, where the player generally
+ has random access to the media data.
+
+If the {{VideoDecoderConfig/description}} is not present, the bitstream is
+assumed to be in {{HevcBitstreamFormat/annexb}} format.
+
+NOTE: "annexb" format is described in greater detail by [[ITU-T-REC-H.265]],
+ Annex B. This format is commonly used in live-streaming applications, where
+ including the VPS, SPS, and PPS data periodically allows users to easily
+ start from the middle of the stream.
+
+EncodedVideoChunk type {#encodedvideochunk-type}
+================================================
+
+If an {{EncodedVideoChunk}}'s {{EncodedVideoChunk/[[type]]}} is
+{{EncodedVideoChunkType/key}}, and the bitstream is in
+{{HevcBitstreamFormat/hevc}} format, then the {{EncodedVideoChunk}} is expected
+to contain a base layer primary coded picture that is an instantaneous decoding
+refresh (IDR), clean random access (CRA), or broken link access (BLA) picture.
+
+NOTE: If the bitstream is in {{HevcBitstreamFormat/hevc}} format, parameter sets
+ necessary for decoding are included in
+ {{VideoDecoderConfig/description|VideoDecoderConfig.description}}.
+
+If an {{EncodedVideoChunk}}'s {{EncodedVideoChunk/[[type]]}} is
+{{EncodedVideoChunkType/key}}, and the bitstream is in
+{{HevcBitstreamFormat/annexb}} format, then the {{EncodedVideoChunk}} is
+expected to contain both a base layer coded picture that is an instantaneous
+decoding refresh (IDR), clean random access (CRA), or broken link access (BLA)
+picture, and all parameter sets necessary to decode all video data
+NAL units in the {{EncodedVideoChunk}}.
+
+VideoEncoderConfig extensions {#videoencoderconfig-extensions}
+==============================================================
+
+++ ++partial dictionary VideoEncoderConfig { + HevcEncoderConfig hevc; +}; + +
++ ++dictionary HevcEncoderConfig { + HevcBitstreamFormat format = "hevc"; +}; + +
++ +The {{HevcBitstreamFormat}} determines the location of HEVC parameter sets, and +mechanisms for packaging the bitstream. + ++enum HevcBitstreamFormat { + "annexb", + "hevc", +}; + +
-spec:webidl; type:interface; text:Promise spec:html; type:attribute; text:hidden +spec:infra; type:dfn; text:list +spec:infra; type:dfn; text:enqueue
-spec: media-source; urlPrefix: https://www.w3.org/TR/media-source/
- type: method
- for: MediaSource; text: isTypeSupported(); url: #dom-mediasource-istypesupported
-
spec: html; urlPrefix: https://html.spec.whatwg.org/multipage/;
- for: HTMLMediaElement;
- type: method; text: canPlayType(); url: #dom-navigator-canplaytype
- for: PlatformObject;
- type: attribute; text: [[Detached]]; url: structured-data.html#detached
for: ImageBitmap;
- type: attribute; text: resizeWidth; url:#dom-imagebitmapoptions-resizewidth
- type: attribute; text: resizeHeight; url:#dom-imagebitmapoptions-resizeheight
- type: dfn; text: cropped to the source rectangle with formatting; url: imagebitmap-and-animations.html#cropped-to-the-source-rectangle-with-formatting
type: dfn; text: bitmap data; url: imagebitmap-and-animations.html#concept-imagebitmap-bitmap-data
for: Canvas;
type: dfn; text: Check the usability of the image argument; url: canvas.html#check-the-usability-of-the-image-argument
- for: origin;
- type: dfn; text: origin; url: origin.html#concept-origin
+ type: dfn; text: is not origin-clean; url: canvas.html#the-image-argument-is-not-origin-clean
for: webappapis;
- type: dfn; text: global object; url: webappapis.html#global-object
type: dfn; text: entry settings object; url: webappapis.html#entry-settings-object
for: media;
type: dfn; text: current playback position; url: media.html#current-playback-position
type: dfn; text: live; url: infrastructure.html#live
-spec: mediacapture-streams; urlPrefix: https://www.w3.org/TR/mediacapture-streams/
- for: mediaDevices;
- type: method; text: getUserMedia(); url: #dom-mediadevices-getusermedia
-
-spec: mediacapture-screen-share; urlPrefix: https://w3c.github.io/mediacapture-screen-share/
- for: mediaDevices; type: method; text: getDisplayMedia(); url: #dom-mediadevices-getdisplaymedia
-
-spec: mediacapture-main; urlPrefix: https://w3c.github.io/mediacapture-main/
- for:MediaStreamTrackState;
- type: enum-value; text: live; url: #idl-def-MediaStreamTrackState.live
- type: enum-value; text: ended; url: #idl-def-MediaStreamTrackState.ended
-
-spec: mimesniff; urlPrefix: https://mimesniff.spec.whatwg.org/#
- type: dfn; text: MIME type; url: mime-type
- type: dfn; text: valid MIME type string; url:valid-mime-type
-
-spec: infra; urlPrefix: https://infra.spec.whatwg.org/#
- type: dfn; text: queue; url: queues
- type: dfn; text: enqueuing; url: queue-enqueue;
- type: dfn; text: dequeued; url: queue-dequeue;
- type: dfn; text: empty; url: list-is-empty;
- type: dfn; text: list; url: lists;
-
-spec: mediastream-recording; urlPrefix: https://www.w3.org/TR/mediastream-recording/#
- type: interface; text: MediaRecorder; url: mediarecorder
- type: interface; text: BitrateMode; url: bitratemode
- for: BitrateMode;
- type: enum-value; text: constant; url: dom-bitratemode-constant
- type: enum-value; text: variable; url: dom-bitratemode-variable
-
-spec: media-capabilities; urlPrefix: https://w3c.github.io/media-capabilities/#
- type: method; text: decodingInfo(); url: dom-mediacapabilities-decodinginfo
- type: attribute; text: powerEfficient; url: dom-mediacapabilitiesinfo-powerefficient
-
-spec: css-images-3; urlPrefix: https://www.w3.org/TR/css-images-3/
- type: dfn; text: natural dimensions; url: #natural-dimensions
- type: dfn; text: natural width; url: #natural-width
- type: dfn; text: natural height; url: #natural-height
-
spec: webrtc-svc; urlPrefix: https://w3c.github.io/webrtc-svc/
type: dfn; text: scalability mode identifier; url:#scalabilitymodes*
+
+spec: ECMASCRIPT; urlPrefix: https://tc39.es/ecma262/
+ type: dfn; text: the current Realm; url: #current-realm
@@ -108,6 +61,15 @@
"title": "Coding-independent code points for video signal type identification",
"publisher": "ITU",
"date": "December 2016"
+ },
+ "WEBCODECS-VIDEO-FRAME-METADATA-REGISTRY": {
+ "href": "https://w3c.github.io/webcodecs/video_frame_metadata_registry.html",
+ "title": "WebCodecs VideoFrame Metadata Registry",
+ "publisher": "W3C",
+ "authors": [
+ "Youenn Fablet"
+ ],
+ "status": "ED"
}
}
@@ -142,8 +104,8 @@
==========================
: Codec
-:: Refers generically to an instance of AudioDecoder, AudioEncoder,
- VideoDecoder, or VideoEncoder.
+:: Refers generically to an instance of {{AudioDecoder}}, {{AudioEncoder}},
+ {{VideoDecoder}}, or {{VideoEncoder}}.
: Key Chunk
:: An encoded chunk that does not depend on any other frames for decoding. Also
@@ -200,6 +162,17 @@
"matrix" → {{VideoMatrixCoefficients/bt709}},
"fullRange" → `false`]».
+: Codec Saturation
+:: The state of an underlying codec implementation where the number of active
+ decoding or encoding requests has reached an implementation specific
+ maximum such that it is temporarily unable to accept more work. The maximum
+ may be any value greater than 1, including infinity (no maximum). While
+ saturated, additional calls to `decode()` or `encode()` will be buffered
+ in the [=control message queue=], and will increment the respective
+ `decodeQueuSize` and `encodeQueueSize` attributes. The codec implementation
+ will become unsaturated after making sufficient progress on the current
+ workload.
+
Codec Processing Model {#codec-processing-model-section}
===================================================================
@@ -213,68 +186,84 @@
codec tasks can be scheduled while previous tasks are still pending. For
example, web authors can call `decode()` without waiting for a previous
`decode()` to complete. This is achieved by offloading underlying codec tasks
- to a separate thread for parallel execution.
+ to a separate [=parallel queue=] for parallel execution.
This section describes threading behaviors as they are visible from the
- perspective of web authors. Implementers can choose to use more or less
- threads as long as the exernally visible behaviors of blocking and sequencing
- are maintained as follows.
+ perspective of web authors. Implementers can choose to use more threads, as
+ long as the externally visible behaviors of blocking and sequencing are
+ maintained as follows.
-Control Thread and Codec Thread {#control-thread-and-codec-thread}
-------------------------------------------------------------------
-
-All steps in this specification will run on either a [=control thread=] or
-a [=codec thread=].
-
-The control thread is the thread from which authors will construct
-a [=codec=] and invoke its methods. Invoking a codec's methods will typically
-result in the creation of [=control messages=] which are later executed on the
-[=codec thread=]. Each [=webappapis/global object=] has a separate control
-thread.
+Control Messages {#control-messages}
+------------------------------------
-The codec thread is the thread from which a [=codec=] will
-[=dequeue=] [=control messages=] and execute their steps. Each [=codec=]
-instance has a separate codec thread. The lifetime of a codec thread matches
-that of its associated [=codec=] instance.
+A control message defines a sequence of steps corresponding to a
+method invocation on a [=codec=] instance (e.g. `encode()`).
-The [=control thread=] uses a traditional event loop, as described in
-[[!HTML]].
-
-The [=codec thread=] uses a specialized [=codec processing loop=].
-
-Communication from the [=control thread=] to the [=codec thread=] is done using
-[=control message=] passing. Communication in the other direction is done using
-regular event loop tasks.
-
-Each [=codec=] instance has a single control message queue that is
-a [=queue=] of control messages.
+A control message queue is a [=queue=] of
+[=control messages=]. Each [=codec=] instance has a control message queue
+stored in an internal slot named
+[[control message queue]].
Queuing a control
-message means [=enqueuing=] the message to a [=codec=]’s [=control
-message queue=]. Invoking codec methods will often queue a control message
+message means [=enqueuing=] the message to a [=codec=]’s [=[[control
+message queue]]=]. Invoking codec methods will generally queue a control message
to schedule work.
Running a control
message means performing a sequence of steps specified by the method
that enqueued the message.
-The codec processing loop MUST run these steps:
-1. While true:
- 1. If the [=control message queue=] is empty, [=continue=].
- 2. Dequeue |front message| from the [=control message queue=].
- 3. Run [=control message steps=] described by |front message|.
+The steps of a given control message can block processing later messages in the
+control message queue. Each [=codec=] instance has a boolean internal slot named
+[[message queue blocked]] that is set to `true` when this occurs. A
+blocking message will conclude by setting [=[[message queue blocked]]=] to
+`false` and rerunning the [=Process the control message queue=] steps.
+
+All control messages will return either `"processed"` or `"not processed"`.
+Returning `"processed"` indicates the message steps are being (or have been)
+executed and the message may be removed from the [=control message queue=].
+`"not processed"` indicates the message must not be processed at this time
+and should remain in the [=control message queue=] to be retried later.
+
+To Process the control message queue, run these steps:
+1. While [=[[message queue blocked]]=] is `false` and
+ [=[[control message queue]]=] is not empty:
+ 1. Let |front message| be the first message in
+ [=[[control message queue]]=].
+ 2. Let |outcome| be the result of running the [=control message steps=]
+ described by |front message|.
+ 3. If |outcome| equals `"not processed"`, break.
+ 4. Otherwise, dequeue |front message| from the
+ [=[[control message queue]]=].
+
+Codec Work Parallel Queue {#codec-work-parallel-queue}
+------------------------------------------------------
+
+Each [=codec=] instance has an internal slot named
+[[codec work queue]] that is a [=parallel queue=].
+
+Each [=codec=] instance has an internal slot named
+[[codec implementation]] that refers to the underlying platform
+encoder or decoder. Except for assignment, any steps that reference
+[=[[codec implementation]]=] will be enqueued to the [=[[codec work queue]]=].
+
+Each [=codec=] instance has a unique codec task source. Tasks
+[=queue a task|queued=] from the [=[[codec work queue]]=] to the [=/event loop=]
+will use the [=codec task source=].
+
AudioDecoder Interface {#audiodecoder-interface}
================================================
true,
+ throw an {{InvalidStateError}} {{DOMException}}.
+ 2. Return the result of calling [=Copy VideoFrame metadata=]
+ with {{VideoFrame/[[metadata]]}}.
+
### Algorithms ###{#videoframe-algorithms}
: Create a VideoFrame (with |output|, |timestamp|, |duration|, |displayAspectWidth|, |displayAspectHeight|, and |colorSpace|)
:: 1. Let |frame| be a new {{VideoFrame}}, constructed as follows:
@@ -3391,7 +3749,7 @@
|output| in pixels.
8. Let |visibleLeft|, |visibleTop|, |visibleWidth|, and |visibleHeight| be
the left, top, width and height for the visible rectangle of |output|.
- 7. Let |displayWidth| and |displayHeight| be the the display size of
+ 7. Let |displayWidth| and |displayHeight| be the display size of
|output| in pixels.
8. If |displayAspectWidth| and |displayAspectHeight| are provided,
increase |displayWidth| or |displayHeight| until the ratio of
@@ -3436,10 +3794,10 @@
{{VideoFrameInit/visibleRect}}.{{DOMRectInit/height}} == `0` return
`false`.
5. If {{VideoFrameInit/visibleRect}}.{{DOMRectInit/y}} +
- {{VideoFrameInit/visibleRect}}.{{DOMRectInit/height}} >=
+ {{VideoFrameInit/visibleRect}}.{{DOMRectInit/height}} >
|codedHeight|, return `false`.
6. If {{VideoFrameInit/visibleRect}}.{{DOMRectInit/x}} +
- {{VideoFrameInit/visibleRect}}.{{DOMRectInit/width}} >=
+ {{VideoFrameInit/visibleRect}}.{{DOMRectInit/width}} >
|codedWidth|, return `false`.
2. If |codedWidth| = 0 or |codedHeight| = 0,return `false`.
3. If only one of {{VideoFrameInit/displayWidth}} or
@@ -3455,10 +3813,10 @@
2. If any attribute of {{VideoFrameBufferInit/visibleRect}} is negative or
not finite, return `false`.
3. If {{VideoFrameBufferInit/visibleRect}}.{{DOMRectInit/y}} +
- {{VideoFrameBufferInit/visibleRect}}.{{DOMRectInit/height}} >=
+ {{VideoFrameBufferInit/visibleRect}}.{{DOMRectInit/height}} >
{{VideoFrameBufferInit/codedHeight}}, return `false`.
4. If {{VideoFrameBufferInit/visibleRect}}.{{DOMRectInit/x}} +
- {{VideoFrameBufferInit/visibleRect}}.{{DOMRectInit/width}} >=
+ {{VideoFrameBufferInit/visibleRect}}.{{DOMRectInit/width}} >
{{VideoFrameBufferInit/codedWidth}}, return `false`.
5. If only one of {{VideoFrameBufferInit/displayWidth}} or
{{VideoFrameBufferInit/displayHeight}} [=map/exists=], return `false`.
@@ -3492,14 +3850,16 @@
algorithm with |init|, |frame|, |defaultVisibleRect|,
|defaultDisplayWidth|, and |defaultDisplayHeight|.
9. If {{VideoFrameInit/duration}} [=map/exists=] in |init|, assign it to
- |frame|.{{VideoFrame/duration}}. Otherwise, assign
+ |frame|'s {{VideoFrame/[[duration]]}}. Otherwise, assign
|otherFrame|.{{VideoFrame/duration}} to
- |frame|.{{VideoFrame/duration}}.
+ |frame|'s {{VideoFrame/[[duration]]}}.
10. If {{VideoFrameInit/timestamp}} [=map/exists=] in |init|, assign it to
- |frame|.{{VideoFrame/timestamp}}. Otherwise, assign
- |otherFrame|.{{VideoFrame/timestamp}} to
- |frame|.{{VideoFrame/timestamp}}.
+ |frame|'s {{VideoFrame/[[timestamp]]}}. Otherwise, assign
+ |otherFrame|'s {{VideoFrame/timestamp}} to
+ |frame|'s {{VideoFrame/[[timestamp]]}}.
11. Assign |format| to |frame|.{{VideoFrame/[[format]]}}.
+ 12. Assign the result of calling [=Copy VideoFrame metadata=]
+ with |init|'s {{VideoFrameInit/metadata}} to |frame|.{{VideoFrame/[[metadata]]}}.
: Initialize Frame With Resource and Size (with
|init|, |frame|, |resource|, |width| and |height|)
@@ -3522,9 +3882,9 @@
algorithm with |init|, |frame|, |defaultVisibleRect|, |width|, and
|height|.
11. Assign `init`.{{VideoFrameInit/duration}} to
- |frame|.{{VideoFrame/duration}}.
+ |frame|'s {{VideoFrame/[[duration]]}}.
12. Assign `init`.{{VideoFrameInit/timestamp}} to
- |frame|.{{VideoFrame/timestamp}}.
+ |frame|'s {{VideoFrame/[[timestamp]]}}.
13. If |resource| has a known {{VideoColorSpace}}, assign its value to
{{VideoFrame/[[color space]]}}.
14. Otherwise, assign a new {{VideoColorSpace}}, constructed with an empty
@@ -3576,8 +3936,7 @@
{{VideoFrame/[[visible top]]}}, {{VideoFrame/[[visible width]]}},
{{VideoFrame/[[visible height]]}}, {{VideoFrame/[[display width]]}},
and {{VideoFrame/[[display height]]}}.
- 5. Assign `null` to |frame|'s {{VideoFrame/[[duration]]}} and
- {{VideoFrame/[[timestamp]]}}.
+ 5. Assign a new {{VideoFrameMetadata}} to |frame|.{{VideoFrame/[[metadata]]}}.
: Parse VideoFrameCopyToOptions (with |options|)
:: 1. Let |defaultRect| be the result of performing the getter steps for
@@ -3617,7 +3976,7 @@
subsample for |plane|.
4. If |rect|.{{DOMRectReadOnly/x}} is not a multiple of |sampleWidth|,
return `false`.
- 5. If |rect|.{{DOMRectReadOnly/y}} is not a multiple of |sampleHeight|,
+ 5. If |rect|.{{DOMRectReadOnly/y}} is not a multiple of |sampleHeight|,
return `false`.
8. Increment |planeIndex| by `1`.
5. Return `true`.
@@ -3709,7 +4068,8 @@
1. Assign |minAllocationSize| to |computedLayout|'s
[=computed plane layout/destinationOffset=].
- 2. Assign |computedLayout|'s [=computed plane layout/sourceWidthBytes=] to
+ 2. Assign |computedLayout|'s
+ [=computed plane layout/sourceWidthBytes=] to
|computedLayout|'s [=computed plane layout/destinationStride=].
13. Let |planeSize| be the product of multiplying |computedLayout|'s
[=computed plane layout/destinationStride=] and
@@ -3750,6 +4110,13 @@
[=combined buffer layout/allocationSize=].
9. Return |combinedLayout|.
+: Copy VideoFrame metadata (with |metadata|)
+:: 1. Let |metadataCopySerialized| be [$StructuredSerialize$](|metadata|).
+ 2. Let |metadataCopy| be [$StructuredDeserialize$](|metadataCopySerialized|, [=the current Realm=]).
+ 3. return |metadataCopy|.
+
+The goal of this algorithm is to ensure that metadata owned by a {{VideoFrame}} is immutable.
+
### Transfer and Serialization ###{#videoframe-transfer-serialization}
: The {{VideoFrame}} [=transfer steps=] (with |value| and |dataHolder|) are:
@@ -3770,7 +4137,7 @@
|forStorage|) are:
:: 1. If |value|'s {{platform object/[[Detached]]}} is `true`, throw a
{{DataCloneError}} {{DOMException}}.
- 2. If |forStorage| is `true`, throw a {{TypeError}}.
+ 2. If |forStorage| is `true`, throw a {{DataCloneError}}.
3. Let |resource| be the [=media resource=] referenced by
|value|'s {{VideoFrame/[[resource reference]]}}.
4. Let |newReference| be a new reference to |resource|.
@@ -3795,7 +4162,7 @@
Color space conversion during {{ImageBitmap}} construction is controlled by
{{ImageBitmapOptions}} {{ImageBitmapOptions/colorSpaceConversion}}. Setting this
-value to "none" disables color space conversion.
+value to {{ColorSpaceConversion/"none"}} disables color space conversion.
VideoFrame CopyTo() Options {#videoframe-copyto-options}
------------------------------------------------------------
@@ -3813,7 +4180,7 @@
NOTE: The steps of {{VideoFrame/copyTo()}} or {{VideoFrame/allocationSize()}}
will enforce the following requirements:
* The coordinates of {{VideoFrameCopyToOptions/rect}} are
- sample-aligned as determiend by {{VideoFrame/[[format]]}}.
+ sample-aligned as determined by {{VideoFrame/[[format]]}}.
* If {{VideoFrameCopyToOptions/layout}} [=map/exists=], a {{PlaneLayout}}
is provided for all planes.
@@ -3836,7 +4203,7 @@
DOMRects in VideoFrame {#videoframe-domrect}
--------------------------------------------
-The {{VideoFrame}} interfaces uses {{DOMRect}}s to specify the position and
+The {{VideoFrame}} interface uses {{DOMRect}}s to specify the position and
dimensions for a rectangle of pixels. {{DOMRectInit}} is used with
{{VideoFrame/copyTo()}} and {{VideoFrame/allocationSize()}} to describe the
dimensions of the source rectangle. {{VideoFrame}} defines
@@ -3853,7 +4220,7 @@
{{PlaneLayout}}s MAY be provided to {{VideoFrame}}'s
{{VideoFrame/copyTo()}} to specify how the plane is laid out in the destination
{{BufferSource}}}. Alternatively, callers can inspect {{VideoFrame/copyTo()}}'s
-returned sequence of {{PlaneLayout}}s to learn the the offset and stride for
+returned sequence of {{PlaneLayout}}s to learn the offset and stride for
planes as decided by the User Agent.
[Exposed=(Window,DedicatedWorker)] -interface ImageTrack : EventTarget { +interface ImageTrack { readonly attribute boolean animated; readonly attribute unsigned long frameCount; readonly attribute unrestricted float repetitionCount; - attribute EventHandler onchange; attribute boolean selected; }; @@ -5072,10 +5510,6 @@ :: The {{ImageTrack/repetitionCount}} getter steps are to return the value of {{ImageTrack/[[repetition count]]}}. -: onchange -:: An [=event handler IDL attribute=] whose [=event handler event type=] is - {{ImageTrack/change}}. - : selected :: The {{ImageTrack/selected}} getter steps are to return the value of {{ImageTrack/[[selected]]}}. @@ -5106,19 +5540,17 @@ 11. [=Queue a control message=] to {{ImageTrack/[[ImageDecoder]]}}'s [=control message queue=] to update the internal selected track index with |selectedIndex|. + 12. [=Process the control message queue=] belonging to + {{ImageTrack/[[ImageDecoder]]}}. [=Running a control message=] to update the internal selected track index means running these steps: - 1. Assign |selectedIndex| to - {{ImageDecoder/[[internal selected track index]]}}. - 2. Remove all entries from - {{ImageDecoder/[[progressive frame generations]]}}. - - -### Event Summary ### {#imagetracklist-eventsummary} - -: change -:: Fired at the {{ImageTrack}} when the {{ImageTrack/frameCount}} is altered. + 1. Enqueue the following steps to {{ImageTrack/[[ImageDecoder]]}}'s + {{ImageDecoder/[[codec work queue]]}}: + 1. Assign |selectedIndex| to + {{ImageDecoder/[[internal selected track index]]}}. + 2. Remove all entries from + {{ImageDecoder/[[progressive frame generations]]}}. Resource Reclamation{#resource-reclamation} ============================================== @@ -5165,7 +5597,7 @@ - An {{AudioDecoder}} or {{VideoDecoder}}, when there is, respectively, an [=active=] {{AudioEncoder}} or {{VideoEncoder}} in the same - [=webappapis/global object=]. + [=/global object=]. NOTE: This prevents prevents breaking long running transcoding tasks. @@ -5174,11 +5606,14 @@ Security Considerations{#security-considerations} ================================================= ++This section is non-normative. + The primary security impact is that features of this API make it easier for an attacker to exploit vulnerabilities in the underlying platform codecs. -Additionally, new abilities to configure and control the codecs MAY allow for new exploits that rely on a specific -configuration and/or sequence of control operations. +Additionally, new abilities to configure and control the codecs can allow for +new exploits that rely on a specific configuration and/or sequence of control +operations. Platform codecs are historically an internal detail of APIs like {{HTMLMediaElement}}, [[WEBAUDIO]], and [[WebRTC]]. In this way, it has always @@ -5195,43 +5630,43 @@ affords attackers the ability to invoke sequences of control methods that were not previously possible via the higher level APIs. -User Agents SHOULD mitigate this risk by extensively +The Working Group expects User Agents to mitigate this risk by extensively fuzzing their implementation with random inputs and control method invocations. Additionally, User Agents are encouraged to isolate their underlying codecs in processes with restricted privileges (sandbox) as a barrier against successful exploits being able to read user data. An additional concern is exposing the underlying codecs to input mutation race -conditions. Specifically, it SHOULD not be possible for -a site to mutate a codec input or output while the underlying codec MAY still be operating on that data. This concern is -mitigated by ensuring that input and output interfaces are immutable. +conditions, such as allowing a site to mutate a codec input or output while +the underlying codec is still operating on that data. This concern is mitigated +by ensuring that input and output interfaces are immutable. Privacy Considerations{#privacy-considerations} =============================================== ++This section is non-normative. + The primary privacy impact is an increased ability to fingerprint users by querying for different codec capabilities to establish a codec feature profile. Much of this profile is already exposed by existing APIs. Such profiles are very -unlikely to be uniquely identifying, but MAY be used -with other metrics to create a fingerprint. - -An attacker MAY accumulate a codec feature profile by -calling `IsConfigSupported()` methods with a number of different configuration -dictionaries. Similarly, an attacker MAY attempt to -`configure()` a codec with different configuration dictionaries and observe -which configurations are accepted. - -Attackers MAY also use existing APIs to establish much -of the codec feature profile. For example, the [[media-capabilities]] -{{decodingInfo()}} API describes what types of decoders are supported and its -{{powerEfficient}} attribute MAY signal when a decoder -uses hardware acceleration. Similarly, the [[WebRTC]] -{{RTCRtpSender/getCapabilities()}} API MAY be used to -determine what types of encoders are supported and the -{{RTCPeerConnection/getStats()}} API MAY be used to -determine when an encoder uses hardware acceleration. WebCodecs will expose some -additional information in the form of low level codec features. +unlikely to be uniquely identifying, but can be used with other metrics to +create a fingerprint. + +An attacker can accumulate a codec feature profile by calling +`IsConfigSupported()` methods with a number of different configuration +dictionaries. Similarly, an attacker can attempt to `configure()` a codec with +different configuration dictionaries and observe which configurations are +accepted. + +Attackers can also use existing APIs to establish much of the codec feature +profile. For example, the [[media-capabilities]] {{decodingInfo()}} API +describes what types of decoders are supported and its {{powerEfficient}} +attribute can signal when a decoder uses hardware acceleration. Similarly, the +[[WebRTC]] {{RTCRtpSender/getCapabilities()}} API can be used to determine what +types of encoders are supported and the {{RTCPeerConnection/getStats()}} API can +be used to determine when an encoder uses hardware acceleration. WebCodecs will +expose some additional information in the form of low level codec features. A codec feature profile alone is unlikely to be uniquely identifying. Underlying codecs are often implemented entirely in software (be it part of the User Agent @@ -5240,9 +5675,9 @@ are often implemented with hardware acceleration, but such hardware is mass produced and devices of a particular class and manufacture date (e.g. flagship phones manufactured in 2020) will often have common capabilities. There will be -outliers (some users MAY run outdated versions of -software codecs or use a rare mix of custom assembled hardware), but most of the -time a given codec feature profile is shared by a large group of users. +outliers (some users can be running outdated versions of software codecs or use +a rare mix of custom assembled hardware), but most of the time a given codec +feature profile is shared by a large group of users. Segmenting groups of users by codec feature profile still amounts to a bit of entropy that can be combined with other metrics to uniquely identify a user. @@ -5263,7 +5698,7 @@ realtime media or in contended main thread environments are encouraged to ensure their media pipelines operate in worker contexts entirely independent of the main thread where possible. For example, realtime media processing of {{VideoFrame}}s - are generally be done in a worker context. + are generally to be done in a worker context. The main thread has significant potential for high contention and jank that can go unnoticed in development, yet degrade inconsistently across devices and User @@ -5275,3 +5710,14 @@ their target frame rates, main thread workload, how their application will be embedded, and the class of devices their users will be using.+ +Acknowledgements{#acknowledgements} +=================================== + +The editors would like to thank Alex Russell, Chris Needham, Dale Curtis, Dan +Sanders, Eugene Zemtsov, Francois Daoust, Guido Urdaneta, Harald Alvestrand, +Jan-Ivar Bruaroey, Jer Noble, Mark Foltz, Peter Thatcher, Steve Anton, Matt +Wolenetz, Rijubrata Bhaumik, Thomas Guilbert, Tuukka Toivonen, and Youenn Fablet +for their contributions to this specification. Thank you also to the many +others who contributed to the specification, including through their +participation on the mailing list and in the issues. diff --git a/mp3_codec_registration.src.html b/mp3_codec_registration.src.html index df920787..184ab396 100644 --- a/mp3_codec_registration.src.html +++ b/mp3_codec_registration.src.html @@ -7,15 +7,16 @@ Group: mediawg ED: https://w3c.github.io/webcodecs/mp3_codec_registration.html TR: https://www.w3.org/TR/webcodecs-mp3-codec-registration/ -Editor: Chris Cunningham, w3cid 114832, Google Inc. https://www.google.com/ Editor: Paul Adenot, w3cid 62410, Mozilla https://www.mozilla.org/ Editor: Bernard Aboba, w3cid 65611, Microsoft Corporation https://www.microsoft.com/ +Former Editor: Chris Cunningham, w3cid 114832, Google Inc. https://www.google.com/ Abstract: This registration is entered into the [[webcodecs-codec-registry]]. It describes, for MP3, the (1) fully qualified codec strings, (2) - the {{AudioDecoderConfig.description}} bytes, (3) the codec-specific - {{EncodedAudioChunk}} [=EncodedAudioChunk/[[internal data]]=] bytes, and (4) - the values of {{EncodedAudioChunk}} [=EncodedAudioChunk/[[type]]=]. + the {{AudioDecoderConfig/description|AudioDecoderConfig.description}} bytes, + (3) the codec-specific {{EncodedAudioChunk}} + {{EncodedAudioChunk/[[internal data]]}} bytes, and (4) + the values of {{EncodedAudioChunk}} {{EncodedAudioChunk/[[type]]}}. The registration is not intended to include any information on whether a codec format is encumbered by intellectual property claims. Implementers and @@ -31,21 +32,6 @@ !Version History: https://github.com/w3c/webcodecs/commits --spec: WEBCODECS; urlPrefix: https://w3c.github.io/webcodecs/# - type: attribute - text: AudioDecoderConfig.description; url: dom-audiodecoderconfig-description - text: AudioDecoderConfig.sampleRate; url: dom-audiodecoderconfig-samplerate - text: AudioDecoderConfig.numberOfChannels; url: dom-audiodecoderconfig-numberofchannels - type: dfn - for: EncodedAudioChunkType; text: key; url: dom-encodedaudiochunktype-key - for: EncodedAudioChunk; text: [[internal data]]; url: dom-encodedaudiochunk-internal-data-slot - for: EncodedAudioChunk; text: [[type]]; url: dom-encodedaudiochunk-type-slot - type: interface - text: EncodedAudioChunk; url: encodedaudiochunk - type: dictionary - text: AudioDecoderConfig; url: dictdef-audiodecoderconfig -{ @@ -72,31 +58,36 @@ EncodedAudioChunk data {#encodedaudiochunk-data} ================================================ -{{EncodedAudioChunk}} [=EncodedAudioChunk/[[internal data]]=] is expected to be +{{EncodedAudioChunk}} {{EncodedAudioChunk/[[internal data]]}} is expected to be a "frame", as described in the section 2.4.2.2 of the [[MP3]] specification. AudioDecoderConfig description {#audiodecoderconfig-description} ================================================================ -{{AudioDecoderConfig.description}} is not used for this codec. +{{AudioDecoderConfig/description}} is not used for this codec. -The {{AudioDecoderConfig.sampleRate}} and {{AudioDecoderConfig.numberOfChannels}} +The {{AudioDecoderConfig/sampleRate}} and {{AudioDecoderConfig/numberOfChannels}} members are ignored. EncodedAudioChunk type {#encodedaudiochunk-type} ================================================ -The [=EncodedAudioChunk/[[type]]=] for an {{EncodedAudioChunk}} containing -mp3 is always "[=EncodedAudioChunkType/key=]". +The {{EncodedAudioChunk/[[type]]}} for an {{EncodedAudioChunk}} containing +mp3 is always "{{EncodedAudioChunkType/key}}". NOTE: Once the initialization has succeeded, any mp3 packet can be decoded at any time without error, but this might not result in the expected audio output. -Privacy and Security Considerations {#privacy-and-security-considerations} +Privacy Considerations {#privacy-considerations} +========================================================================== + +Please refer to the section [[WEBCODECS#privacy-considerations|Privacy +Considerations]] in [[WEBCODECS]]. + +Security Considerations {#security-considerations} ========================================================================== -Please refer to the [[WEBCODECS#privacy-considerations|Privacy Considerations]] -and [[WEBCODECS#security-considerations|Security Considerations]] sections in -[[WEBCODECS]]. +Please refer to the section [[WEBCODECS#security-considerations|Security +Considerations]] in [[WEBCODECS]]. diff --git a/opus_codec_registration.src.html b/opus_codec_registration.src.html index 90c46b38..65b96f2b 100644 --- a/opus_codec_registration.src.html +++ b/opus_codec_registration.src.html @@ -7,16 +7,16 @@ Group: mediawg ED: https://w3c.github.io/webcodecs/opus_codec_registration.html TR: https://www.w3.org/TR/webcodecs-opus-codec-registration/ -Editor: Chris Cunningham, w3cid 114832, Google Inc. https://www.google.com/ Editor: Paul Adenot, w3cid 62410, Mozilla https://www.mozilla.org/ Editor: Bernard Aboba, w3cid 65611, Microsoft Corporation https://www.microsoft.com/ +Former Editor: Chris Cunningham, w3cid 114832, Google Inc. https://www.google.com/ Abstract: This registration is entered into the [[webcodecs-codec-registry]]. It describes, for Opus, the (1) fully qualified codec strings, (2) the - codec-specific {{EncodedAudioChunk}} [=EncodedAudioChunk/[[internal data]]=] - bytes, (3) the {{AudioDecoderConfig.description}} bytes, (4) the values of - {{EncodedAudioChunk}} [=EncodedAudioChunk/[[type]]=], and (5) the - codec-specific extensions to {{AudioEncoderConfig}} + codec-specific {{EncodedAudioChunk}} {{EncodedAudioChunk/[[internal data]]}} + bytes, (3) the {{AudioDecoderConfig/description|AudioDecoderConfig.description}} + bytes, (4) the values of {{EncodedAudioChunk}} {{EncodedAudioChunk/[[type]]}}, + and (5) the codec-specific extensions to {{AudioEncoderConfig}} The registration is not intended to include any information on whether a codec format is encumbered by intellectual property claims. Implementers and @@ -32,22 +32,6 @@ !Version History: https://github.com/w3c/webcodecs/commits--spec: WEBCODECS; urlPrefix: https://w3c.github.io/webcodecs/# - type: attribute - text: AudioDecoderConfig.description; url: dom-audiodecoderconfig-description - type: dfn - for: EncodedAudioChunkType; text: key; url: dom-encodedaudiochunktype-key - for: EncodedAudioChunk; text: [[internal data]]; url: dom-encodedaudiochunk-internal-data-slot - for: EncodedAudioChunk; text: [[type]]; url: dom-encodedaudiochunk-type-slot - type: method - for: AudioDecoder; text: AudioDecoder.decode; url: dom-audiodecoder-decode - type: interface - text: EncodedAudioChunk; url: encodedaudiochunk - type: dictionary - text: AudioDecoderConfig; url: dictdef-audiodecoderconfig - text: AudioEncoderConfig; url: dictdef-audioencoderconfig -{ @@ -87,25 +71,25 @@ AudioDecoderConfig description {#audiodecoderconfig-description} ================================================================ -{{AudioDecoderConfig.description}} can optionally set to an Identification +{{AudioDecoderConfig/description}} can optionally set to an Identification Header, described in section 5.1 of [[OPUS-IN-OGG]]. -If an {{AudioDecoderConfig.description}} has been set, the bistream is assumed +If a {{AudioDecoderConfig/description}} has been set, the bistream is assumed to be in {{OpusBitstreamFormat/ogg}} format. -If an {{AudioDecoderConfig.description}} has not been set, the bitstream is +If a {{AudioDecoderConfig/description}} has not been set, the bitstream is assumed to be in {{OpusBitstreamFormat/opus}} format. EncodedAudioChunk type {#encodedaudiochunk-type} ================================================ -The [=EncodedAudioChunk/[[type]]=] for an {{EncodedAudioChunk}} containing -Opus is always "[=EncodedAudioChunkType/key=]". +The {{EncodedAudioChunk/[[type]]}} for an {{EncodedAudioChunk}} containing +Opus is always "{{EncodedAudioChunkType/key}}". NOTE: Once the initialization has succeeded, any packet can be decoded at any time without error, but this might not result in the expected audio output. -AudiEncoderConfig extensions {#audioencoderconfig-extensions} +AudioEncoderConfig extensions {#audioencoderconfig-extensions} =============================================================@@ -131,16 +115,60 @@+To check if an {{OpusEncoderConfig}} is valid, run these steps: +1. If {{OpusEncoderConfig/frameDuration}} is not a valid frame duration, + which is described section 2.1.4 of [[RFC6716]], return `false`. +1. If {{OpusEncoderConfig/complexity}} is specified and not within the range of + `0` and `10` inclusively, return `false`. +1. If {{OpusEncoderConfig/packetlossperc}} is specified and not within the range of + `0` and `100` inclusively, return `false`. +2. Return `true`. +dictionary OpusEncoderConfig { OpusBitstreamFormat format = "opus"; + [EnforceRange] unsigned long long frameDuration = 20000; + [EnforceRange] unsigned long complexity; + [EnforceRange] unsigned long packetlossperc = 0; + boolean useinbandfec = false; + boolean usedtx = false; }; OpusBitstreamFormat {#opus-bitstream-format} @@ -165,13 +193,18 @@
- format
- Configures the format of output {{EncodedAudioChunk}}s. See {{OpusBitstreamFormat}}.
+- frameDuration
+- + Configures the frame duration, in microseconds, of output {{EncodedAudioChunk}}s. +
+- complexity
+- + Configures the encoder's computational complexity, as described in section 2.1.9. + of [[RFC6716]]. The valid range is `0` to `10`, with `10` representing the highest + complexity. If no value is specificied, the default value is platform-specific: + User Agents SHOULD set a default of `5` for mobile + platforms, and a default of `9` for all other platforms. +
+- packetlossperc
+- + Configures the encoder's expected packet loss percentage. The valid range is + `0` to `100`. + + NOTE: The packet loss percentage might be updated over the course of an + encoding, and it is recommended for User Agents to support these reconfigurations. +
+- useinbandfec
+- + Specifies whether the encoder provides Opus in-band Forward Error Correction + (FEC), as described by section 2.1.7. of [[RFC6716]]. +
+- usedtx
+- + Specifies if the encoder uses Discontinuous Transmission (DTX), as described + by section 2.1.9. of [[RFC6716]]. +
ogg The metadata of the encoded audio stream are provided at configuration via - {{AudioDecoderConfig.description}}. + {{AudioDecoderConfig/description|AudioDecoderConfig.description}}. -Privacy and Security Considerations {#privacy-and-security-considerations} +Privacy Considerations {#privacy-considerations} +========================================================================== + +Please refer to the section [[WEBCODECS#privacy-considerations|Privacy +Considerations]] in [[WEBCODECS]]. + +Security Considerations {#security-considerations} ========================================================================== -Please refer to the [[WEBCODECS#privacy-considerations|Privacy Considerations]] -and [[WEBCODECS#security-considerations|Security Considerations]] sections in -[[WEBCODECS]]. +Please refer to the section [[WEBCODECS#security-considerations|Security +Considerations]] in [[WEBCODECS]]. diff --git a/pcm_codec_registration.src.html b/pcm_codec_registration.src.html index 0d6698af..663c6780 100644 --- a/pcm_codec_registration.src.html +++ b/pcm_codec_registration.src.html @@ -7,16 +7,16 @@ Group: mediawg ED: https://w3c.github.io/webcodecs/pcm_codec_registration.html TR: https://www.w3.org/TR/webcodecs-pcm-codec-registration/ -Editor: Chris Cunningham, w3cid 114832, Google Inc. https://www.google.com/ Editor: Paul Adenot, w3cid 62410, Mozilla https://www.mozilla.org/ Editor: Bernard Aboba, w3cid 65611, Microsoft Corporation https://www.microsoft.com/ +Former Editor: Chris Cunningham, w3cid 114832, Google Inc. https://www.google.com/ Abstract: This registration is entered into the [[webcodecs-codec-registry]]. It describes, for Linear PCM, the (1) fully qualified codec strings, (2) the codec-specific {{EncodedAudioChunk}} - [=EncodedAudioChunk/[[internal data]]=] bytes, (3) the - {{AudioDecoderConfig.description}}, and (4) the values of - {{EncodedAudioChunk}} [=EncodedAudioChunk/[[type]]=]. + {{EncodedAudioChunk/[[internal data]]}} bytes, (3) the + {{AudioDecoderConfig/description|AudioDecoderConfig.description}}, and + (4) the values of {{EncodedAudioChunk}} {{EncodedAudioChunk/[[type]]}}. Linear PCM is the [[webcodecs#audio-sample-formats|raw audio format]] used in WebCodecs and does not require decoding. The motivation for registering @@ -38,28 +38,8 @@spec: WEBCODECS; urlPrefix: https://w3c.github.io/webcodecs/# - type: attribute - text: AudioDecoderConfig.description; url: dom-audiodecoderconfig-description type: dfn - for: EncodedAudioChunkType; text: key; url: dom-encodedaudiochunktype-key - for: EncodedAudioChunk; text: [[internal data]]; url: dom-encodedaudiochunk-internal-data-slot - for: EncodedAudioChunk; text: [[type]]; url: dom-encodedaudiochunk-type-slot - text: sample; url: sample text: interleaved; url: interleaved - text: section on sample magnitude; url: audio-samples-magnitude - type: interface - text: EncodedAudioChunk; url: encodedaudiochunk - text: AudioData; url: audiodata-interface - type: dictionary - text: AudioDecoderConfig; url: dictdef-audiodecoderconfig - text: AudioEncoderConfig; url: dictdef-audioencoderconfig - type: enum - text: AudioSampleFormat; url: enumdef-audiosampleformat - text: u8; url: dom-audiosampleformat-u8 - text: s16; url: dom-audiosampleformat-s16 - text: s24; url: dom-audiosampleformat-s24 - text: s32; url: dom-audiosampleformat-s32 - text: f32; url: dom-audiosampleformat-f32@@ -80,7 +60,7 @@ NOTE: [[WEBCODECS]] does not define a 24-bit {{AudioSampleFormat}}. 24-bit samples are permitted within an {{EncodedAudioChunk}}, but such samples will be "decoded" in {{AudioData}} objects as either {{s32}} of {{f32}}. Please - see [[WEBCODECS]] [=section on sample magnitude=] for addtional details. + see [[WEBCODECS#audio-samples-magnitude]] for additional details. EncodedAudioChunk data {#encodedaudiochunk-data} ================================================ @@ -89,7 +69,7 @@ values are sampled at a regular interval, and where the quantization levels between two successive values are linearly uniform. -{{EncodedAudioChunk}} [=EncodedAudioChunk/[[internal data]]=] is expected to be +{{EncodedAudioChunk}} {{EncodedAudioChunk/[[internal data]]}} is expected to be a sequence of bytes of arbitrary length, with a [=sample=] occuring every N bits, where N is defined by the codec string. For multichannel PCM, [=samples=] from different channels are [=interleaved=]. @@ -98,18 +78,23 @@ AudioDecoderConfig description {#audiodecoderconfig-description} ================================================================ -An {{AudioDecoderConfig.description}} is expected to be omitted from the +The {{AudioDecoderConfig/description}} is expected to be omitted from the {{AudioDecoderConfig}}. EncodedAudioChunk type {#encodedaudiochunk-type} ================================================ -The [=EncodedAudioChunk/[[type]]=] for an {{EncodedAudioChunk}} containing -Linear PCM is always "[=EncodedAudioChunkType/key=]". +The {{EncodedAudioChunk/[[type]]}} for an {{EncodedAudioChunk}} containing +Linear PCM is always "{{EncodedAudioChunkType/key}}". -Privacy and Security Considerations {#privacy-and-security-considerations} +Privacy Considerations {#privacy-considerations} ========================================================================== -Please refer to the [[WEBCODECS#privacy-considerations|Privacy Considerations]] -and [[WEBCODECS#security-considerations|Security Considerations]] sections in -[[WEBCODECS]]. +Please refer to the section [[WEBCODECS#privacy-considerations|Privacy +Considerations]] in [[WEBCODECS]]. + +Security Considerations {#security-considerations} +========================================================================== + +Please refer to the section [[WEBCODECS#security-considerations|Security +Considerations]] in [[WEBCODECS]]. diff --git a/samples/README.md b/samples/README.md index 68980b2c..54cc85db 100644 --- a/samples/README.md +++ b/samples/README.md @@ -1,6 +1,6 @@ # Samples -Try out our [WebCodecs samples](https://w3c.github.io/webcodecs/samples/). +Try out our [WebCodecs samples](https://webcodecs-samples.netlify.app/). Please make sure you are using the latest version of Chrome Canary to run these pages, as the latest API changes might not have made it to Chrome Stable. diff --git a/samples/_headers b/samples/_headers new file mode 100644 index 00000000..f47c3755 --- /dev/null +++ b/samples/_headers @@ -0,0 +1,4 @@ +/* + Cross-Origin-Opener-Policy: same-origin + Cross-Origin-Embedder-Policy: require-corp + diff --git a/samples/audio-video-player/audio_video_player.html b/samples/audio-video-player/audio_video_player.html new file mode 100644 index 00000000..c6c8a47b --- /dev/null +++ b/samples/audio-video-player/audio_video_player.html @@ -0,0 +1,185 @@ + + +++ ++ This sample combines WebCodecs and WebAudio to create a media player that + renders synchronized audio and video. +
++ Check out the Video Decoding and Display + demo for a simpler introduction to video decoding and rendering. View + this video presentation + for an overview of audio rendering stack. +
++ This sample requires + cross origin isolation to use + SharedArrayBuffer. You may use + node server.js to host this sample locally + with the appropriate HTTP headers. +
+++ ++ Video Codec: + + + + + +
+ + + +