Description
Migrating from this comment thread
@mathiasbynens wrote:
Why is this a promise-based API? Can we instead return the boolean synchronously? dalecurtis/image-decoder-api#6
@chcunningham wrote:
This was made promise as we anticipate cases where the UA may not synchronously have the answer. As image formats have started to use video codecs, decoding an image may require instantiating video decoders backed by platform APIs that. A browser architecture may be such that these APIs are called in a separate process, sandboxed for improved security. Supported types then becomes a question for these same APIs, which is implemented via async IPC.
Earlier media capability detection APIs,
@mathiasbynens wrtoe:
I understand that actual image decoding might depend on another process, but answering the question "do I know what to do with this image type at all (without necessarily doing the work)" seems orthogonal to that. Couldn't browsers just maintain a list of supported image types and synchronously check it whenever isTypeSupported is called?
If not, then have we considered changing the name of this API? IMHO it's surprising to give it the same name as an existing API without matching the signature of its return value.
@chcunningham wrote
If the browser entirely relies on the OS to provide the codec, it may not be possible to know statically what codecs are supported (particularly for newer formats). Instead, we may be forced to query OS apis that are adjacent to the actual decoding APIs. Often this involves the same async IPC to a privaledged sandboxed process.
If not, then have we considered changing the name of this API? IMHO it's surprising to give it the same name as an existing API without matching the signature of its return value.
Open to suggestions. I liked this name for its similarity actually. It is performing essentially the same function as its predecessor. I think any confusion would be pretty immediately resolved at dev-time.
@padenot wrote:
If the browser entirely relies on the OS to provide the codec, it may not be possible to know statically what codecs are supported (particularly for newer formats). Instead, we may be forced to query OS apis that are adjacent to the actual decoding APIs. Often this involves the same async IPC to a privaledged sandboxed process.
In my experience, this is mostly true for video, in the sense that it's well possible that the device that allows (say) power or cpu-efficient decoding is simply physically removed, and so that it's impossible to store the capabilities somewhere on the browser for synchronous access.
Do we have any evidence of a similar constraint for images? In our experience, hardware decoding for images has this problem where the setup time drawfs the decoding time, and the setup time need to happen per image (with a possible edge case when lots of images have the same dimensions/format maybe?).
@dalecurtis wrote:
For HEIF I would expect implementors will require hardware support to enable the functionality. Early in process startup, this may not yet be known.
@padenot wrote:
Do you mean HEIC? HEIF is supported in software today by Chrome and Firefox.
@dalecurtis wrote:
Yes, sorry. I keep forgetting that last letter :) In my head AVIF is what we call what Chrome and Firefox have.
@padenot wrote:
Yes, same, but our image folks made the distinction when I asked :-)
Per the TAG (a precedent with the autoplay policy, that took ages to resolve), a promise is to be used only if the information can change during the lifetime of the document. If not, then it's on the implementer to have the information ready. For example, in Gecko, this is part of the information received when a process is created.
If we anticipate that the support for this image format can be dynamic, it's going to be necessary (but annoying to have to do this async, because of an edge case that is not shipped anywhere and where it's not clear if it's going to be shipped anywhere). It would be a strange precedent however.
@dalecurtis wrote:
At least in Chrome, in the event of a gpu process crash, it is possible support for some formats is no longer available.
Otherwise I agree, we're just making this async for symmetry and a hypothetical. Maybe @jernoble or @aboba want to chime in to avoid a decision that would preclude any formats they might want to support.
@jrmuizel wrote:
@dalecurtis what formats aren't supported in Chrome when the GPU process crashes?
@dalecurtis wrote:
In the hypothetical world where a browser only has support for HEIC via a platform decoder, there is a case where repeated GPU crashes may put the browser into a software-rendering/no-gpu mode that prevents access to the platform decoder.
As a practical example, in some cases the platform decoder may have hard limits on the number of instances available and/or may not be working reliably (e.g., it's hanging or crashing too frequently) to the point that it's disabled at runtime. This happens today in Chrome on Android for H.264 support.
@jrmuizel wrote:
Ok, I think it's unlikely that a browser would choose to support an image format and not have a software fallback, but even if such a browser did exist the web author would still need to deal with decoding support going away after they've already checked for support withisTypeSupported().
@dalecurtis wrote:
Practically we don't have software fallback on Android for H.264 today -- so that at least is a real case. In the case where the codec goes away during usage the decoder would trigger a decoding error. The same would occur if the loss occurred between construction and the first decode call.
I think authors would have to handle this case regardless of if iTS is sync or not though. For security reasons (e.g., malicious invocation of the platform decoder), even if software fallback is available, it's unclear that automatic fallback is the right operation.
Ultimately my initial implementation was always synchronous, so if everyone wants this to be sync and isn't swayed by the hypotheticals I don't mind switching it back. I believe @chcunningham only suggested it for symmetry with the rest of the WebCodecs APIs.