Support --preload-file in the shell environment and d8

[kripken]

This does so using a polyfill for XHR. That has downsides, but the shell
environment is only really used for local testing, so that seems ok.

Other options to fix the general issue of embed-file being slow on huge
files is to optimize embed-file's output. However, testing of emitting the
base64-encoded data in chunks is inconclusive: faster to parse in our
JS optimizer but slower in closure.

[ngzhian]

Nice! Thank you, can confirm the compile is really fast now, and also the polyfill works with d8.

[ngzhian]

Forgot to LGTM.

[sbc100]

We already have a helper function called readAsync which works on all platforms.. can we not instead use that existing abstraction instead of relying on a web API and then poly-filling it?

IIRC that hard part of allowing file package to run both with and without the emscripten glue code.. but maybe file_packager can use readAsync when it knows the output will be embedded in the emscripten glue?

[kripken]

Yes, the problem is the file packager code is separate. It can't depend on readAsync being present for it, as it may have been compiled separately. We can't even export readAsync for it, as the file packager output is designed to start to run before the emscripten code even arrives - so that it can fire off async fetches while the emscripten code is being loaded.

(Note that I don't think polyfilling a web API is that bad. In fact, it makes the code simpler: all the code uses the Web API, and the polyfill is the one place that has special code for d8. That is the approach suggested in #12203. In comparison the node support in the file packager atm depends on having special node code in that file.)

[sbc100]

Yes, the problem is the file packager code is separate. It can't depend on readAsync being present for it, as it may have been compiled separately. We can't even export readAsync for it, as the file packager output is designed to start to run before the emscripten code even arrives - so that it can fire off async fetches while the emscripten code is being loaded.

If the file packager output is designed to run before the emscripten code arrives.. then how would this polyfil work.. since the polyfill itself lives in the emscripten code?

[kripken]

Ah, good point... it won't work in totally separate output, you're right.

It does work in emcc --preload-file, that is, when file packager code is included as part of the build. That code appears first, before almost anything else, to allow it to start the fetches as early as possible. That's why the polyfill appears where it does.

[sbc100]

Ah, good point... it won't work in totally separate output, you're right.

It does work in emcc --preload-file, that is, when file packager code is included as part of the build. That code appears first, before almost anything else, to allow it to start the fetches as early as possible. That's why the polyfill appears where it does.

Thats what I man. Is file_packer aware when its running in --preload-file mode? can it call readAsync.. in that case?

[sbc100]

OK, how about an internal settings called INCLUDE_XHR_POLYFILL which can be set when the file packager is used? Then we can avoid this unless is needed?

[kripken]

Thinking more on this, when the new FS is ready we should be able to embed files entirely in C. That would avoid this issues that led to this PR, as it would be easy and efficient. So let's close this, which would only help the old FS.