Support UTF-8 strings out of the box

[pannous]

currently strings seem to be encoded in something like utf-16.
(data (i32.const 1000) "\03\00\00\00H\00I\00H\00")
would it be possible to use utf-8 instead (as compile flag or default)?

[dcodeIO]

In general, the standard library is designed in a way that it can be replaced with a custom one, including replacing the string implementation, but UTF-8 support would also be a great addition as a compile time flag (or maybe a special string class), as you propose.

The idea behind picking UTF-16 in the first place was that the most prominent host environment will be a browser with JavaScript, where string methods usually deal with UTF-16 code units. Though, little is known on the exact requirements future bindings will have. Whatever default implementation(s) we finally pick should ideally not require translating strings from one encoding to another when calling between the host and WASM.

[pannous]

Is it possible to directly instantiate a wasm UTF-16 array into a js string?

what I've seen so far were transformations like:

	for (var i=offset;buffer[i] && i<buffer.length; i++) 
	  	str += String.fromCharCode(buffer[i]);

Or put another way: is let string = buffer.toString('utf16le') so much more efficient then let string = buffer.toString('utf8') ?

If no direct instantiation is possible that would be another reason to use utf-8.
By the time wasm gets direct access to js objects, the marshaling will probably be solved by the runtime anyways, or browsers might even change their internal representation. (?)

[dcodeIO]

If no direct instantiation is possible that would be another reason to use utf-8.

The expectation I have is that JS VMs use UTF-16 internally, so it isn't necessary to convert between encodings when loading / storing strings from WASM. That's somewhat shadowed by the fact that one cannot directly instantiate a string from a buffer, of course (except when using node, but even node has to convert bytes to a string some way or another). I also must admit that haven't I investigated if UTF-16 is actually what a browser uses.

Ideally, we should pick the format as the default that implies the least conversion steps (even if the runtime does them for us), but with just reading memory and calling imports / exports, it's hard to tell at this point in time.

[MaxGraey]

UTF-8 also require additional methods for separate work with utf8 and utf16. For example Rust have String::from_utf16 and String::from_utf8, String.len() and String.chars.count() and etc. All that not back compatible with TypeScript and should require additional steps for interop with JavaScript side

[toonsevrin]

Is there a std function already to convert to utf-8? If not, I'll implement it if you provide a prefered location for it.

[MaxGraey]

This can be close for now as well?

[pannous]

is utf-8 default now, or is it optional?

[MaxGraey]

There are special methods toUTF8() and get lengthUTF8(). But no method for fromUTF8. Is it really need?

And internally for all strings using UTF-16LE same as JS/TS

[fcrick]

I was also surprised to find a double byte internal encoding. It's double the memory...

[dcodeIO]

There are two major reasons for picking UTF-16:

• Compatibility: JavaScript, and thus TypeScript, commonly uses Web APIs that specifically expect UTF-16 character codes. While JS also supports some UTF-8 APIs nowadays like String.fromCodePoint and String#codePointAt, the latter, for example, still expects the underlying data to be UTF-16, just like all the other String APIs.
• Efficiency: When transfering string data between WebAssembly and JavaScript, there's no additional encoding or decoding step required as it'd be with UTF-8.

Nonetheless, better support for UTF-8 encoded data is of course something we should investigate further. Possible improvements range from efficient conversion functions as partly implemented above to a full-fledged UTF8String class with an UTF-16 compatibility layer that can be used alternatively, if someone is willing to implement it.