implement uc and lc in C for UTF-8 strings

[StefanKarpinski]

A single function may be able to do this depending on the details of UTF-8 encoding.

[StefanKarpinski]

The best resource I've found for this is: http://developer.gnome.org/glib/2.29/glib-Unicode-Manipulation.html

[JeffBezanson]

Are these towupper and towlower?

[StefanKarpinski]

For characters, yes, and that's how they're implemented. For ASCII strings, this is pretty easy. For UTF-8 strings, it's complicated because in principle, you have to decode each character, call towupper/towlower on it, and then append it to a new string. But that's a really slow way to do it, when I suspect that UTF-8 might well be designed so that it can be done much faster than that. The glib reference is disheartening though since it says:

The exact manner that this is done depends on the current locale, and may result in the number of characters in the string increasing. (For instance, the German ess-zet will be changed to SS.)

For now, I think I'm just going to keep the TransformedString approach, which is inefficient, but works.

[StefanKarpinski]

The German ß is an excellent example — although not for the reasons glib gives. It shouldn't be capitalized as two letters, but rather as a different unicode character that has one more byte in UTF-8:

julia> length("\u00DF")
2

julia> length("\u1E9E")
3

That makes it really hard to write a fast, single-pass, in-place uppercasing function. We can still get most of the benefit by having something fast for the ASCII string case though.

[StefanKarpinski]

Commit 3f51323 implements fast, copying ucfirst, lcfirst, uc, and lc for ASCIIString objects. UTF8String objects still use the slow TransformedString approach, but for now, that's fine.

[StefanKarpinski]

Closed by 7010db8.