Re: Should glibc provide a builtin C.UTF-8 locale?

[keld at keldix dot com]

On Wed, Feb 11, 2015 at 11:24:35AM -0500, Carlos O'Donell wrote:
> Is anyone opposed to having glibc contain a builtin C.UTF-8 locale?
> This locale would have the same rules as the C locale when set for
> LC_ALL.
> 
> The locale would provide sensible fallback for developers that need
> UTF-8 but until C.UTF-8 was provided, could not rely upon it.
> 
> My best guess is that it will take ~1.5MB of data to include the
> UTF-8 locale in the runtime. If you do it right this is shared
> for all processes, and give you, in this the 20th century, a fallback
> that is sensible for all developers of all languages.
> 
> We have had on-and-off requests for this for years as UTF-8 has become
> the defacto standard.
> 
> The most recent request is from the Python 3 folks who want to be able
> to assume there is some kind of UTF-8 support in the system regardless
> of the installed locales.
> 
> Is this the right way forward? Or should we tell the distributions
> that it is their responsibility to ship and always provide a C.UTF-8?

I think it is a good way forward. It should probably be the "i18n" locale
of ISO 30112 that is the base, the "i18n" locale is built directly on glibc data.

A lot of optimisation could be done on the data with two-level or more tables,
giving special data where the data is not well-formed for algoritmic 
handling, noting that some case mappings are not suited for algoritmic handling.
Some properties tables are well suited for combined bitmap handling and index handling.
Collating tables could possibly also be optimized by multilevel tables.

Also a pet idea of mine is to have compressed locales - that could significantly reduce
the disk footprint of a more complete locale database. Also good for message catalogues.

Best regards
keld