Re: [RFC] higher level tool for generating locale data needed

["Carlos O'Donell" <carlos at redhat dot com>]

On 10/14/2015 10:44 AM, Mike Frysinger wrote:
> localedef is a fine tool -- it generates a single locale with the requested
> parameters and that's it.  but in practice, distro maintainers need something
> higher level: they need to generate all the locales that the system supports.
> how do they do that today ?  lots-o-pain:
> - Debian: sets of wrapper scripts including one called `locale-gen` that reads
>   /etc/locale.gen and then takes care of generating all the locales listed there
>   via localedef; if no locales are listed, manually grub around installed glibc
>   files and parse their content/meaning to come up with the list to generate
> - Ubuntu: same deal pretty much as Debian
> - Gentoo: same high-level concept as Debian (script+config), but more user
>   friendly interface and runs in parallel
> - Fedora/RedHat: runs localedef by hand in rpm spec file
> - Arch Linux: same deal as Debian, but diff scripts
> i assume other distros fall into similar buckets.
> 
> can we standardize this in glibc itself ?  either in a new program (let's call
> it "locale-gen" since distros that support it have standardized on that), or in
> localdef itself.

We can and we should.

> requirements:
> - generate in parallel (generating 400+ locales in serial is sloooow)
> - respect a config file (let's just use /etc/locale.gen as it too is standard)
> - option to automatically generate all available locales (in case config is 
>   empty)
> - option to generate locales with the user-friendly name
>   e.g. this is good: locale-gen en_US.UTF-8
>   e.g. this is bad: localedef -i en_US -f UTF-8 en_US.UTF-8
> 
> nice-to-have:
> - option to incrementally sync locale state to a config by either deleting or
>   adding locales

This is a good start.

The only problem I see is that your mixing two things:

(a) Maintaining local state about which locales are selected by the user.

    (a.1) /etc/locale.gen is a local user choice saved to a data store (file)

    (a.2) stat synchronization.

with

(b) The process of generate locales.

To give you some context around Fedora and Red Hat.

Fedora and Red Hat do not wish to rebuild the locale on the remote side.
We want to send pre-built pre-tested locales to the installed system.
This means that the above mechanism needs to express in some way that there
are two parts of the equation: generation, and update.

We know generation and update are important because after every glibc update
we need to generate (or skip this step in Fedora) and upgrade (all distros do
this).

My only suggestion is that you split the discussion into two parts and try
to address both aspects separately, this would allow Fedora to skip the generation
discussions, but help out with the update/upgrade process.

Lastly, consider that a distribution may not ship locale source nor charmap
source, therefore (b) should document explicitly what it needs to operate,
and (a) should certainly not depend on any sources.

Cheers,
Carlos.