This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
Re: Consensus: Locale format may change from release to release?
- From: Allan McRae <allan at archlinux dot org>
- To: Carlos O'Donell <carlos at redhat dot com>, GNU C Library <libc-alpha at sourceware dot org>
- Date: Wed, 14 Oct 2015 00:41:36 +1000
- Subject: Re: Consensus: Locale format may change from release to release?
- Authentication-results: sourceware.org; auth=none
- References: <56185EE7 dot 5040106 at redhat dot com> <561B198D dot 4090800 at archlinux dot org> <561D0F2F dot 7070303 at redhat dot com>
On 14/10/15 00:03, Carlos O'Donell wrote:
> On 10/11/2015 10:23 PM, Allan McRae wrote:
>> On 10/10/15 10:42, Carlos O'Donell wrote:
>>> I've backported the fix for bug 18589 to 2.22, which is
>>> the only public release with the defect that causes strcoll_l
>>> to incorrectly sort digraphs.
>>>
>>> Those of you with distributions based on 2.22 may wish to rebase
>>> to get these changes. They will cause problems with sorting and
>>> are relatively serious.
>>>
>>
>>
>> Fun story...
>>
>> [2015-10-12 09:47] [ALPM] upgraded glibc (2.22-3 -> 2.22-4)
>> [2015-10-12 09:47] [ALPM-SCRIPTLET] bash: loadlocale.c:130:
>> _nl_intern_locale_data: Assertion `cnt < (sizeof
>> (_nl_value_type_LC_COLLATE) / sizeof (_nl_value_type_LC_COLLATE[0]))'
>> failed.
>
> You have loaded a locale onto your system without recompiling it for
> the running glibc. This is a failure in your upgrade process. Though
> we don't make this easy for user installed locales (which are static).
>
> In general no new processes should be started during the glibc upgrade
> until after all libraries have been replaced as a single atomic unit
> and all dependent data upgraded as single atomic unit.
>
> We don't do this in Fedora. There is a race window where the libraries
> are updated one by one (as the writes hit disk), and then the locale
> archive is processed entirely and 100% replaced (along with the C.utf8
> binary locale). During this race window applications may fail to start
> with library dependency issues and locale data format issues. Distributions
> should fix this, but they haven't.
>
>> And that was given every time I tried running anything. It is fun to
>> watch your system slowly die while trying to figure a way to remove that
>> update from the repos!
>
> This is a failure in your upgrade process, it's a failure in the expectations
> that we as a project set with our users, and a failure in our own code for
> locale handling.
>
> I suggest we make a "Best Practices for upgrading glibc" document that
> starts like this:
>
> (1) No new processes should be started during the glibc upgrade
> until after all libraries have been replaced as a single atomic
> unit and all dependent data upgraded as single atomic unit.
>
> (2) Locales should be recompiled for the glibc that is about to run.
> Any old compiled locales cannot be considered safe to load with
> the new glibc. There is no guarantee, even between point releases
> that the binary format remains compatible. This includes recompiling
> the locale-archive and any other binary locales.
>
> (3) The project assures that invalid locales will be detected to the best
> ability of the project, specifically when the project itself changes
> the binary on-disk format and such old locales which can't be loaded
> will be ignored or if specifically requested, errors returned, and the
> fallback locales used.
>
This sounds good to me.
>> The only changes between our 2.22-3 and 2.22-4 builds were these patches
>> you committed. It seems they introduce incompatibilities to locale
>> files generated with builds from earlier in the 2.22 release branch? I
>> was using a relatively benign locale (en_AU.UTF8).
>
> They do introduce a binary change in the binary compiled locales, and
> as a distribution provider you always have to recompile locales for any
> glibc upgrade even point releases.
I had (incorrectly) assumed that the recompile of locales was only
needed for updates between releases (e.g. 2.21 -> 2.22), and my
packaging reflected that. So a change within a release branch was a
surprise. Now the locales will be recompiled every glibc update in Arch.
> If we want to change this policy, which has been true since binary locales
> were started, we should discuss this in a new thread.
>
> The bug fixed, bug 18589, turns out to be a serious issues with sorting
> languages that digraphs. We could have left the binary format the same
> and still fixed the bug, but I did not consider that distributions were
> not recompiling the locales for the upgraded glibc.
>
> In Fedora we recompile every locale after every glibc build in order to
> ensure the right and working locales are installed after every upgrade.
> We also use %_install_lang to allow users to minimize the set of locales
> that get installed.
Slightly off-topic... I have noticed there is great variation in the
handling of locales across distributions. Arch uses a locale-gen script
that compiles locales listed in /etc/locale.gen. I believe this was
obtained from Debian at some stage. It is something that I think would
be nice to standardize more across distros.
> Lastly, to treat our users better we should fail gracefully. I'll backport
> Ludovic's patches to 2.22 when we get them sorted out.
>
Great!
Allan