[Bug localedata/14094] Update locale data to Unicode 7.0.0

["maiku.fabian at gmail dot com" <sourceware-bugzilla at sourceware dot org>]

https://sourceware.org/bugzilla/show_bug.cgi?id=14094

--- Comment #19 from Mike FABIAN <maiku.fabian at gmail dot com> ---
I extended Pravinâs ctype-compatibility.py script to produce more
human readable output and added many extra tests.

Joseph Myers> * Ensure the character type data in
Joseph Myers>   localedata/charmaps/i18n can be properly reproduced from
Joseph Myers>   Unicode 5.0 data using gen-unicode-ctype.c, adapting
Joseph Myers>   gen-unicode-ctype.c as needed to replicate any changes
Joseph Myers>   that may have been made not using that program.

When using gen-unicode-ctype.c with UnicodeData.txt-5.0.0
to generate LC_CTYPE, the generated file lacks
many characters which apparently have been manually added
to glibcâs i18n file:

alpha: Missing 1238 characters of old ctype in new ctype 
blank: Missing 0 characters of old ctype in new ctype 
cntrl: Missing 0 characters of old ctype in new ctype 
combining: Missing 124 characters of old ctype in new ctype 
combining_level3: Missing 49 characters of old ctype in new ctype 
digit: Missing 0 characters of old ctype in new ctype 
graph: Missing 1571 characters of old ctype in new ctype 
lower: Missing 115 characters of old ctype in new ctype 
print: Missing 1571 characters of old ctype in new ctype 
punct: Missing 335 characters of old ctype in new ctype 
space: Missing 0 characters of old ctype in new ctype 
tolower: Missing 19 characters of old ctype in new ctype 
totitle: Missing 8 characters of old ctype in new ctype 
toupper: Missing 18 characters of old ctype in new ctype 
upper: Missing 100 characters of old ctype in new ctype 
xdigit: Missing 0 characters of old ctype in new ctype 

I.e. reproducing the localedata/charmaps/i18n character type data
from Unicode 5.0 data using gen-unicode-ctype.c does not work
well because glibcâs i18n file apparently has been edited
manually a lot already to include newer Unicode data.

Apparently quite a few mistake have been made by manually editing
the i18n file. For example, the report from ctype-compatibility.py
also produces for the old i18n file:

error: 0xa67f ê punct True: 0xa67f CYRILLIC PAYEROK. Not in Unicode 5.0.0. In
Unicode
            7.0.0. General category Lm (Letter
            modifier). DerivedCoreProperties.txt says it is
            âAlphabeticâ. Apparently added manually to punct by mistake in
            glibcâs old LC_CTYPE.
error: 0xa67f ê alpha False: 0xa67f CYRILLIC PAYEROK. Not in Unicode 5.0.0. In
Unicode
            7.0.0. General category Lm (Letter
            modifier). DerivedCoreProperties.txt says it is
            âAlphabeticâ. Apparently added manually to punct by mistake in
            glibcâs old LC_CTYPE.

Another example:

error: 0x9f4 à alpha True: 
            â09F4;BENGALI CURRENCY NUMERATOR ONE;No;0;L;;;;1/16;N;;;;;â
            â09F5;BENGALI CURRENCY NUMERATOR TWO;No;0;L;;;;1/8;N;;;;;â
            â09F6;BENGALI CURRENCY NUMERATOR THREE;No;0;L;;;;3/16;N;;;;;â
            â09F7;BENGALI CURRENCY NUMERATOR FOUR;No;0;L;;;;1/4;N;;;;;â
            â09F8;BENGALI CURRENCY NUMERATOR ONE LESS THAN THE
DENOMINATOR;No;0;L;;;;3/4;N;;;;;â
            â09F9;BENGALI CURRENCY DENOMINATOR SIXTEEN;No;0;L;;;;16;N;;;;;â
            â09FA;BENGALI ISSHAR;So;0;L;;;;;N;;;;;â
            According to DerivedCoreProperties.txt (7.0.0) these are *not*
            âAlphabeticâ.

So this has been mistakenly added to âalphaâ in the old i18n file
of glibc (but gen-unicode-ctype.c correctly puts in into âpunctâ,
i.e. this seems to be another mistake by manual editing).

Some of the errors reported by ctype-compatibility.py

error: 0x250 É lower False: Should be lower in Unicode 7.0.0 (was not lower in
            Unicode 5.0.0).

would be fixed by using gen-unicode-ctype.c with Unicode 7.0.0 input.

There are many more problems like this in the old i18n file,
my tests found 133 errors total:

------------------------------------------------------------
Old file = /local/mfabian/src/glibc/localedata/locales/i18n
Number of errors in old file = 133
------------------------------------------------------------

Iâll attach the full report.

-- 
You are receiving this mail because:
You are on the CC list for the bug.