This is the mail archive of the
cygwin-talk
mailing list for the cygwin project.
internationalized text processing
- From: "Nellis, Kenneth" <Kenneth dot Nellis at acs-inc dot com>
- To: <cygwin-talk at cygwin dot com>
- Date: Mon, 23 May 2011 08:44:40 -0500
- Subject: internationalized text processing
- Reply-to: The Vulgar and Unprofessional Cygwin-Talk List <cygwin-talk at cygwin dot com>
Never posted here, but somewhat off-topic for the general Cygwin list...
I've been frustrated at the lack of guidance I've found online regarding
how to build an internationalized text-based Unix application, in particular
with reference to handling different character sets. For example, "man"
recognizes the character set specified in locale settings and uses Unicode
punctuation appropriately. OTOH, the cygutils "ascii -e" utility does not
recognize that my LANG specifies UTF-8 and outputs garbage for the
extended half. Should this be considered a bug?
Closer to a real application, how should a program that reads, processes,
and outputs text detect the character set so that it can process the text
correctly? In the absence of a BOM in the input file, should the program
blindly presume the locale-specified character set or analyze the file to
guess it? And what advice is there for guessing? And, once detected, what
takes precedence for output: the input file's character set or the locale
setting?
I presume these issues are either settled or being debated somewhere.
--Ken Nellis