Re: [1.7] Invalid UTF8 while creating a file -> cannot delete?

[Andy Koppe <andy dot koppe at gmail dot com>]

2009/9/21 Corinna Vinschen:
>> % cat t.c
>> int main() {
>> Â Â fopen("a-\xF6\xE4\xFC\xDF", "w"); //ISO-8859-1
>> Â Â fopen("b-\xF6\xE4\xFC\xDFz", "w");
>> Â Â fopen("c-\xF6\xE4\xFC\xDFzz", "w");
>> Â Â fopen("d-\xF6\xE4\xFC\xDFzzz", "w");
>> Â Â fopen("e-\xF6\xE4\xFC\xDF\xF6\xE4\xFC\xDF", "w");
>> Â Â return 0;
>> }
>
> Ok, I see what happens. ÂThe problem is that the mechanism which is
> supposed to handle invalid multibyte sequences handles the first such
> byte, but misses to reset the multibyte shift state after the byte has
> been handled. ÂBasically, resetting the shift state after such a
> sequence has been encountered fixes that problem.

Great!


> Unfortunately this is only the first half of a solution. ÂThis is what
> `ls' prints after running t:
>
> Â$ ls -l --show-control-chars
> Âtotal 21
> Â-rw-r--r-- 1 corinna vinschen   0 Sep 21 17:35 a-ÃÃÃÃ
> Â-rw-r--r-- 1 corinna vinschen   0 Sep 21 17:35 c-ÃÃÃÃzz
> Â-rw-r--r-- 1 corinna vinschen   0 Sep 21 17:35 d-ÃÃÃÃzzz
> Â-rw-r--r-- 1 corinna vinschen   0 Sep 21 17:35 e-ÃÃÃÃÃÃÃÃ
>
> But this is what ls prints when setting $LANG to something "non-C":
>
> Â$ setenv LANG en   Â(implies codepage 1252)
> Â$ ls -l --show-control-chars
> Âls: cannot access a-ÃÃÃÃ: No such file or directory
> Âls: cannot access c-ÃÃÃÃzz: No such file or directory
> Âls: cannot access d-ÃÃÃÃzzz: No such file or directory
> Âls: cannot access e-ÃÃÃÃÃÃÃÃ: No such file or directory
> Âtotal 21
> Â-????????? ? ? Â Â Â ? Â Â Â Â Â Â? Â Â Â Â Â Â Â Â? a-ÃÃÃÃ
> Â-????????? ? ? Â Â Â ? Â Â Â Â Â Â? Â Â Â Â Â Â Â Â? c-ÃÃÃÃzz
> Â-????????? ? ? Â Â Â ? Â Â Â Â Â Â? Â Â Â Â Â Â Â Â? d-ÃÃÃÃzzz
> Â-????????? ? ? Â Â Â ? Â Â Â Â Â Â? Â Â Â Â Â Â Â Â? e-ÃÃÃÃÃÃÃÃ

Btw, the same thing will happen with en.C-ISO-8859-1 or C.ASCII too.


> As you might know, invalid bytes >= 0x80 are translated to UTF-16 by
> transposing them into the 0xdc00 - 0xdcff range by just or'ing 0xdc00.
> The problem now is that readdir() will return the transposed characters
> as if they are the original characters.

Yep, that's where the bug is. Those 0xDC?? words represent invalid
UTF-8 bytes. They do not represent CP1252 or ISO-8859-1 characters.

Therefore, when converting a UTF-16 Windows filename to the current
charset, 0xDC?? words should be treated like any other UTF-16 word
that can't be represented in the current charset: it should be encoded
as a ^N sequence.


> ls uses some mbtowc function
> to create a valid widechar string, and then uses the resulting widechar
> string in some wctomb function to call stat().

It's not 'ls' that does that conversion. On the POSIX side, filenames
are simply sequences of bytes, hence 'ls' would be very wrong to do
any conversion between readdir() and stat().

No, it's stat() itself converting the CP1252 sequence "a-ÃÃÃÃ" to
UTF-16, which yields L"a-ÃÃÃÃ". This does not contain the 0xDC??
codepoints that the actual filename contained, hence stat() fails.


> So it looks like the current mechanism to handle invalid multibyte
> sequences is too complicated for us. ÂAs far as I can see, it would be
> much simpler and less error prone to translate the invalid bytes simply
> to the equivalent UTF-16 value. ÂThat creates filenames with UTF-16
> values from the ISO-8859-1 range.

This won't work correctly, because different POSIX filenames will map
to the same Windows filename. For example, the filenames "\xC3\xA4"
(valid UTF-8 for a-umlaut) and "\xC4" (invalid UTF-8 sequence that
represents a-umlaut in 8859-1), will both map to Windows filename
"U+00C4", i.e a-umlaut in UTF-16. Furthermore, after creating a file
called "\xC4", a readdir() would show that file as "\xC3\xA4".

Note also that invalid UTF-8 sequences would be much less of an issue
if the C locale didn't mix UTF-8 filenames with a ISO-8859-1 console.
They'd still occur e.g. when unpacking a tarball with ISO-encoded
filenames while a UTF-8 locale is active. However, that sort of
situation is not handled well on Linux either.

Regards,
Andy

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple