Re: Python pretty-printers and non-ASCII strings do not play well together :-(

[Paul Pluzhnikov <ppluzhnikov at google dot com>]

On Tue, Nov 4, 2008 at 4:58 PM, Tom Tromey <tromey@redhat.com> wrote:

> Tom> What should happen here, though?  The string contains invalid
> Tom> characters for its declared (via set target-charset) encoding.
>
> Paul> As an end-user, I would expect something like
> Paul>   $2 = <"\xef\xcd\xab">
>
> It occurs to me I am not completely certain where this error
> originates.  My theory is that it is the call to PyUnicode_Decode in
> valpy_str.

The 'PyUnicode_Decode()' returns a PyObject, for which
PyUnicode_AsEncodedString() returns NULL.

Here is the trace of this happening:

Breakpoint 1, valpy_str (self=0x2aaaaaae7250) at
../../gdb/python/python-value.c:246
246       char *s = NULL;
(top) n
253       stb = mem_fileopen ();
(top)
254       old_chain = make_cleanup_ui_file_delete (stb);
(top)
256       TRY_CATCH (except, RETURN_MASK_ALL)
(top)
258           common_val_print (((value_object *) self)->value, stb, 0, 0, 0,
(top)
260           s = ui_file_xstrdup (stb, &dummy);
(top)
256       TRY_CATCH (except, RETURN_MASK_ALL)
(top) p s
$4 = 0xb04c90 "\"ïÍ\""
(top) n
262       GDB_PY_HANDLE_EXCEPTION (except);
(top)
264       do_cleanups (old_chain);
(top)
266       result = PyUnicode_Decode (s, strlen (s), host_charset (), NULL);
(top)
267       xfree (s);
(top) p result
$5 = (PyObject *) 0x2aaaaab71a80
(top) n
269       return result;
(top)
270     }
(top)

### Now return into Python interpreter ###

PyObject_Str (v=<value optimized out>) at ../../Objects/object.c:361
361             if (res == NULL)
(top)
360             res = (*v->ob_type->tp_str)(v);
(top)
361             if (res == NULL)
(top) p res
$6 = (PyObject *) 0x2aaaaab71a80
(top) n
364             if (PyUnicode_Check(res)) {
(top)
366                     str = PyUnicode_AsEncodedString(res, NULL, NULL);
(top)
367                     Py_DECREF(res);
(top) p str
$7 = (PyObject *) 0x0

> If so, then we aren't seeing a value representation problem, which is
> what I was worried about.  Instead, I think common_val_print is
> emitting a string which is not actually valid according to
> host_charset.  That seems wrong.
>
> We could work around this in valpy_str, I suppose.  But I'm curious to
> know why this is happening -- why isn't common_val_print printing the
> escape sequences itself?

I don't see any escape sequences here.
Note that 'raw' GDB doesn't print any escape sequences either,
just raw contents of the buffer.

> My guess is that the target and host charsets are the same, and
> charset.c is passing character through without checking them for
> validity.  I didn't debug it, but when I set host-charset to ASCII (my
> target-charset is ISO-8859-1), I do see the escapes.
>
> Every time I look at this stuff I'm reminded that the gdb charset code
> could use a good scrubbing.  For example, the default host charset
> ought to come from the locale settings.  I have a patch to implement
> this, but there's no point submitting it since it breaks gdb on
> typical Linux systems -- most people use UTF-8 locales, but gdb
> doesn't handle UTF-8.
>
> Maybe we should just install a smart Python printer for 'char *' ;-)
>
> Paul> What are some of the good Python references?
> Tom> http://www.python.org/doc/2.5.2/api/api.html
>
> Paul> Yes, I've seen the above, but it didn't have the answers I was
> Paul> looking for :(
>
> What do you want to know?  Both Thiago and I have worked in this area,
> maybe one of us knows.

How to turn raw buffer contents with unprintable characters into something
which will print as "\xef\xcd\xab" :)

Or "what PyUnicode_AsEncodedString() is actually supposed to do?"

-- 
Paul Pluzhnikov