Re: Python 3 and GDB

[<Paul_Koning at Dell dot com>]

On Aug 16, 2013, at 7:47 AM, Phil Muldoon <pmuldoon@redhat.com> wrote:

> 
> Paul,
> 
> I think you did the original conversion for GDB to compile with
> Python 3. It seems we still have some stuff in there that could fail
> (or, stuff that we introduced later, after your patch).  One example
> is "PyString_AsString".
> 
> So I was curious if you used any tools to highlight areas of concern
> with the Python 2.x -> Python 3 change.  I would be interested in
> automating this process, perhaps with the ARI tool, to ensure that all
> future commits comply with Python 2 and 3.

No, I did not use any conversion tools. Instead, it was a case of reading some docs, beating on the sources until they build correctly, and run the test suites until they pass.  It certainly would be excellent if an automated checker could be found.  I know of none, and it's not clear how to do a complete one.  A partial one might be possible.

Is this a case of incomplete testing?  I don't know if we clearly stated that proposed changes in the Python area have to be tested against Python 2 and Python 3 both.  Many problems will become obvious just from attempting a build, because missing API calls become compile errors.  The remaining details tend to show up in a run of the test suite.  (For that matter, test suite changes also require care -- there is work in those to handle the Python differences and additions to the test suite need to be bilingual, too.)  In fact, it would be good to call out specifically what versions are supposed to be tested.  The oldest 2.x we support plus 2.7, and 3.1 plus 3.latest is what I would suggest.

I found some very limited documentation on the API mapping between Python 2 and 3.  At the time I did the work, there were some serious gaps in that documentation.  The module initialization comes to mind; the Python 3 conditionals you find in the gdb Python initialization was the result of reverse engineering code in Python, not anything documented.  Another example is the s# argument parsing string, which was superseded in Python 3 by s* with a different API.

At this point, it seems that things have improved a little. http://wiki.python.org/moin/PortingExtensionModulesToPy3k has a bunch of useful documentation, plus pointers to some other documents. 

There are also some things to watch out for due to the changed to unified Unicode support in Python 3.3.  A bunch of API calls are deprecated because of that, and while they may still work (for suitable definition of "work"), the best answer is to avoid them right now.  The "What's new in Python 3.3" document describes this.

I don't remember if there is a document that describes how to write Python code that's acceptable in both versions.  Maybe there is now; if so that would be a useful thing to track down.  

At one point early on I saw a statement that implied "bilingual code" is possible but messy, so you might not want to do that.  I've since learned (partly through tweaking the test suite and the Python files that are part of gdb) that it isn't all that hard, at least in straightforward Python code like that.  Other projects have done the same; pycrypto is a notable example.  A few things are a bit messy because they are syntax differences, so you can't easily write simple conditional code for the two alternative.  One of those is the u prefix on strings (in Python 3.0 through 3.2 that is rejected); another is "except C as v" which Python 2 rejects. Most other things are simple version checks and creation of aliases for type or function names.

	paul