Log in

No account? Create an account
20 November 2009 @ 07:55 pm
2to3c: an implementation of Python's 2to3 for C code  
I'm hoping that we'll package python 3 versions of as many modules as possible in Fedora 13, so the easier it is to port them, the better.

To that end, I've written a tool to help people port their C python extensions from Python 2 to Python 3.

It uses the Coccinelle tool to apply a series of "semantic patches" to .c files. I also had to code one of the refactorings in python with regular expressions (due to the need to manipulate preprocessor macros containing commas).

Sample session, running on a tarball of dbus-python:
[david@brick 2to3]$ ./2to3c --help
Usage: 2to3c [options] filenames...

  -h, --help   show this help message and exit
  -w, --write  Write back modified files
[david@brick 2to3]$ ./2to3c ../../python3/packaging/modules/by-hand/dbus-python/devel/dbus-python-0.83.0/_dbus_bindings/*.c > dbus-python.patch 

[david@brick 2to3]$ diffstat dbus-python.patch
 abstract.c       |   28 ++----
 bus.c            |    4 
 bytes.c          |   16 +--
 conn.c           |    7 -
 containers.c     |   21 ++--
 float.c          |    6 -
 generic.c        |    4 
 int.c            |   31 ++-----
 libdbusconn.c    |    5 -
 mainloop.c       |    3 
 message-append.c |    4 
 message.c        |   17 +--
 module.c         |  243 ++++++++++++++++++++++++++++++++++++++++++++-----------
 pending-call.c   |    3 
 server.c         |    7 -
 signature.c      |    6 -
 string.c         |    9 --
 17 files changed, 267 insertions(+), 147 deletions(-)

[david@brick 2to3]$ head -n 30 dbus-python.patch
--- ../../python3/packaging/modules/by-hand/dbus-python/devel/dbus-python-0.83.0/_dbus_bindings/abstract.c.orig 
+++ ../../python3/packaging/modules/by-hand/dbus-python/devel/dbus-python-0.83.0/_dbus_bindings/abstract.c 
@@ -54,7 +54,7 @@
     if (!vl_obj)
         return 0;
-    return PyInt_AsLong(vl_obj);
+    return PyLong_AsLong(vl_obj);
@@ -76,7 +76,7 @@
     else {
-        PyObject *vl_obj = PyInt_FromLong(variant_level);
+        PyObject *vl_obj = PyLong_FromLong(variant_level);
         if (!vl_obj) {
             return FALSE;
@@ -127,7 +127,7 @@
     if (!value)
-        return PyInt_FromLong(0);
+        return PyLong_FromLong(0);
     return value;

You can see the full patch it generated here: http://dmalcolm.fedorapeople.org/dbus-python.patch

It hasn't done all of the work, there are some places involving the preprocessor where it didn't quite generate correct code, and there are some remaining issues - for example, a human is going to have to decide whether the strings are bytes or unicode.

However, I think this ought to save a lot of time: it takes care of a lot of the tedious parts of such patches.

The public git repo can be seen here:

You should be able to download it by cloning it thus:
git clone git://fedorapeople.org/home/fedora/dmalcolm/public_git/2to3c.git

Patches most welcome! (send them to dmalcolm@redhat.com) I intend to license this under LGPLv2.1, but am happy to relicense as the upstream Python community see fit.
ex_cgwalter on November 21st, 2009 01:28 am (UTC)
Does this generate code like #ifdef PYTHON3, or do maintainers need to fork the code?
ex_cgwalter on November 21st, 2009 01:28 am (UTC)
Forgot to say, cool stuff =)
dmalcolm on November 21st, 2009 01:38 pm (UTC)
I've tried to generate code that will compile against both 2 and 3. In some places it inserts stuff like this:
but I'm having some issues with spatch's handling of preprocessor directives. One idea I had is to make them look like real C code fragments:
...then postprocess the output from spatch to turn them back into preprocessor hooks.

I think it's easier to support both 2 and 3 from a single source tree. We managed to do this for rpm's Python bindings. But I fear that for some modules it's going to be too much of a stretch.

In any case, hopefully this tool makes it easier.