You are viewing dmalcolm

 
 
13 February 2010 @ 01:02 am
Debugging the innards of Python... with Python  
I've been trying a new approach for debugging the internals of Python.

One of the strengths of the C implementation of Python ("CPython") is how easy it is to wrap libraries written in C/C++ so that they're callable from Python code.

The drawback of this is that you're running "native" machine code, and that code can crash the python process, or worse, corrupt the internals of the python process so that the crash seems to be coming from somewhere else.

Time to break out the debugger...

Unfortunately, practically everything inside CPython is a pointer to a PyObject structure, with some additional type-specific data lurking after it in memory. A typical debugger by default just shows you the addresses of these structures, or shows you the two values they contain ("ob_refcnt" and "ob_type", which aren't necessarily enlightening).

I typically use the "gdb" debugger. Python has long had a file named "gdbinit" which contains various hooks to make it easier to debug CPython within gdb (In Fedora and RHEL that file can be found in our "python-devel" packages).

If you copy this file to ~/.gdbinit and restart gdb you can then use "pyo" to print a more human-readable representation of a PyObject*, and use "pyframe" and other commands to figure out where the currently selected thread of the process is in Python code.

This approach has some drawbacks:
  • it isn't very robust - it effectively injects calls to the repr() function into the process being debugged (the so-called "inferior process" in debugger parlance). If the data in the inferior process is corrupt, attempting to print it can lead to another segmentation fault ("SIGSEGV") within that process inside the implementation of "repr". For example, if the allocator of the inferior process has become corrupt, then Python won't be able to create the string representation of the data without a crash (be it in Python's memory arena code or the underlying heap), and you've got another SIGSEGV. If one of the objects inside a list has been splatted with 0xDEADBEEF garbage, attempting to print the repr() object will crash, and you won't even know that you had a list.
  • you have to go into gdb manually and run these commands by hand. Automated crash reports can't get at the "real" data, most significantly: what lines of Python code were being executed when this crash happened?
  • as well as being a manual task, it's hard to do this correctly; any mistakes when doing this will typically cause a SIGSEGV in the inferior process
  • the script is written in gdb's own language and is thus hard to work with and extend
  • complicated intermeshing of C and Python (e.g. invocations of callbacks that then trigger other callbacks each time crossing a language boundary) can be hard to figure out


Is there a better way?

(...drum roll...)

As of gdb 7.0, it's now possible to write custom "pretty-printers" for data in gdb, using Python as the extension language. One of the reasons for this was to improve visualization of C++ data in the debugger, so that if you try to print a C++ std::string, rather than get this craziness:

(gdb) print str
$1 = {static npos = 4294967295,
_M_dataplus = {> = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0x804a014 "hello world"}}


you now get:

(gdb) print str
$1 = hello world


The above is thanks to python hooks running inside the gdb process, using a "gdb" module to walk the graph of gdb.Value and gdb.Type objects representing the state of the process being debugged.


What this is building up to is that I've written a similar set of Python hooks for CPython itself.

The code can be seen in this git repository. I've been going through the various types within CPython's internals, writing Python code to run inside the debugger. For each type I've written a Python class that runs inside the debugger process, and knows how to convert an object in the process being debugged into an object within the debugger, and can then print it.

To demonstrate, let's try breaking Python:

The real use case for this for me is when a library decides to do error handling using "assert", and takes down the whole process, or has a crasher bug. Rather than pick on a 3rd-party library, let's inject our own segfault. We can do this by abusing the "ctypes" module, by picking a random location in the process' address space and trying to read it as a string. We can do this as a one-liner, but let's set up some data so we can see what it looks like in the crash report:
>>> class Foo:
...     def bar(self):
...         from ctypes import string_at
...         string_at(0xDEADBEEF) # this code will (probably) cause Python to segfault
... 
>>> f = Foo()
>>>
>>> # Let's assign some data of various kinds to the instance:
>>> f.someattr = 42
>>> f.someotherattr = {'one':1, 'two':2L, 'three':[(), (None,), (None, None)]}
>>>
>>> # Now let's trigger the segfault
>>> f.bar()


At this point we've generated a segmentation fault inside Python.

Let's see the old behavior of a backtrace, using the "bt" command:

Program received signal SIGSEGV, Segmentation fault.
__strlen_sse2 () at ../sysdeps/i386/i686/multiarch/strlen.S:87
87 pcmpeqb (%esi), %xmm0
Current language: auto
The current source language is "auto; currently asm".
(gdb) bt
#0 __strlen_sse2 () at ../sysdeps/i386/i686/multiarch/strlen.S:87
#1 0x07113d30 in PyString_FromString (str=0xdeadbeef <Address 0xdeadbeef out of bounds>) at Objects/stringobject.c:116
#2 0x00167e18 in string_at (ptr=0xdeadbeef <Address 0xdeadbeef out of bounds>, size=-1) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/_ctypes.c:5348
#3 0x0018247f in ffi_call_SYSV () at src/x86/sysv.S:61
#4 0x001822b0 in ffi_call (cif=<value optimized out>, fn=<value optimized out>, rvalue=<value optimized out>, avalue=<value optimized out>) at src/x86/ffi.c:213
#5 0x00171315 in _call_function_pointer (pProc=0x167de0 <string_at>, argtuple=0xb7f3d02c, flags=4357, argtypes=0xb7f45a4c, restype=0x80f3dc4, checker=0x0) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/callproc.c:815
#6 _CallProc (pProc=0x167de0 <string_at>, argtuple=0xb7f3d02c, flags=4357, argtypes=0xb7f45a4c, restype=0x80f3dc4, checker=0x0) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/callproc.c:1162
#7 0x0016a6f2 in CFuncPtr_call (self=0xb7f9d5dc, inargs=0xb7f3d02c, kwds=0x0) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/_ctypes.c:3857
#8 0x070c478c in PyObject_Call (func=0xb7f9d5dc, arg=0xb7f3d02c, kw=0x0) at Objects/abstract.c:2492
#9 0x0716069c in do_call (f=0x80f37bc, throwflag=0) at Python/ceval.c:3917
#10 call_function (f=0x80f37bc, throwflag=0) at Python/ceval.c:3729
#11 PyEval_EvalFrameEx (f=0x80f37bc, throwflag=0) at Python/ceval.c:2389
#12 0x07162642 in PyEval_EvalCodeEx (co=0xb7f3bda0, globals=0xb7f3768c, locals=0x0, args=0x80ec788, argcount=1, kws=0x80ec78c, kwcount=0, defs=0xb7f45d78, defcount=1, closure=0x0) at Python/ceval.c:2968
#13 0x07160983 in fast_function (f=0x80ec644, throwflag=0) at Python/ceval.c:3802
#14 call_function (f=0x80ec644, throwflag=0) at Python/ceval.c:3727
#15 PyEval_EvalFrameEx (f=0x80ec644, throwflag=0) at Python/ceval.c:2389
#16 0x07161b79 in fast_function (f=0x80eb1cc, throwflag=0) at Python/ceval.c:3792
#17 call_function (f=0x80eb1cc, throwflag=0) at Python/ceval.c:3727
#18 PyEval_EvalFrameEx (f=0x80eb1cc, throwflag=0) at Python/ceval.c:2389
#19 0x07162642 in PyEval_EvalCodeEx (co=0xb7f2e578, globals=0xb7fc70b4, locals=0xb7fc70b4, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2968
#20 0x071627a3 in PyEval_EvalCode (co=0xb7f2e578, globals=0xb7fc70b4, locals=0xb7fc70b4) at Python/ceval.c:522
#21 0x0717d94b in run_mod (mod=<value optimized out>, filename=<value optimized out>, globals=0xb7fc70b4, locals=0xb7fc70b4, flags=0xbffff2fc, arena=0x80e8628) at Python/pythonrun.c:1335
#22 0x0717f4a6 in PyRun_InteractiveOneFlags (fp=0x5b5420, filename=0x71c3e7d "<stdin>", flags=0xbffff2fc) at Python/pythonrun.c:840
#23 0x0717f6ab in PyRun_InteractiveLoopFlags (fp=0x5b5420, filename=0x71c3e7d "<stdin>", flags=<value optimized out>) at Python/pythonrun.c:760
#24 0x0717f7eb in PyRun_AnyFileExFlags (fp=0x5b5420, filename=<value optimized out>, closeit=0, flags=0xbffff2fc) at Python/pythonrun.c:729
#25 0x0718c212 in Py_Main (argc=1, argv=0xbffff3f4) at Modules/main.c:599


Looking closely at the above, we can see that there's a problem inside ctypes, but it's hard to see what triggered the problem within the Python code.


Below is a screendump of what it looks like with my new debugging hooks (warning: some very long lines; hopefully this won't get too badly mangled)

I've highlighted some of it: we can see
  • the location we're at (file/line/function) within the Python code at each point in the call stack
  • the dictionary of globals
  • the arguments passed to functions
and so on - there's almost too much information. This should come in handy for nasty issues when the stack involves C code calling into Python and back again (e.g. when callbacks get invoked); should also be useful for debugging all threads at once in a multithreaded app.

There are probably many things that could be improved about this code, but I'm already finding it very useful, and I hope other people will too.

I've integrated this code into Fedora's Python 2 and Python 3 builds for the upcoming Fedora 13 release so that the hooks should get automatically loaded by the debugger. It should all Just Work - the crash-handling tool will fetch the debugging hooks and the bug report that's generated should contain all of this rich information on what's going on inside python - without the user needing to be a debugging guru.

(gdb) bt
#0  __strlen_sse2 () at ../sysdeps/i386/i686/multiarch/strlen.S:87
#1  0x07113d30 in PyString_FromString (str=0xdeadbeef <Address 0xdeadbeef out of bounds>) at Objects/stringobject.c:116
#2  0x00167e18 in string_at (ptr=0xdeadbeef <Address 0xdeadbeef out of bounds>, size=-1) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/_ctypes.c:5348
#3  0x0018247f in ffi_call_SYSV () at src/x86/sysv.S:61
#4  0x001822b0 in ffi_call (cif=<value optimized out>, fn=<value optimized out>, rvalue=<value optimized out>, avalue=<value optimized out>) at src/x86/ffi.c:213
#5  0x00171315 in _call_function_pointer (pProc=0x167de0 <string_at>, argtuple=(3735928559L, -1), flags=4357, argtypes=(<builtin_function_or_method at remote 0xb7f45d2c>, <builtin_function_or_method at remote 0xb7f45d4c>), restype=
    <_ctypes.SimpleType at remote 0x80f3dc4>, checker=<unknown at remote 0x0>) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/callproc.c:815
#6  _CallProc (pProc=0x167de0 <string_at>, argtuple=(3735928559L, -1), flags=4357, argtypes=(<builtin_function_or_method at remote 0xb7f45d2c>, <builtin_function_or_method at remote 0xb7f45d4c>), restype=
    <_ctypes.SimpleType at remote 0x80f3dc4>, checker=<unknown at remote 0x0>) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/callproc.c:1162
#7  0x0016a6f2 in CFuncPtr_call (self=0xb7f9d5dc, inargs=(3735928559L, -1), kwds=<unknown at remote 0x0>) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/_ctypes.c:3857
#8  0x070c478c in PyObject_Call (func=<CFunctionType at remote 0xb7f9d5dc>, arg=(3735928559L, -1), kw=<unknown at remote 0x0>) at Objects/abstract.c:2492
#9  0x0716069c in do_call (f=File /usr/lib/python2.6/ctypes/__init__.py, line 492, in string_at (ptr=3735928559L, size=-1), throwflag=0) at Python/ceval.c:3917
#10 call_function (f=File /usr/lib/python2.6/ctypes/__init__.py, line 492, in string_at (ptr=3735928559L, size=-1), throwflag=0) at Python/ceval.c:3729
#11 PyEval_EvalFrameEx (f=File /usr/lib/python2.6/ctypes/__init__.py, line 492, in string_at (ptr=3735928559L, size=-1), throwflag=0) at Python/ceval.c:2389
#12 0x07162642 in PyEval_EvalCodeEx (co=0xb7f3bda0, globals=
    {'Union': <_ctypes.UnionType at remote 0x17c120>, 'c_wchar': <_ctypes.SimpleType at remote 0x80fb32c>, 'c_bool': <_ctypes.SimpleType at remote 0x80fab54>, 'c_double': <_ctypes.SimpleType at remote 0x80f7a0c>, 'CFUNCTYPE': <function at remote 0xb7f3264c>, '__path__': ['/usr/lib/python2.6/ctypes'], 'byref': <builtin_function_or_method at remote 0xb7f40eec>, 'pointer': <builtin_function_or_method at remote 0xb7f40d6c>, 'alignment': <builtin_function_or_method at remote 0xb7f40eac>, '_memmove_addr': 4962832, 'c_longlong': <_ctypes.SimpleType at remote 0x80f8544>, 'c_short': <_ctypes.SimpleType at remote 0x80f407c>, 'get_errno': <builtin_function_or_method at remote 0xb7f39f4c>, '__file__': '/usr/lib/python2.6/ctypes/__init__.pyc', '_calcsize': <builtin_function_or_method at remote 0xb7f96bec>, 'c_ulong': <_ctypes.SimpleType at remote 0x80f5974>, 'c_int': <_ctypes.SimpleType at remote 0x80f5124>, 'c_int32': <_ctypes.SimpleType at remote 0x80f5124>, 'memmove': <CFunctionType at remote 0xb7f9d4a4>, '_sys': <module at remote 0xb7fa308c>, '_cast': <CFunctionType at remote 0xb7f9d574>, 'addressof': <builtin_function_or_method at remote 0xb7f40f0c>, 'ArgumentError': <type at remote 0x80f2fdc>, 'c_buffer': <function at remote 0xb7f32614>, 'c_longdouble': <_ctypes.SimpleType at remote 0x80f821c>, 'cdll': <LibraryLoader at remote 0xb7f459ac>, 'memset': <CFunctionType at remote 0xb7f9d50c>, 'string_at': <function at remote 0xb7f32a04>, 'sizeof': <builtin_function_or_method at remote 0xb7f40ecc>, '_FUNCFLAG_PYTHONAPI': 4, 'create_string_buffer': <function at remote 0xb7f325dc>, 'set_errno': <builtin_function_or_method at remote 0xb7f40d2c>, '_pointer_type_cache': {<_ctypes.SimpleType at remote 0x80f9ec4>: <_ctypes.PointerType at remote 0x80fb864>, <_ctypes.SimpleType at remote 0x80fb32c>: <_ctypes.PointerType at remote 0x80fb504>, <NoneType at remote 0x72061e0>: <_ctypes.SimpleType at remote 0x80fa6a4>}, '_Pointer': <_ctypes.PointerType at remote 0x17bd00>, 'create_unicode_buffer': <function at remote 0xb7f326bc>, 'c_long': <_ctypes.SimpleType at remote 0x80f5124>, 'c_char_p': <_ctypes.SimpleType at remote 0x80fa37c>, '__builtins__': {'bytearray': <type at remote 0x71fb540>, 'IndexError': <type at remote 0x71ff0e0>, 'all': <builtin_function_or_method at remote 0xb7fafccc>, 'help': <_Helper at remote 0xb7f814ec>, 'vars': <builtin_function_or_method at remote 0xb7fb280c>, 'SyntaxError': <type at remote 0x71fed60>, 'unicode': <type at remote 0x720e2c0>, 'sorted': <builtin_function_or_method at remote 0xb7fb274c>, 'isinstance': <builtin_function_or_method at remote 0xb7fb22cc>, 'copyright': <_Printer at remote 0xb7f81d6c>, 'NameError': <type at remote 0x71feac0>, 'BytesWarning': <type at remote 0x72006c0>, 'dict': <type at remote 0x7205960>, 'input': <builtin_function_or_method at remote 0xb7fb224c>, 'oct': <builtin_function_or_method at remote 0xb7fb246c>, 'bin': <builtin_function_or_method at remote 0xb7fafd8c>, 'SystemExit': <type at remote 0x71fe2e0>, 'StandardError': <type at remote 0x71fdf60>, 'format': <builtin_function_or_method at remote 0xb7fb20ac>, 'repr': <builtin_function_or_method at remote 0xb7fb268c>, 'UnicodeDecodeError': <type at remote 0x71ff540>, 'False': <bool at remote 0x71f9624>, 'RuntimeWarning': <type at remote 0x7200340>, 'bytes': <type at remote 0x7209c80>, 'iter': <builtin_function_or_method at remote 0xb7fb230c>, 'reload': <builtin_function_or_method at remote 0xb7fb264c>, 'Warning': <type at remote 0x71ffee0>, 'round': <builtin_function_or_method at remote 0xb7fb26cc>, 'dir': <builtin_function_or_method at remote 0xb7faff4c>, 'cmp': <builtin_function_or_method at remote 0xb7fafe4c>, 'set': <type at remote 0x7207000>, 'list': <type at remote 0x7204420>, 'reduce': <builtin_function_or_method at remote 0xb7fb260c>, 'intern': <builtin_function_or_method at remote 0xb7fb228c>, 'issubclass': <builtin_function_or_method at remote 0xb7fb22ec>, 'apply': <builtin_function_or_method at remote 0xb7fafd4c>, 'EOFError': <type at remote 0x71fe820>, 'locals': <builtin_function_or_method at remote 0xb7fb238c>, 'BufferError': <type at remote 0x71ffe00>, 'slice': <type at remote 0x7207900>, 'FloatingPointError': <type at remote 0x71ff8c0>, 'sum': <builtin_function_or_method at remote 0xb7fb278c>, 'buffer': <type at remote 0x71f9840>, 'getattr': <builtin_function_or_method at remote 0xb7fb20cc>, 'abs': <builtin_function_or_method at remote 0xb7fafc8c>, 'exit': <Quitter at remote 0xb7fd1d2c>, 'print': <builtin_function_or_method at remote 0xb7fb256c>, 'IndentationError': <type at remote 0x71fee40>, 'True': <bool at remote 0x71f9630>, 'FutureWarning': <type at remote 0x7200420>, 'ImportWarning': <type at remote 0x7200500>, 'None': <NoneType at remote 0x72061e0>, 'hash': <builtin_function_or_method at remote 0xb7fb218c>, 'len': <builtin_function_or_method at remote 0xb7fb234c>, 'credits': <_Printer at remote 0xb7f8156c>, 'frozenset': <type at remote 0x72070e0>, '__name__': '__builtin__', 'ord': <builtin_function_or_method at remote 0xb7fb24ec>, 'super': <type at remote 0x720ade0>, 'TypeError': <type at remote 0x71fe040>, 'license': <_Printer at remote 0xb7f8170c>, 'KeyboardInterrupt': <type at remote 0x71fe3c0>, 'UserWarning': <type at remote 0x71fffc0>, 'filter': <builtin_function_or_method at remote 0xb7fb206c>, 'range': <builtin_function_or_method at remote 0xb7fb25ac>, 'staticmethod': <type at remote 0x72035e0>, 'SystemError': <type at remote 0x71ffb60>, 'BaseException': <type at remote 0x7200a60>, 'pow': <builtin_function_or_method at remote 0xb7fb252c>, 'RuntimeError': <type at remote 0x71fe900>, 'float': <type at remote 0x72028a0>, 'GeneratorExit': <type at remote 0x71fe200>, 'StopIteration': <type at remote 0x71fe120>, 'globals': <builtin_function_or_method at remote 0xb7fb210c>, 'divmod': <builtin_function_or_method at remote 0xb7faff8c>, 'enumerate': <type at remote 0x71fdaa0>, 'Ellipsis': <ellipsis at remote 0x7207840>, 'LookupError': <type at remote 0x71ff000>, 'open': <builtin_function_or_method at remote 0xb7fb24ac>, 'quit': <Quitter at remote 0xb7fd120c>, 'basestring': <type at remote 0x7209ba0>, 'UnicodeError': <type at remote 0x71ff380>, 'zip': <builtin_function_or_method at remote 0xb7fb284c>, 'hex': <builtin_function_or_method at remote 0xb7fb21cc>, 'long': <type at remote 0x7204f60>, 'next': <builtin_function_or_method at remote 0xb7fb244c>, 'int': <type at remote 0x7203a40>, 'chr': <builtin_function_or_method at remote 0xb7fafe0c>, '__import__': <builtin_function_or_method at remote 0xb7fafc6c>, 'type': <type at remote 0x720ac20>, '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", 'Exception': <type at remote 0x71fde80>, 'tuple': <type at remote 0x720a6a0>, 'UnicodeTranslateError': <type at remote 0x71ff620>, 'reversed': <type at remote 0x71fdb80>, 'UnicodeEncodeError': <type at remote 0x71ff460>, 'IOError': <type at remote 0x71fe660>, 'hasattr': <builtin_function_or_method at remote 0xb7fb214c>, 'delattr': <builtin_function_or_method at remote 0xb7faff0c>, 'setattr': <builtin_function_or_method at remote 0xb7fb270c>, 'raw_input': <builtin_function_or_method at remote 0xb7fb25ec>, 'PendingDeprecationWarning': <type at remote 0x7200180>, 'compile': <builtin_function_or_method at remote 0xb7fafecc>, 'ArithmeticError': <type at remote 0x71ff7e0>, 'str': <type at remote 0x7209c80>, 'property': <type at remote 0x71fcec0>, 'MemoryError': <type at remote 0x71ffd20>, 'ImportError': <type at remote 0x71fe4a0>, 'xrange': <type at remote 0x7206640>, 'KeyError': <type at remote 0x71ff1c0>, 'coerce': <builtin_function_or_method at remote 0xb7fafe8c>, 'SyntaxWarning': <type at remote 0x7200260>, 'file': <type at remote 0x7201d80>, 'EnvironmentError': <type at remote 0x71fe580>, 'unichr': <builtin_function_or_method at remote 0xb7fb27cc>, 'id': <builtin_function_or_method at remote 0xb7fb220c>, 'OSError': <type at remote 0x71fe740>, 'DeprecationWarning': <type at remote 0x72000a0>, 'min': <builtin_function_or_method at remote 0xb7fb242c>, 'UnicodeWarning': <type at remote 0x72005e0>, 'execfile': <builtin_function_or_method at remote 0xb7fb202c>, '__package__': <NoneType at remote 0x72061e0>, 'complex': <type at remote 0x71fc840>, 'bool': <type at remote 0x71f9560>, 'ValueError': <type at remote 0x71ff2a0>, 'NotImplemented': <NotImplementedType at remote 0x72061e8>, 'map': <builtin_function_or_method at remote 0xb7fb23cc>, 'any': <builtin_function_or_method at remote 0xb7fafd0c>, 'max': <builtin_function_or_method at remote 0xb7fb240c>, 'object': <type at remote 0x720ad00>, 'TabError': <type at remote 0x71fef20>, 'callable': <builtin_function_or_method at remote 0xb7fafdcc>, 'ZeroDivisionError': <type at remote 0x71ffa80>, 'eval': <builtin_function_or_method at remote 0xb7faffcc>, '__debug__': <bool at remote 0x71f9630>, 'ReferenceError': <type at remote 0x71ffc40>, 'AssertionError': <type at remote 0x71ff700>, 'classmethod': <type at remote 0x7203500>, 'UnboundLocalError': <type at remote 0x71feba0>, 'NotImplementedError': <type at remote 0x71fe9e0>, 'AttributeError': <type at remote 0x71fec80>, 'OverflowError': <type at remote 0x71ff9a0>}, '_FUNCFLAG_USE_ERRNO': 8, '_memset_addr': 4962944, '_dlopen': <builtin_function_or_method at remote 0xb7f40e0c>, '__name__': 'ctypes', 'RTLD_LOCAL': 0, 'c_int16': <_ctypes.SimpleType at remote 0x80f407c>, '_SimpleCData': <_ctypes.SimpleType at remote 0x17bde0>, 'wstring_at': <function at remote 0xb7f32a3c>, 'c_void_p': <_ctypes.SimpleType at remote 0x80fa6a4>, 'set_conversion_mode': <builtin_function_or_method at remote 0xb7f40dec>, 'PyDLL': <type at remote 0x80fc3dc>, 'DEFAULT_MODE': 0, 'LittleEndianStructure': <_ctypes.StructType at remote 0x17c040>, 'c_uint64': <_ctypes.SimpleType at remote 0x80f8d54>, 'c_ulonglong': <_ctypes.SimpleType at remote 0x80f8d54>, '_FUNCFLAG_USE_LASTERROR': 16, '_cast_addr': 1490912, 'ARRAY': <function at remote 0xb7f3279c>, 'c_ushort': <_ctypes.SimpleType at remote 0x80f48ac>, '__doc__': 'create and manipulate C data types in Python', '_check_size': <function at remote 0xb7f32684>, 'CDLL': <type at remote 0x80fbeac>, '_wstring_at': <CFunctionType at remote 0xb7f9d644>, 'c_ubyte': <_ctypes.SimpleType at remote 0x80f9564>, 'RTLD_GLOBAL': 256, 'c_char': <_ctypes.SimpleType at remote 0x80f9ec4>, 'c_uint32': <_ctypes.SimpleType at remote 0x80f5974>, 'c_float': <_ctypes.SimpleType at remote 0x80f71fc>, 'SetPointerType': <function at remote 0xb7f32764>, 'resize': <builtin_function_or_method at remote 0xb7f40dcc>, '_c_functype_cache': {(<_ctypes.SimpleType at remote 0x80f5124>, (), 1): <_ctypes.CFuncPtrType at remote 0x8100b14>, (<_ctypes.SimpleType at remote 0x80fa6a4>, (<_ctypes.SimpleType at remote 0x80fa6a4>, <_ctypes.SimpleType at remote 0x80f5124>, <_ctypes.SimpleType at remote 0x80f5974>), 1): <_ctypes.CFuncPtrType at remote 0x80fd84c>}, '---Type <return> to continue, or q <return> to quit---
_os': <module at remote 0xb7fa314c>, '_wstring_at_addr': 1494192, 'cast': <function at remote 0xb7f329cc>, 'c_int8': <_ctypes.SimpleType at remote 0x80f9a14>, 'c_byte': <_ctypes.SimpleType at remote 0x80f9a14>, 'c_int64': <_ctypes.SimpleType at remote 0x80f8544>, 'c_voidp': <_ctypes.SimpleType at remote 0x80fa6a4>, '_string_at_addr': 1474016, '_FUNCFLAG_CDECL': 1, 'pythonapi': <PyDLL at remote 0xb7f45a0c>, 'PYFUNCTYPE': <function at remote 0xb7f327d4>, '_CFuncPtr': <_ctypes.CFuncPtrType at remote 0x17bb40>, '_endian': <module at remote 0xb7fa3944>, '__package__': 'ctypes', 'c_uint16': <_ctypes.SimpleType at remote 0x80f48ac>, 'BigEndianStructure': <_swapped_meta at remote 0x81006ec>, 'pydll': <LibraryLoader at remote 0xb7f459ec>, '__version__': '1.1.0', 'Structure': <_ctypes.StructType at remote 0x17c040>, 'c_uint': <_ctypes.SimpleType at remote 0x80f5974>, 'py_object': <_ctypes.SimpleType at remote 0x80f3dc4>, 'c_wchar_p': <_ctypes.SimpleType at remote 0x80fae7c>, '_string_at': <CFunctionType at remote 0xb7f9d5dc>, 'c_size_t': <_ctypes.SimpleType at remote 0x80f5974>, 'c_uint8': <_ctypes.SimpleType at remote 0x80f9564>, 'LibraryLoader': <type at remote 0x80fc704>, 'Array': <_ctypes.ArrayType at remote 0x17bc20>, 'POINTER': <builtin_function_or_method at remote 0xb7f40d4c>}, locals=<unknown at remote 0x0>, args=0x80ec788, argcount=1, kws=0x80ec78c, kwcount=0, defs=0xb7f45d78, defcount=1, closure=
    <unknown at remote 0x0>) at Python/ceval.c:2968
#13 0x07160983 in fast_function (f=
    File <stdin>, line 2, in bar (self=<Foo({'someattr': 42, 'someotherattr': {'three': [(), (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, string_at=<function at remote 0xb7f32a04>), throwflag=0) at Python/ceval.c:3802
#14 call_function (f=
    File <stdin>, line 2, in bar (self=<Foo({'someattr': 42, 'someotherattr': {'three': [(), (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, string_at=<function at remote 0xb7f32a04>), throwflag=0) at Python/ceval.c:3727
#15 PyEval_EvalFrameEx (f=
    File <stdin>, line 2, in bar (self=<Foo({'someattr': 42, 'someotherattr': {'three': [(), (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, string_at=<function at remote 0xb7f32a04>), throwflag=0) at Python/ceval.c:2389
#16 0x07161b79 in fast_function (f=File <stdin>, line 1, in <module> (), throwflag=0) at Python/ceval.c:3792
#17 call_function (f=File <stdin>, line 1, in <module> (), throwflag=0) at Python/ceval.c:3727
#18 PyEval_EvalFrameEx (f=File <stdin>, line 1, in <module> (), throwflag=0) at Python/ceval.c:2389
#19 0x07162642 in PyEval_EvalCodeEx (co=0xb7f2e578, globals=
    {'f': <Foo({'someattr': 42, 'someotherattr': {'three': [(), (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, '__builtins__': <module at remote 0xb7fa3074>, '__package__': <NoneType at remote 0x72061e0>, '__name__': '__main__', 'Foo': <classobj at remote 0xb7f3817c>, '__doc__': <NoneType at remote 0x72061e0>}, locals=
    {'f': <Foo({'someattr': 42, 'someotherattr': {'three': [(), (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, '__builtins__': <module at remote 0xb7fa3074>, '__package__': <NoneType at remote 0x72061e0>, '__name__': '__main__', 'Foo': <classobj at remote 0xb7f3817c>, '__doc__': <NoneType at remote 0x72061e0>}, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, 
    closure=<unknown at remote 0x0>) at Python/ceval.c:2968
#20 0x071627a3 in PyEval_EvalCode (co=0xb7f2e578, globals=
    {'f': <Foo({'someattr': 42, 'someotherattr': {'three': [(), (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, '__builtins__': <module at remote 0xb7fa3074>, '__package__': <NoneType at remote 0x72061e0>, '__name__': '__main__', 'Foo': <classobj at remote 0xb7f3817c>, '__doc__': <NoneType at remote 0x72061e0>}, locals=
    {'f': <Foo({'someattr': 42, 'someotherattr': {'three': [(), (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, '__builtins__': <module at remote 0xb7fa3074>, '__package__': <NoneType at remote 0x72061e0>, '__name__': '__main__', 'Foo': <classobj at remote 0xb7f3817c>, '__doc__': <NoneType at remote 0x72061e0>}) at Python/ceval.c:522
#21 0x0717d94b in run_mod (mod=<value optimized out>, filename=<value optimized out>, globals=
    {'f': <Foo({'someattr': 42, 'someotherattr': {'three': [(), (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, '__builtins__': <module at remote 0xb7fa3074>, '__package__': <NoneType at remote 0x72061e0>, '__name__': '__main__', 'Foo': <classobj at remote 0xb7f3817c>, '__doc__': <NoneType at remote 0x72061e0>}, locals=
    {'f': <Foo({'someattr': 42, 'someotherattr': {'three': [(), (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, '__builtins__': <module at remote 0xb7fa3074>, '__package__': <NoneType at remote 0x72061e0>, '__name__': '__main__', 'Foo': <classobj at remote 0xb7f3817c>, '__doc__': <NoneType at remote 0x72061e0>}, flags=0xbffff2fc, arena=0x80e8628) at Python/pythonrun.c:1335
#22 0x0717f4a6 in PyRun_InteractiveOneFlags (fp=0x5b5420, filename=0x71c3e7d "<stdin>", flags=0xbffff2fc) at Python/pythonrun.c:840
#23 0x0717f6ab in PyRun_InteractiveLoopFlags (fp=0x5b5420, filename=0x71c3e7d "<stdin>", flags=<value optimized out>) at Python/pythonrun.c:760
#24 0x0717f7eb in PyRun_AnyFileExFlags (fp=0x5b5420, filename=<value optimized out>, closeit=0, flags=0xbffff2fc) at Python/pythonrun.c:729
#25 0x0718c212 in Py_Main (argc=1, argv=0xbffff3f4) at Modules/main.c:599
#26 0x080485c7 in main (argc=1, argv=0xbffff3f4) at Modules/python.c:23
 
 
 


Tags: ,
 
 
( 5 comments — Leave a comment )
alsurenalsuren on February 24th, 2010 05:38 pm (UTC)
Cool, but needed to hack around a few backtraces.
This is really cool, and is helping me debug some deadlocks I had in dbus-python in Debian/python2.5. I did notice a few backtraces from python within my C backtrace when I was debugging a core file, which were a bit annoying, so I hacked around them.

http://git.collabora.co.uk/?p=user/alsuren/gdbpython.git

They really are hacks at the moment, because I didn't actually read any of the gdb python docs before writing them (just read the backtraces and silenced them). Treat my patches more like bug reports for now.

I'll probably pull from your branch from time to time. Should I continue to submit bug reports here or is there a better place?

David.
dmalcolm on February 27th, 2010 06:43 pm (UTC)
Re: Cool, but needed to hack around a few backtraces.
Thanks!

Sorry for not replying sooner, I've been on the road at PyCon.

I've been hacking on the code quite a bit, and only just got a chance to push the changes to the public repo.

I'm afraid that the code has changed quite a bit, and I suspect the changes in your repo aren't going to apply cleanly. Sorry about that. (for example, I've eliminated the need for the "is_py3k" hack)

I've written a selftest suite for the code (see test_gdb.py in my repo), and I'm able to inject certain types of corruption into the inferior process, and assert that gdb prints something sane in the face of such abuse.

Hopefully I've fixed the exceptions you were seeing. If you still see any, I'm keen to fix them - is it possible for you to isolate the conditions that trigger exceptions, and come up with a reproducer?

dorontal on February 28th, 2010 04:44 pm (UTC)
Does it work with python release version
To my best recollection the gdb macros requires Python debug version (never tried it though).
dmalcolm on February 28th, 2010 09:43 pm (UTC)
Re: Does it work with python release version
You need the usual "DWARF" debug data (generated with the "-g" option to gcc) so that the debugger knows the in-memory layouts of structures and so on, but you _don't_ need the --with-pydebug stuff. I've enabled this for Fedora 13's standard build of Python 2 and Python 3, for instance.
dorontal on February 28th, 2010 04:44 pm (UTC)
Does it work with python release version?
To my best recollection the gdb macros requires Python debug version (never tried it though).
( 5 comments — Leave a comment )