Winpdb - A Platform Independent Python Debugger Threads
Unique Little Beings
While few python debuggers support threading, Winpdb may be the first Python debugger to do it right. Winpdb uses a novel approach to handling threads in the context of a debugger. Python threads are unique little beings. Unlike C++, you can’t always break into them (make them stop), since they are not always doing Python code, and may actually be blocked indefinitely in some C++ code. And yet, even more peculiar is the fact that a Python session may exist without any threads of execution at all, for example, think of the python interactive interpreter.
Breaking Into the Debugger
In debugger lingo “breaking into the debugger” means requesting the debuggee to pause for our inspection. In Winpdb this is nothing more than a polite request. Individual threads will break at their leisure, and until they do their state is reported as running. The cool thing about the Winpdb model is that we can still control the debuggee in this half broken state as if it was completely broken.
The second scenario, in which no threads exist at all when the break is requested, is only relevant to embedded debugging. In that case we can do very little until the first thread shows up on the radar and the debugger finally really breaks.
Threads of the thread module
There are three kinds of threads in Python. The main thread, threads created through the threading module, and threads created via the thread module. The first two types of threads are traced by Winpdb automatically. However threads created via the threads module are born invisible to Winpdb. To make Winpdb trace a thread created with the thread module, add the following line to the thread’s function:
rpdb2.settrace()
Again, this line is only needed for threads created with thread.start_new_thread() and is ignored for other kind of threads.
Console Commands:
t – List active threads or switch to a particular thread.
Thanks for a nice debugger.
We are using Python embedded in a C application and we use C threads not Python threads (we don’t use the thread or threading modules). Instead the multi-threaded C can call into Python code and Python code can yield. Threading is non-preemptive.
When I first tried debugging this with Winpdb, it worked functionally OK except that Winpdb could not list the threads or switch between them. It was just able to see the context of the thread currently waiting at a breakpoint.
After a bit of hacking I was able to improve the situation. I replaced the get_ident() method of Python’s thread module (also the reference to it in threading) with a new method that detects my C -created threads and returns their identifier. I’ve included the function below. I also replaced threading._DummyThread in order to provide the right thread names to Winpdb.
My question is whether this is good practice or a nasty hack. Are you planning to provide more support for this sort of thing in the future? Is it likely that future versions of Winpdb will break my hack?
Many thanks.
import rpdb2
# set of Python threads which are base-threads for C thread creation
# filled in during thread creation
# (this enables us to avoid changing the ids of threads used by Winpdb)
sc_elab_threads = set()
def get_ident():
“replacement function for thread.get_ident() which is gs-aware”
tmp = orig_get_ident()
if tmp in sc_elab_threads:
# this gets the ID of the C thread
return gs.gsp_sc_get_curr_process_handle()
else:
return tmp
# replace things in thread and threading modules
# note that threading tries to keep a reference to original thread.get_ident()
orig_get_ident = rpdb2.thread.get_ident
rpdb2.thread.get_ident = get_ident
rpdb2.threading._get_ident = get_ident
I just re-read my comment and it sounds like a “please debug my personal problem” text.
Sorry for that. The idea was to discuss more generally Winpdb’s support for other threading possibilities than only “thread” and “threading”. Using my project only as an example.
It would be great if there were an “official” way to instrument a thread package to make it visible to Winpdb, or maybe better, to derive something from Winpdb that understands my threads.
Hi James,
I do not expect to break your modification in the near future.
What is the situation now?
Can Winpdb switch between your C threads?
Do you conform to the following documentation regarding C threads?
http://docs.python.org/c-api/init.html
“It is important to note that when threads are created from C, they don’t have the global interpreter lock, nor is there a thread state data structure for them. Such threads must bootstrap themselves into existence, by first creating a thread state data structure, then acquiring the lock, and finally storing their thread state pointer, before they can start using the Python/C API. When they are done, they should reset the thread state pointer, release the lock, and finally free their thread state data structure.”
If so, once you create the thread state, is it still missing a Python thread identifier?
Hi Nir, thanks for repl.
Yes, our C code does create a thread state structure for each thread and does use the Python GIL properly. But Winpdb uses thread.get_ident() to detect, display and switch between threads. And thread.get_ident() returns the same value for all of my C threads, even though they have separate thread states. So without my hack, Winpdb will only debug the first thread in which I have called rpdb2.settrace(). It ignores the others. Apart from that limitation, there is no functional bug.
If you are interested in a “proper” way of doing this, I would recommend adding a named parameter to start_embedded_debbuger(). For example something like:
g_thread_get_ident = thread.get_ident def start_embedded_debugger(..., get_ident = thread.get_ident): global g_thread_get_ident g_thread_get_ident = get_ident ...Then replace all (~10) calls to thread.get_ident with g_thread_get_ident. If this done properly and well tested, it may be useful as a patch to Winpdb.