Writing a Custom Debugger on Windows

Recently I had to debug an intermittent access violation exception in one of my AutoCAD plug-ins. I needed to get the exception while the debugger was attached so I could break the process and analyze the state of memory before AutoCAD’s global exception handler got control. Unfortunately, sometimes it took several hundred runs before the exception occurred.

To make matters worse, AutoCAD was throwing exceptions on shutdown that had nothing to do with my code. If I started AutoCAD manually under the Visual Studio debugger, I had to dismiss those shutdown exceptions manually every time. I needed a way to script the task of starting AutoCAD under the debugger hundreds of times in a row until the exception finally occurred.

After some futile attempts to script the Visual Studio debugger, I decided to just write my own custom debugger. With my own custom debugger, I could easily write code to decide at runtime which exceptions to ignore and which were of interest. The exception I was interested in always occurred while accessing a memory location that ended in 0x30, so my custom debugger ignores access violations that don’t fit the pattern.

I had also planned to see if I could suspend the main thread, detach my custom debugger, and manually attach the Visual Studio debugger when my target exception was triggered so that I could utilize the more advanced VS debugger UI. In the end I never got around to trying the debugger switch because I discovered the problem by simply displaying some relevant details in a simple message box when the target exception occurred.

As you can see, I hardcoded the application path and command line, which includes a script file that simply quit AutoCAD after it started. I then used a batch file to run the custom debugger in a loop, thereby starting and quitting AutoCAD each time through the loop. I made this debugger a console app, but it could just as well be a window app.

I don’t claim to know anything about writing debuggers; this is just something I cobbled together after doing a bit of searching on the web. I decided to share the code in case someone else runs into a similar debugging scenario.

int _tmain(int argc, _TCHAR* argv[])
{
STARTUPINFO si;
ZeroMemory( &si, sizeof(si) );
si.cb = sizeof(si);
PROCESS_INFORMATION pi;
ZeroMemory( &pi, sizeof(pi) );
static TCHAR szCmdLine[] = _T(" /nologo /b C:\Test\quit.scr");
CreateProcess ( _T("C:\Program Files\Autodesk\AutoCAD 2014\acad.exe"), szCmdLine, NULL, NULL, FALSE, DEBUG_ONLY_THIS_PROCESS, NULL,NULL, &si, &pi );

DWORD dwContinueStatus = DBG_CONTINUE;
DEBUG_EVENT debug_event = {0};
bool bFirstBreakpoint = false;
bool bContinueDebugging = true;
while( bContinueDebugging )
{
if( !WaitForDebugEvent( &debug_event, INFINITE ) )
return 0;
//process event
switch( debug_event.dwDebugEventCode )
{
case EXIT_PROCESS_DEBUG_EVENT :
{
bContinueDebugging = false;
}
break;

case EXCEPTION_DEBUG_EVENT:
{
dwContinueStatus = DBG_EXCEPTION_NOT_HANDLED;
EXCEPTION_DEBUG_INFO& exception = debug_event.u.Exception;
switch( exception.ExceptionRecord.ExceptionCode )
{
case STATUS_BREAKPOINT:
{
if( !bFirstBreakpoint )
{
bFirstBreakpoint = true;
break;
}
dwContinueStatus = DBG_EXCEPTION_HANDLED;
DebugActiveProcessStop(pi.dwProcessId);
bContinueDebugging = false;
MessageBox(NULL,_T("Debug break!"), _T("Breakpoint"), MB_OK);
}
break;

case STATUS_ACCESS_VIOLATION:
if( exception.dwFirstChance == 1 )
{
if( ((DWORD_PTR)exception.ExceptionRecord.ExceptionAddress & 0xFF) == 0x30 )
{
DebugActiveProcessStop(pi.dwProcessId);
DWORD64 Rip = (DWORD64)-1;
HANDLE hThread = OpenThread( THREAD_ALL_ACCESS, FALSE, debug_event.dwThreadId );
CONTEXT Context;
Context.ContextFlags = CONTEXT_ALL;
if( GetThreadContext(hThread, &Context) )
Rip = Context.Rip;
MEMORY_BASIC_INFORMATION mbi = { 0 };
VirtualQueryEx( pi.hProcess, exception.ExceptionRecord.ExceptionAddress, &mbi, 0x36 );
TCHAR szMsg[4096];
_stprintf_s( szMsg, _T("Access violation (%016p) at address %016prnMem State:rnBaseAddress=%016prnAllocationBase=%016prnAllocationProtect=%08xrnRegionSize=%08xrnState=%08xrnProtect=%08xrnType=%08x"), exception.ExceptionRecord.ExceptionCode, exception.ExceptionRecord.ExceptionAddress, mbi.BaseAddress, mbi.AllocationBase, mbi.AllocationProtect, mbi.RegionSize, mbi.State, mbi.Protect, mbi.Type );
MessageBox( NULL, szMsg, _T("Exception"), MB_OK );

}
bContinueDebugging = false;
}
break;

default:
break;
}
}
break;
}
ContinueDebugEvent( debug_event.dwProcessId, debug_event.dwThreadId, dwContinueStatus );
}

return 0;
}