Monthly Archives: June 2008

Bug in single stepping over a popf setting the trap flag

Title of the post nearly sums it up.  In win32, single stepping over a popf that sets the trap flag.  The trap flag when examined using GetThreadContext reports the trap flag as being clear.

I tried for the first time installing OllyDbg today also, but Olly has no problem in detecting the trap flag as set.  I’m not sure how it is able to do this.

To implement a solution in my own debugger, I will have to disassemble from the instruction pointer.  If its a popf, I will retrieve the contents from the stack and check for the trap flag being set.  If it is, I will call DbgContinue with DBG_EXCEPTION_NOT_HANDLED.

Single stepping through NtContinue, faking pushf, and trapping int1

In my bid to emulate win32 seh (in my emulator), I needed (to test my emulator) to single step through exception handling.  My first approach was to just single step through the program by setting the trap flag in the eflags register.  I had some problems with this.

One of the problems is single stepping through NtContinue.  Looking at the code it does a sysenter which transfers control to the kernel.  The kernel then sets the new context to have somewhere to return too.  I tried single stepping past the sysenter – the problem is however, that a single stepping exception isnt raised until it comes too the 2nd instruction of the new context.eip.  Bizarre.

Also bizarre is the fact that NtContinue changes Dr6 and Dr7.  I had some bad code just checking for Dr6 to be non zero for a breakpoint exception, but infact while the lower 4 bits are 0 (1 bit for each possible breakpoint register Dr0-Dr3), the upper bits were thrashed.  Also Dr7 was changed.  Perhaps its even conceivable that there is information leakage from the kernel.  Dunno for sure. 

The only solution for trapping on the first instruction following NtContinue, was setting a breakpoint on context.eip.  Therefore this requires parsing of NtContinue’s argument (for which there is a CONTEXT structure).  This is fairly straight forward.  Set a breakpoint on NtContinue, then parse the CONTEXT and set a new breakpoint on context.eip.

How do we get the breakpoint for NtContinue?  We can do a couple of things.

1) resolve the symbol, and place a breakpoint on it

2) look at the return address of the exception handler, which eventually ends up taking you into NtContinue.

I chose 2).

Both these solutions have problems if the exception handler decided to implement its own version of NtContinue.  Perhaps the best solution is to emulate down to the native api.  I didn’t implement emulation of the native api, so I’m stuck for the time being with the partial solution.

In the process of partially emulating vmprotect, I came across a few calls to pushf, and popf.  These instructions push and pop the eflags register.  Initially I thought the packer might be checking to see if the trap flag was set.  It turns out it wasn’t doing this check, but I none the less implemented in my tracer/debugger the ability to process pushf’s by modifying the stack contents to have the trap flag clear.  This hack should allow my debugger to work on binaries that include this type of anti-debugging.

Also implemented in my emulator is processing of explicit int1’s in the code.  Before DbgContinue is called, I check the current instruction pointer and if its an int1 instruction, I keep note of this, and also allow the application to process the int1 by itself.  Naturally I have to set a breakpoint in the exception handler, If i still want to maintain control.