retn $0xhh consistency across function tails

Some procedures, following a calling convention simply return (ret), without modifying the stack pointer (they expect the caller to perform stack correction).  In another call convention, procedures (callees) modify the stack using retn $0xhh.

Yesterday I made some changes to my disassembler, so that it would look at the stack correction in procedures.  But there is a possibility of inconsistency when there is more than one control path that exits the procedure.  Each seperate path could have different stack correction.

I ran a test involving all the binaries on Windows XP Home Edition in \windows\system32\ that can my disassembler can handle (278 in total).  It doesn’t yet process DLL’s, just executables.  The results were interesting.  9 binaries had inconsistancies in some of its procedures.

  • There were a majority of binaries that had a failed analysis of a procedure using an idiom where a call would be issued, and the following instruction would be an int3.  This apparently is a call to a procedure that should not return, or if it does, it should raise a debug exception (thanks to nevar for telling me this).
  • A couple of binaries failed in procedures that were using the frame pointer (for local variables and procedure parameters), but did not setup or destroy a new frame using the standard prologue and epilogue.  I suspect that these are nested functions.  However Cygwin’s compiler fails to generate such code.  I should experiment with other compiles. 

For comparison, I ran several of the failed binaries through IDA, which fails to perform SP (stack pointer) analysis.  Note that the procedures in question only have one entry point.

My question then, are these code sequences correct?  Is it ever valid for a compiler to generate code that performs stack correction differently, depending on the code path taken in the function tail?

My suspicion is that these are compiler bugs… but maybe someone can answer definitively..


One response to “retn $0xhh consistency across function tails

  1. It depends, sometimes Microsoft add their own obfuscation code. For example, the drm ActiveX dll does a call obfuscation (return x byte after the call, always the same size .. too easy to break) and a friend of mine saw many “return to libc like” in Microsoft binaries (push eax, ret). I think gdi32.dll or user32.dll got at least one.

    I also saw some Microsoft binaries that (like you said) call an exception raising or ExiThread() or ExitProcess() like functions and then not generate next code (then you see some int 3),. It does break ida which cannot found a correct function ending. Is it a weird optimization ? Maybe it saves some size.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s