Some procedures, following a calling convention simply return (ret), without modifying the stack pointer (they expect the caller to perform stack correction). In another call convention, procedures (callees) modify the stack using retn $0xhh.
Yesterday I made some changes to my disassembler, so that it would look at the stack correction in procedures. But there is a possibility of inconsistency when there is more than one control path that exits the procedure. Each seperate path could have different stack correction.
I ran a test involving all the binaries on Windows XP Home Edition in \windows\system32\ that can my disassembler can handle (278 in total). It doesn’t yet process DLL’s, just executables. The results were interesting. 9 binaries had inconsistancies in some of its procedures.
- There were a majority of binaries that had a failed analysis of a procedure using an idiom where a call would be issued, and the following instruction would be an int3. This apparently is a call to a procedure that should not return, or if it does, it should raise a debug exception (thanks to nevar for telling me this).
- A couple of binaries failed in procedures that were using the frame pointer (for local variables and procedure parameters), but did not setup or destroy a new frame using the standard prologue and epilogue. I suspect that these are nested functions. However Cygwin’s compiler fails to generate such code. I should experiment with other compiles.
For comparison, I ran several of the failed binaries through IDA, which fails to perform SP (stack pointer) analysis. Note that the procedures in question only have one entry point.
My question then, are these code sequences correct? Is it ever valid for a compiler to generate code that performs stack correction differently, depending on the code path taken in the function tail?
My suspicion is that these are compiler bugs… but maybe someone can answer definitively..
Silviocesare's Weblog 