Virtob.F isn’t too new, but it was recently the first time I looked at it. I ran it in my emulator and it quickly crashed.
A little inspection and I saw it was using the return address on the stack from the EntryPoint code. At least in Vista, the entry point is called like a regular function from kernel32.dll. The PEB Is passed as an argument. If the function returns, then one of the Rtl ExitProcess functions is called to terminate (using %eax as an argument). This points clearly to the fact that somewhere in kernel32.dll takes control of the function first, probably with the entry point of the program and the PEB as an argument. I haven’t investigate too much furthur beyond disassembling the caller of the entry point.
In any case, Virtob.F took the return address ([esp] at program start], subtracted 15 from it, and checked the contents of that memory address against the value 8. If it didn’t match, it would branch to a non existant address and cause an exception/crash.
Turns out on XP and Vista, the calling code of the entry point changed. On XP, the value described above is indeed 8. In Vista, it’s something else.
I also came into some other interesting anti-debugging tricks with some other packers. An early version of telock (as identified by peid) was overwriting its own instruction with a rep stosb; a prefetch trick. My emulator and tracer don’t support that yet, but I might implement that today.. A few packers were using anti-emulation loops; one piece of malware wanted to iterate through a loop that modified memory 53 million times.
Maybe this post wasn’t blog worthy afterall.. hopefully someone will find it mildly interesting 😉