ACprotect now unpacks successfully, though my emulation isnt exactly ideal. The problem is this packer uses GetThreadContext and SetThreadContext with the current thread obtained with GetCurrentThread (which actually returns a constant ~1). In windows, the ‘context’ is actually the register state of the CPU. This gives your program access to things like the debug registers for instance.
The MSDN documentation says that in this particular case of using the current thread, the calls should succeed but the context will be undefined. My tracer however consistantly showed these calls to fail (a return value of 0). I couldn’t reproduce the failure of these calls in small test cases, and I still don’t know why they return such a value. The context was mostly ignored in both of these calls, however in GetThreadContext, the extended registers (sp?) section of the context appeared modified and it showed up during debugging as an inconsistancy in a traced version compared to the emulated version.
So I ended up doing nothing really for the implementation except to return from each function, and it appears to work fine. I ended up doing other fixes to the emulator during coming up with the above results; I now correctly handle access violations based on memory permissions – before it was only if memory didn’t exist would I raise an exception.
I also tried unpacking Yoda’s Protector 1.02 without success. It used the %ss register at one point, to which I realized I wasn’t setting all the selectors as they should; I was only setting cs and ds. My emulator should generally handle segmented memory OK, but for some instructions I use the cs segment instead of the implied segments like es or ds. I’ll fix this one day maybe, but almost no code uses this type of assembly anymore so it’s not a priority.
Right now I’m stuck in yoda’s where its parsing the TIB->PEB->LoaderData which holds a list of modules that have been loaded by the process. I’ve actually been past this point in the past (and ended up getting stuck in the file handling code – which I’ve now implemented), but I guess the hack I used to originally succeed is no longer present when I worked on getting the emulator to run standalone.