First up, I got winupack to unpack successfully which required me to implement and fix some x86 emulated instructions. I also implemented tracking/emulation of export forwarding, though I think there are still some bugs in it.
A few other things got fixed up, including writing to the process image during tracing of a PEB without the BeingDebugged flag set. This doesn’t seem to affect anything badly during tracing, but can evade the debugger checks. peloc was doing that check.
pelock also uses the sidt instruction, presumably to check if its running inside a VM. I implemented that instruction in the emulator along with sgdt and sldt. This makes me think I should use QEMU and not VMWare for malware testing. pelock does some other anti-debugging checks including checking for the existance of some drivers using CreateFileA.
pelock fails early when emulating in standalone mode, but runs longer when being traced. I checked the instruction trace, and it seems it uses the status flags after a few library calls (LoadLibraryA and MessageBoxA iirc). In the traced version I copy the flags from the traced program back into the emulator, as I consider them undefined after a library call (I try to keep the register state the same in the emulator and the tracer so I can track when there are real differences and so easily detect bugs). I’ll try to track down this problem a bit later, as I think the next thing I should do is implement file system emulation, and hopefully then be able to unpack telock and pespin.