Merry xmas everyone. After a long day of family festivities I had a spare few hours so I worked on my emulator.
I tried unpacking telock. The special thing about telock is that it uses hardware execution breakpoints.
I had to modify my program tracer to not use any hardware breakpoints at all. There is one slight cavaet in that i use an int3 on NtContinue that exists while the exception handler is executing, which could be detected. I might change the int3 to a clc to avoid simple checks, but for now nothing I am trying to emulate does these kind of checks.
Now I could trace telock, I went about implementing the hardware debug emulation. It took a few hours but I was able to implement most of the behaviour. I don’t set dr6 (the status) in my emulation so thats something I should do in the future, but its not necessary for now.
Which reminds me. After implementing DLL loading from the other day, I was able to unpack rlpack without futher modification. I finally finished today the code to do the library loading completely within the emulator (without the program tracer run in parallel).
I fixed a number of other bugs in the emulator, including some misbehaving x86 instructions, and a number of win32 functions.
I came across what seems to be a problem in the msdn documentation with VirtualProtect in one instance returning zero, which is meant to indicate failure. But from all accounts including GetLastError showing the call a success. There is some discussion on the wine mailing list from 2001 saying that in win98 and earlier VirtualProtect returns the original protection flags on the pages that were set. I couldn’t find a consistant explanation so I decided to simply follow the msdn documentation ignoring what actually happens in win32. Hopefully nothing bad happens..
I have another thing I have to implement.. import/export forwarding.. During program tracing, a call to GetProcAddress returned a RtlExitUserThread from ntdll.dll, when infact ExitThread from kernel32.dll had been requested. This apparently is a feature of the PE format, which I don’t implement. That’s what is currently the problem with telock, though I know that it also uses win32 file functions (CreateFileA iirc), which I don’t have implemented yet. That is one of the next things I’m going to implement – a virtual file system for the emulator. I had done the code for Linux emulation and attempted to merge it with win32 emulation, but its kind of broken now.
Oh.. I can also unpack expressor now. And also packman – dunno if I mentioned that earlier.
Oh.. one last thing that needs to be implemented. It’s hard to know when to stop the unpacking process. breaking on execution on priorly written to memory locations is the basic algorithm, but sometimes there are multiple layers. I am thinking of implementing a check of entropy to guess if memory is still packed to decide if the unpacking process should be reset and continued.