I was trying to come up with ideas to automatically generate a static unpacker given a random packer executable. I didn’t arrive at anything concrete but did have a couple ideas that could aid the analysis.
Take a packer (the packer), and a binary (the binary) to be packed. The idea is to do dynamic taint analysis of the binary as the packer is run. The analysis would need to follow file reads and writes. What you would end up with a packed executable with well data inside it that is tainted, being the ‘data’ (what the unpacking code unpacks), and the rest of the packed executable being the code (the static code being put in by the packer).
This would be useful when doing manual analysis I think, and I guess is part of a step involved in making an automatic static unpacker. That is, it can automatically identify what is data and what is code, except for the fact that data/code separation is specific to that single packed binary… but
How about doing dynamic taint analysis on PE header members of the binary. Say the entry point member in the PE headers. By following where that ends up you could determine references to the original entry point in the resulting packed executable. By doing a bindiff and structurally analysing the ‘code’ (from above), you could find equivalent locations in other executables that have been packed by the same packer.
Maybe by tracking other members such as members indicating the size of sections, you could arrive at ways to automatically determine the length of the packed executable data.
I think this is something worth implementing, but it seems to be a fair bit of work from where I am now.. The easiest way to do this would be to modify something like Pandora’s Bochs (even though it uses python. arggh), which can trace individual windows processes (in our case, the packer that we are running), then perform analysis on each instruction traced to do the dynamic taint analysis.
I don’t see my emulator being as useful as something like Bochs, since my emulator would have extreme trouble emulating a regular standalone program in its entirety.
I hope when I start working again soon, I will have time to implement some of these things.