Another CPU bug uncovered while testing my emulator. I came across a repne scasb while emulating the win32 version of upx. The logic of scasb (scan string), to paraphrase the intel manuals is
SRC = dereference(edi)
temp = al - SRC
In the code I ran across, %al was set to 0, the byte at (%edi) was 70 (decimal). %ecx was large. Following the operation, the carry flag was cleared. This is incorrect, the carry flag should be set (0 – 70 sets carry).
I was unsure if my understanding of carry was wrong, so I tried 0 – 70 in a sub. Carry was set as expected. scasb’s logic is to perform a temporary subtraction of %al-(%edi) and set the status flags using the temporary result as explained earlier.
When scasb was performed in isolation with the same test case, carry was set. It seems that including repne in the scasb, changes the carry flag to an incorrect result.