Binary Auditing part 7, Identifying the Conditional Operator

This is a fun one. Assembly has a lot of conditional statements. Remember, a conditional is when there is a comparison. An example:

If a = 1;
   print "A is equal to 1";
else
   print "A is not equal to 1";
end

In the pseudo code above, "If a = 1;" is the conditional. The other lines are the action taken depending upon the conditional's results.

The instructions we're going to take a look at in this exercise: cmp, jle, test, jge, jz, jbe, setnz, setnle.

**

cmp (compare): this instruction subtracts from each other and sets the ZF flag (ref: http://en.wikibooks.org/wiki/X86_Assembly/Control_Flow#Comparison_Instructions)

At 00401008, we see the compare of the value of var_4 to 0. In this case if var_4 is not equal to 0, ZF is set to true (or 1). If var_4 was equal to 0, the ZF flag would remain at 0 (or false).

**

setnle (set if not less than or equal): this checks the flags in the EFLAGS register (CF, SF, OF, ZF and PF), then acts upon whatever register (in this case, the lower half of eax which is al) is passed to it. (ref: http://faydoc.tripod.com/cpu/setnle.htm)

At 0040100C, setnle sets al to 1 because the ZF flag is set.

**

jle (jump if less than or equal to): acts upon a comparision before it, and looks at the ZF or the SF/OF CPU flags (http://en.wikibooks.org/wiki/X86_Assembly/Control_Flow#Jump_if_Less)

At 0040101A, jle sees if the cmp above it is true or false. If var_8 is less than 0, it takes the jump.

**

jge (jump if greater than or equal to): there is a compare before this statement, but it checks to see if the SF flag is equal to the OF flag.

At 0040107D, you'll see the jge instruction. Play with it in a debugger: 1) set a breakpoint at 0040107D, 2) look at the SF and OF flags. Now, SF and OF, by default, are 0 in this program, so IDA shows us that it wants to jump to 00401087. However, if you modify the SF or OF flag to 1 so that the two CPU flags are not equal, the jump is not performed. You can modify the flag values by right clicking on the flag you want to modify.

**

test: compares to see if the two are equal to each other ( http://en.wikipedia.org/wiki/TEST_(x86_instruction) )

At 00401076, it does a comparision to see if eax is equal to eax. This may seem odd, but it's because eax contains the results of the function getmainargs. If the function was successful, eax will equal 0. If it is not, eax will be a negative number (http://msdn.microsoft.com/en-us/library/ff770599.aspx). So the test will find the results of whether or not the function call was successful.

**

jz/je (jump if zero/equal): jump if ZF CPU flag is 1
jnz/jne (jump not zero/not equal): jump if ZF flag is 0

At 00401263, there is an example of this. The previous instruction is comparing the hex values 5A4D to what's located on the stack. If you hover over it, you see it is equal to ZM aka 5A4D.

**

setnz (set is not zero): this looks at the ZF CPU flag. If it's 0, it'll set whatever register is passed to it to 1. If not, that register will remain.

At 004012B5, by default the ZF flag will be set. Again, modify it to 0 and see that ecx is changed to 00000000.