Advice for the buffer overflow assignment

Understanding roughly how to solve the phases of the buffer overflow assignment isn't too hard once you get the idea. But it is very easy to be off by a byte or two, or to misunderstand some small detail that then prevents your solution from working at all. This can be very frustrating.

The main tool you need to help cut through the confusion is the ability to use gdb, so you can step through the code and watch what happens when you use a particular collection of bytes to overflow your buffer. Usually, if you step carefully through your code and pay attention to rsp and the memory it points to (i.e., the stuff on the top of the stack), you can see what's happening when and why.

How to debug ctarget with gdb

• Draw a diagram of what you want your overflowed buffer to look like.

• Put your proposed solution in ctarget.phaseN. For example, suppose you want 24 bytes of 0xAB followed by the 8-byte integer 0xAABBCCDDDDCCBBAA. Then put the following in ctarget.phaseN. (C-style comments are allowed in these files, as long as you have a space after /* and a space before */)

  /* ctarget.phaseN */ /* 24 garbage bytes */ AB AB AB AB AB AB AB AB AB AB AB AB AB AB AB AB AB AB AB AB AB AB AB AB /* the address of something or other */ AA BB CC DD DD CC BB AA

  I usually put my bytes in rows of 8 so I can count them easily. It is very easy to be off by a byte or two if you try to put them all in one line of text.

• Convert your solution from the hexadecimal representation in ctarget.phaseN to the actual bytes themselves (i.e., the "raw bytes" the assignment refers to) like so:

  cat ctarget.phaseN | ./hex2raw > ctarget.phaseN.raw

• Launch gdb, set a breakpoint, and run the program with your raw input.

  gdb ctarget (gdb) layout asm (gdb) br getbuf (gdb) run -i ctarget.phaseN.raw (gdb) i r rsp (gdb) x/20xw $rsp etc.

• Pay close attention to what happens to rip, rsp, and the top of the stack after each callq or retq instruction. If you understand what those two instructions are doing, the rest of this should be a lot easier.

Good luck!