Lab 6: Debugging x86-64 with gdb
Goals
Your primary goal for this lab is to get familiar with using the GNU debugger (gdb
) to explore and debug x86-64 programs, in which you may not be able to view the original C source. As a secondary goal, you’ll get some practice that may help with any sneaky escapades you may undertake.
Documentation and tutorials
You will find some potentially useful gdb resources posted on our Resources page. There is also a really handy summary sheet on page 280 of the textbook.
Getting started
In this lab, we’ll be working with a pre-compiled program named call_proc
, which conveniently builds on the proc
function (that really complicated one that takes eight parameters) that we’ve been discussing in class. Grab the program and what source code I’m giving you, and copy them to mantis
.
Here are some steps to follow:
-
SSH into
mantis
via VS Code and open a terminal in yourcs208
folder. -
Make a new directory for this lab:
mkdir lab6
-
Change to your new directory:
cd lab6
-
Grab the compiled program and source code:
wget https://cs.carleton.edu/faculty/tamert/courses/cs208-f23/resources/samples/call_proc.tar
-
Un-tar it:
tar xvf call_proc.tar
-
Look at the source code in
call_proc.c
. You should see the familiarproc
function, and some other stuff we’ll explore soon. -
You won’t be able to compile the program yourself, as you don’t have a definition for the function
phase_0
. You do have the executable, however. Try to runcall_proc
. Unless you get really lucky, you won’t win. :)./call_proc
-
Note: If you have any issues running
call_proc
, you can use the following command to make it executable:chmod +x call_proc
Step 1: Exploring proc
with gdb
We’ll now explore the proc
function we’ve been discussing in class.
-
Open the file
call_proc.c
. You’ll find this function on lines 5–14. -
Open a Terminal window in VS Code.
-
Run
gdb
, ready to step throughcall_proc
:gdb call_proc
We want to explore the inputs to proc
. We expect to find the first six arguments in registers, with arguments 7 and 8 (x4
and p4
, respectively) instead on the stack. In this case, proc
is called on line 23, as follows:
proc(10, &a, 20, &b, 30, &c, 40, &d);
We drew the following picture in class, to represent the state of the stack at the start of a call to proc
:
Let’s find the arguments!
-
We’re using
gdb
, so set a breakpoint at the start ofproc
:(gdb) b proc
-
Run the
call_proc
program:(gdb) r
-
You’ll have to enter a number, and then you should hit the breakpoint. We expect the first argument in register
%rdi
; we should find the value10
there:(gdb) print $rdi
(Note the very silly convention in
gdb
that you use$
for registers instead of%
.)
-
The next argument should be in register
%rsi
. This one is an address, and we don’t have a good sense of what the address is yet. We’ll explore that later, in Step 2:(gdb) print $rsi
-
By default, this seems to display in decimal, which is kind of useless for addresses. Use
/x
to make it display in hex instead:(gdb) print /x $rsi
-
Arguments 3–6 should be in registers
%rdx
,%rcx
,%r8
, and%r9
, respectively. It’s tedious to view each individually, so let’s look at all registers instead:(gdb) i r
-
You’ll likely notice that the contents of
%rdx
and%r8
don’t look like what we expect. Keep in mind that they’re in hex, not decimal! Conveniently, usingi r
ingdb
displays both, one per column. Make sure you know which one you’re reading. -
Now let’s look at the stack to find our 7th and 8th arguments (when I ran this,
%rsp
contained the value0x7fffffffe458
):(gdb) x/3gx 0x7fffffffe458
The first 64 bits (a “giant” word in
gdb
, hence theg
) contain the return address for whenproc
is done. The next 8 bytes are our 7th parameter (0x28 in hex is 2*16+8 = 40 in decimal), followed by an address that is our 8th parameter.
-
If you type
list
, you’ll see the C code for theproc
function:(gdb) list
-
If instead you want to see the assembly, you can type
layout asm
. This gives us a split screen with the assembly code on the top and the(gdb)
command prompt on the bottom. Note that it can be a bit erratic, so if it gets messed up visually, press the key combination Ctrl-L to redraw the screen.(gdb) layout asm
You should find very similar assembly to what we looked at in our classwork in Lesson 16. Read through it (it’s just seven lines) and make sure you can follow what it’s doing.
Next, we’ll look at a function that calls proc
; this function is aptly named call_proc
.
Step 2: Exploring call_proc
with gdb
We’re going to step through call_proc
to see how well it lines up with our understanding of its use of local variables and argument prep on the stack. We’re assuming the stack will look like this after the four local variables are given their values:
If you’re continuing from Step #1, type c
to continue the program (this should complete its execution), then type d
to delete all breakpoints. If you’re starting fresh, run gdb call_proc
.
-
Set a breakpoint in the function
call_proc
:(gdb) b call_proc
-
Run the program, and enter a number:
(gdb) r Guess a number: 123
-
This should hit the breakpoint. We expect
a
to be stored in24(%rsp)
. The first thing you should notice is that isn’t the case. Where is it storing$0x1
instead? Think about why this is okay (hint: look for later modifications to the stack before callingproc
). -
Draw a picture for yourself that shows the state of the stack now that we’ve seen the actual assembly.
-
Let’s move down to check the stack. Type
ni
for next instruction, and repeat it several times until you’re about to execute the firstlea
instruction. -
Inspect the stack. Let’s look at the 32 bytes starting at the top of the stack (for me,
%rsp
has value0x7fffffffe7c0
as I run this):(gdb) x/32bx 0x7fffffffe7c0
You should see
0x01
followed by 70x00
s. That’sa
. On the line above, you’ll see each ofb
(4 bytes),c
(2 bytes), andd
(1 byte), all next to each other.
-
Let’s keep going, and see the argument building to prep for calling
proc
. Notice thelea
instructions; for example, address0x10(%rsp)
(that’sa
) is put in%rsi
, as&a
is provided as our second argument on line 23 of the source code. Additionally, we see that%rax
and$0x28
(that’s 40 in hex) are pushed to the stack; these are arguments 8 and 7, in that order. -
Finally, let’s print out the state of the stack right before the
callq
to jump toproc
executes. Hit return several more times (and/or typeni
again) to get there, then get the new value of the stack pointer:(gdb) print $rsp
-
Finally, let’s look at the stack (my new value of
%rsp
is0x7fffffffe7b0
):(gdb) x/48bx 0x7fffffffe7b0
The stack should approximately match this state of the world:
(There are some extra push
instructions up above, so we have some saved registers, too, but this should be close.)
Step 3: Solving a puzzle with gdb
Follow along with the video to see the steps you can take to work on your zoo escape.