CS 332: Operating Systems

Adding a system call to Linux

These instructions are valid for CS307 students during Carleton's spring term, 2010. We were using Red Hat Enterprise Linux installed on VMware Workstation (running in turn on Windows).

VMware

  1. Go to CMC 304 and login to Windows on one of the five machines on the left side The five machines are named reddy, karp, yellow, englebart, and floyd.
  2. Launch VMware. You should have a VMware icon on your desktop. If not, you can find it in the Start menu (possibly under System Tools).
  3. Select the "Red Hat Enterprise Linux 5" VM from the sidebar, right-click, and select Snapshot Manager. From here, double-click the CS332 Syscall Lab snapshot.
  4. Once Linux is booted up, login using "root" and the password I gave you in class.
  5. Right-click on the Linux desktop, open a terminal, and off you go.

Before you continue...

  1. Mike has installed VMtools on all four VMware machines inside the kernel labeled "Red Hat Enterprise Linux Server (2.6.18-8.el5)". Thus, when you are booted into that kernel, you can drag files between Windows and Linux. Note that when you build a new kernel and boot into it, the VMTools will not be available unless you install them yourself. You can install them onto your compile via the VM < Reinstall VMWare Tools menu item.
  2. When you're finishing up a work session, you should make backup copies of all the files you changed. To do this using a VMTools-enabled kernel, navigate to each file and drag it from VMware to the Windows desktop. Then either e-mail the files to yourself, or save them in your Collab space, or put them on a flash drive, etc. Save them somewhere you can readily retrieve them later, in any case. Once someone else starts up the lab snapshot, they will disappear.

Linux source

The root directory of the Linux source code is at /usr/src/redhat/SOURCES/linux-2.6.18. Relative paths will refer to that base from here on.

Modify the system call table

  1. Edit arch/i386/kernel/syscall_table.S. Add ".long sys_yourname" to the end of the table of sys-call function pointers.
  2. Edit include/asm-i386/unistd.h. Add "#define __NR_yourname [next available number]" to the bottom of the list of __NR constants. Then, increase NR_syscalls by 1.
  3. Copy include/asm/unistd.h to /usr/include/asm/.

Edit the Makefile structure

  1. Create a yourname/ subdirectory of the Linux source root directory.
  2. Edit Makefile (at the source root). The "core-y += kernel/..." line should become "core-y += yourname/ kernel/...". Additionally, you should add -yourname in the EXTRAVERSION field at the top.
  3. Copy mm/Makefile to yourname/.
  4. Edit yourname/Makefile to have only the line:
    obj-y := yourname.o
    plus an appropriate comment.

Add the source for your system call

  1. Create yourname/yourname.c with code for your system call. Note that the name of this function needs to be the same as the name ("sys_yourname", or whatever) that you used in the ".long" table in entry.S. Here's an example, which can also be found in root's home directory.

Compile and install the new kernel

  1. Make sure you're in the Linux source root directory, and then execute "make". This should take 5-10 minutes. Since you put yourname/ at the beginning of the "core-y" line in the root Makefile, you should see evidence fairly quickly that your new code is compiling (or not).

  2. Then:

    • make modules
    • make modules_install
    • make install

    in that order. Once "make install" completes, it will have installed your kernel as one of the boot-up options.

Reboot

  1. Restart Linux, and when the OS choice screen appears, select the one that ends with yourname.

Test your system call

  1. Login as root again, and create a working directory ~/yourname.
  2. In your working directory create a test file mytest.c or some such thing. Here's an example, which can also be found in root's home directory. Note that you have to compile with special options, which are detailed in the comment at the top of the code.
  3. Compile and run your test program. Note that the $PATH environment variable probably doesn't include ., so you'll need to execute "./mytest" or some such thing.
  4. If all goes well, you will see printf output, but not printk output. To see the printk output, do "tail /var/log/messages".