CS 332: Operating Systems

Adding a system call to Linux

UNDER CONSTRUCTION. Some editing yet to be done.

These instructions are valid for CS307 students during Carleton's spring term, 2008. We were using Red Hat Enterprise Linux installed on VMware Workstation (running in turn on Windows).

VMware

  1. Go to CMC 304 and login to Windows on one of the four machines in the left corner (on the east wall beyond the whiteboard--not the ones over by the windows). The four machines are named reddy, karp, yellow, and milner.
  2. Launch VMware. You should have a VMware icon on your desktop. If not, you can find it in the Start menu (possibly under System Tools).
  3. Select Open from the File menu, and select C:\My Virtual Machines\Red Hat Enterprise Linux 5\Red Hat Enterprise Linux 5.
  4. Click on "Start this virtual machine" in the middle of the screen (by a triangular "play" button).
  5. When you are given a choice of which copy of Redhat to boot from, choose "Red Hat Enterprise Linux Server (2.6.18-8.el5)". That's the one that has VMTools installed on it. You definitely want VMTools, because that's what enables you to drag files from Windows to Linux and back.
  6. Once Linux is booted up, login using "root" and "!jondich332".
  7. Right-click on the Linux desktop, open a terminal, and off you go.

Before you continue...

The big trick with the setup we are using is that you will be modifying files in the VMware Linux installation on some machine, but other people will need to use that same machine. Thus, it's important that you (1) revert the Linux sources to the form you found them in, and (2) make copies of your own modified files so you can move them to other machines and avoid the pain of having somebody else from the class delete your work. Here are some general suggestions for you:

  1. Whenever you are going to modify a kernel source file, make sure there is a copy of the original version of the file. For example, if you're going to modify Makefile, do "cp Makefile Makefile.orig" first.

  2. Mike has installed VMtools on all four VMware machines inside the kernel labeled "Red Hat Enterprise Linux Server (2.6.18-8.el5)". Thus, when you are booted into that kernel, you can drag files between Windows and Linux. Note that when you build a new kernel and boot into it, the VMTools will not be available unless you install them yourself. I'll try to obtain instructions for doing so.
  3. When you're finishing up a work session, you should make backup copies of all the files you changed. To do this using a VMTools-enabled kernel, navigate to each file and drag it from VMware to the Windows desktop. Then either e-mail the files to yourself, or save them in your Collab space, or put them on a flash drive, etc. Save them somewhere you can readily retrieve them later, in any case.
  4. Also, please copy the original versions of each modified file back into its original position (e.g. "mv Makefile.orig Makefile"), so the next person to work on that machine will not be messed up by your changes.

Linux source

The root directory of the Linux source code is at /usr/src/redhat/SOURCES/linux-2.6.18. Relative paths will refer to that base from here on.

Modify the system call table

  1. Edit arch/i386/kernel/syscall_table.S. Add ".long sys_yourname" to the end of the table of sys-call function pointers.
  2. Edit include/asm/unistd.h. Add "#define __NR_yourname [next available number]" to the bottom of the list of __NR constants. Then, increase NR_syscalls by 1.
  3. Backup /usr/include/asm/unistd.h to /usr/include/asm/unistd.h.orig, and then copy include/asm/unistd.h to /usr/include/asm.

Edit the Makefile structure

  1. Create a yourname/ subdirectory of the Linux source root directory.
  2. Edit Makefile (at the source root). The "core-y += kernel/..." line should become "core-y += yourname/ kernel/...".
  3. Copy mm/Makefile to yourname/.
  4. Edit yourname/Makefile to have only the line:
    obj-y := yourname.o
    plus an appropriate comment.

Add the source for your system call

  1. Create yourname/yourname.c with code for your system call. Note that the name of this function needs to be the same as the name ("sys_yourname", or whatever) that you used in the ".long" table in entry.S. Here's an example.

Compile and install the new kernel

  1. Make sure you're in the Linux source root directory, and then execute "make". This should take 5-10 minutes. Since you put yourname/ at the beginning of the "core-y" line in the root Makefile, you should see evidence fairly quickly that your new code is compiling (or not).

  2. Everything okay? Does it say "bzImage is ready"? Then:

    • make modules
    • make modules_install
    • make install

    in that order. You may or may not be able to skip the "make modules" and "make modules_install" steps. To save time, you might want to try and see if "make install" goes through successfully without doing those other steps first. In any case, once "make install" completes, it will have installed your kernel as one of the boot-up options.

Reboot

  1. Restart Linux, and when the OS choice screen appears, select the one that ends with yourname.

Test your system call

  1. Login as root again, and create a working directory ~/yourname.
  2. In your working directory create a test file mytest.c or some such thing. Here's an example.
  3. Compile and run your test program. Note that the $PATH environment variable probably doesn't include ., so you'll need to execute "./mytest" or some such thing.
  4. If all goes well, you will see printf output, but not printk output. To see the printk output, do "tail /var/log/messages".