Carleton Comps Project: Spam Filter

Project: Constructing a spam filter

Advisor: Dave Musicant

Meeting time: TTh 3:10-4:55

Final Results

I. Background

Spam, or unsolicited commercial email, is a common annoyance. The amount of spam that appears in our email boxes seems to increase on a regular basis. Fortunately, a number of good techniques have been developed for detecting spam from non-spam. Free tools such as SpamAssassin are becoming popular, as well as a variety of commercial tools. However, most of these tools are designed to be run on mail servers. For users without superuser access to their servers, the only options are to run email software with spam filtering built in. This project will involve writing an email proxy server that will run on a user's local machine and sit between a user's email software of choice and the actual email server.

II. The Project

Here is a list of the concepts and technologies that will be necessary.

Email. The proxy server will need to have some of the capabilities of a standard email client. It will need to communicate with IMAP and POP servers. It must also be able to parse message headers in order to deliver appropriately.
Java. The Javamail API provides a well-made interface to handling many of these issues. Additionally, this will enable the proxy to (easily?) be cross-platform.
User interface. The proxy server needs to be configurable by the user, with regard to settings and storage of sample spam emails.
Machine learning. The Mozilla email client has recently adopted ideas for filtering email from Paul Graham's article "A Plan For Spam." Graham uses a Bayesian learning approach. There are a number of other approaches that have been proposed in the research literature, using techniques such as support vector machines and decision trees. We will develop a number of alternative techniques and use the one that works the best, or allow the user to choose which technique is desired.

III. References

H. Drucker, D. Wu, and V. N. Vapnik, "Support Vector Machines for Spam Categorization," IEEE Transactions on Neural Networks, Vol. 20, No. 5, Sep. 1999.

M. Sahami, S. Dumais, D.. Heckerman, and E. Horvitz, "A Bayesian Approach to Filtering Junk E-Mail," in Proc. AAAI-1998, Jul. 1998.

Jason D. M. Rennie, "ifile: An Application of Machine Learning to E-Mail Filtering," in Proc. KDD-2000 Text Mining Workshop, Aug. 2000.

X. Carreras and L. Marquez, "Boosting Trees for Anti-Spam Email Filtering," in Proc. Euro Conference Recent Advances in NLP (RANLP-2001), Sep. 2001.