Piano Hero

An Intelligent Music Stand

What?

“Piano Hero” is an intelligent music stand – a program that displays sheet music on screen, and uses a computer microphone to follow along to you playing the piece on a piano, showing you where you are and turning the page for you.

It was created by a group of class of 2010 Computer Science majors at Carleton College for their senior integrative exercise. It has no relation to the “Guitar Hero” series of games.

How?

Our music stand is implemented in Python, with some parts implemented as an extension module written in C.

The program takes in a LilyPond sheet music file, and uses LilyPond's command line tool to render the score, and to produce a MIDI file which describes to a computer how the piece should be played.

We read the MIDI file to obtain the sequence of notes in the desired piece of music, and then load it into the matcher – the software component that decides where the player currently is within the piece. At the same time, we begin listening to the sound being captured through the default audio input device.

To find out what notes are being played into the microphone, we pass the audio data through a Fast Fourier Transform to break it down into individual sound waves. We do significant cleanup of this wave data, and pass it into the matcher, which makes the final decision on what note(s) are currently being played. The matcher advances through the piece when the correct upcoming note(s) are heard.

We display LilyPond's rendering of the sheet music on screen, and parse the commands in the PostScript copy of the rendering to figure out what measures are on what page (so that we know when to turn the page), and where each measure is within the page (so that we can highlight the current measure).

Who?

This project was completed by the following fine group of people:

Eric Naeseth is a Computer Science major from Sister Bay, WI who knew nothing about music before this project, but now knows just enough to be dangerous. He plans to find software design work after Carleton, with the later possibility of graduate school.

David Tillery is a Computer Science major from Anchorage, though he has other hobbies that involve getting sunlight. He enjoys coding and hopes to do it for a living.

Emily Kawaler is a Computer Science major from Ames, IA who used to love music and now never wants to hear it again. She plans to get an advanced degree in bioinformatics after leaving Carleton.

Ben Farley is a Computer Science major from Denver. In his spare time, he enjoys playing video games, playing the saxophone and piano, and participating in lots of IM sports. Next year he plans to go to graduate school to earn his master's, and possibly PhD, in Computer Science.

Emma Zhou is a Math and Computer Science double major from Glencoe, IL. She likes cool math puzzles and hates G sharps. She plans to find work in software development next year, and will eventually go to grad school in theoretical computer science.

Nathan White is a Economics and Computer Science double major from St. Louis. He plans to be a high school economics teacher and spent most of his college experience on the Internet, watching television, or making bad jokes.

Want!

The Comps Flower

Piano Hero currently only runs on Mac OS X 10.5 and above systems. To build our project, you must first install the following dependencies:

After ensuring these dependencies are available, cd into the directory containing the project using a terminal, and then run make.

Once the project is built, you can launch Piano Hero by running python musicstand.py from within the project directory. This will bring up a welcome screen.

From the welcome screen you have several options. On the left is a list of songs you have already imported into the program. Double-click one to go to the player view.

On the right, you can either load in a new piece (type the title you wish to give it into the top field, a path to the LilyPond file in the second, and click “Load”), listen to a MIDI of the piece that is selected on the left side, or make a new LilyPond file.

Once in the player view, the player may start playing immediately, at which time the highlighted measure will begin to change, indicating which measure they are currently playing. They may also choose to advance the highlighted measure or turn the page manually. Other options include two different types of page transitions, the ability to select the speed of page transitions, and the ability to select how many measures ahead of the end of the page the player would like the transition to begin. The player may also start the piece over if she wishes, go back to the welcome screen and choose a new piece, or simply quit the application by closing the window.

Later?

Currently, our matcher accepts a wide range of Fourier transform results as being the correct note, which leads to some undesired behavior when a piece contains many similar-sounding notes in a row. This currently works by looking for the next expected sheet music note(s) in the frequency results; instead it would be nice if we could deterministically reduce determine what note(s) were contained in frequency results.

In addition, we would like to use our ability to make instrument "profiles" to support instruments other than pianos, and use that ability to be able to listen to pieces with multiple instruments in it.