Virtual Orchestra
Advisor: Dave Musicant
The Vision
Have you tried out
a Microsoft Kinect? You
stand in front of it, flail your arms, legs, and body around, and it acts as a
controller for a video game. Unlike the Nintendo Wii, you don't hold the game
controller, you are the controller.
Kinect is most popularly known as a game controller for the Microsoft Xbox
360, but there's
a version of it for
computers that is programmable and has slightly better specifications.
The idea here is simple enough to describe: build a system that will enable
you to stand in front of a Microsoft Kinect and conduct a virtual
orchestra. By acting out the motions of an orchestra conductor, communicate to
the computer beat, temp, dynamics, cueing, and other info so that the computer
will play a musical piece as if it were an actual orchestra that you are
conducting.
What You'll Be Doing
You and your team will have to work in subteams and together to try to work
through the following:
- You'll need to build a system that understands conducting gestures. The
toolkits provide you with some capability for recognizing body positions, but
you'll ultimately research and implement machine learning techniques to
train and learn the system to recognize the gestures you want it to use. A
primary goal will be to get the computer to recognize real conducting
gestures, or something that very much approximates them; the idea is
definitely not to recycle predefined Kinect gestures and simply define them to
have conducting meanings.
- You'll need to encode a piece of music in a format that the computer can
read and play, and you'll need to design a system to play that music
back. When playing music, a computer normally just plays it via predetermined
instructions, as if it were executing a program. For this system, you'll need
to figure out how to get the computer to be reactive in its playing based on
the direction it receives from the human computer. Perhaps this is easy to do
regarding tempo and dynamics; the problem becomes considerably more
interesting when taking beat, cueing, and articulation into account.
- At first, you'll likely build the system to work with one person conducting
one orchestra section. Eventually, you'll add more sections, and conceivably
multiple conductors.
- If you get far enough along, it would be wonderful to build a
guitar-hero-like game that involves the player needing to conduct correctly in
order to score points.
- You'll need to answer the question of which programming environment you want
to use. The best and robust toolkit for programming the Kinect is provided by
Microsoft, and so you will definitely be using the Microsoft Kinect
SDK. You'll still have a choice, however, among the languages supported by the
SDK (C++, C#, or Visual Basic). You'll need to learn the language that you'll
use as well as the associated libraries and programming environment (likely
Visual Studio).
Resources
- There's a lot of documentation out there about Kinect programming, as well
as tutorials and papers written by people about machine learning projects
they've done to learn Kinect gestures. One of the first things the team will do
is work to uncover these resources.
- I'm in conversation with the music department, and we may have a music major
help out on a consulting basis with the project. Details are
still being worked out.