The project is to create a virtual desktop drum kit from a video camera and a desk-mounted microphone (e.g. a microphone that is touching, tapped against, or resting upon a desktop).
The camera is mounted above the desk, looking down the desk.
The camera tracks the location of one or more points-of-contact with the desk, and the microphone picks up the sound made by tapping the desk (e.g. tap the desk with a drumstick, or fingernail, or the microphone itself).
Depending on the coordinates of the point-of-contact, the sound is changed. The sound comes from the actual sound made by hitting the desk. This sound is modified depending on where you hit the desk.
Below, see a video of the system in operation, where the passband frequency is made to depend on the left-to-right location where the desk is hit (low notes on the left, and high notes on the right), and the gain (sensitivity) depends on the up-down location (low volume at the bottom of the camera's field-of-view, and high volume near the top of the camera's field-of-view).
Note the colored tape wrapped around the end of the microphone. The computer vision system tracks this color to determine the coordinates of the microphone on the desk.
You can see a demo movie on YouTube: http://www.youtube.com/watch?v=IEkKs8Gcg_o, or Glogger: http://glogger.mobi/v/82258.
Also, each module can be done by competing teams. For example, two ore more teams can do the vision module, and each of the teams' vision modules can be tested with any one of a variety of competing audio modules, to end up with a best-of-best style of competition or coepetition.
A = 220.00 B = 246.94 C = 261.63 D = 293.66 E = 329.63 F = 349.23 G = 392.00 a = 440.00 b = 493.88 c = 523.25 d = 587.33 e = 659.26
This system can go anywhere. For example, the surface can be a sidewalk
or other space, not merely limited to a desk:
Optionally, the surface can be a spinning cylinder instead of a desktop:
The Friction Idioscope: Polyphonic friction idiophone
having continuously variable pitch: Public performance by composer Ryan
Janzen:
Note the colored tape on two fingers of the musician.
These are unique colors tracked by the vision system.
(Rightmost: Alternate playing style on the spinning cylinder)
(Above: audience participation: Teaching and demonstrating the idioscope...)