ECE532 Project: The idioscope

http://wearcam.org/idioscope/ece532.htm

Here is a project idea for a number of ECE532 groups to do.

The project is to create a virtual desktop drum kit from a video camera and a desk-mounted microphone (e.g. a microphone that is touching, tapped against, or resting upon a desktop).

The camera is mounted above the desk, looking down the desk.

The camera tracks the location of one or more points-of-contact with the desk, and the microphone picks up the sound made by tapping the desk (e.g. tap the desk with a drumstick, or fingernail, or the microphone itself).

Depending on the coordinates of the point-of-contact, the sound is changed. The sound comes from the actual sound made by hitting the desk. This sound is modified depending on where you hit the desk.

Below, see a video of the system in operation, where the passband frequency is made to depend on the left-to-right location where the desk is hit (low notes on the left, and high notes on the right), and the gain (sensitivity) depends on the up-down location (low volume at the bottom of the camera's field-of-view, and high volume near the top of the camera's field-of-view).

Note the colored tape wrapped around the end of the microphone. The computer vision system tracks this color to determine the coordinates of the microphone on the desk.

You can see a demo movie on YouTube: http://www.youtube.com/watch?v=IEkKs8Gcg_o, or Glogger: http://glogger.mobi/v/82258.

Glogger Video

Project teams and requirements

This project is modular, so that each separate module can be easily tested separately. One team can do each of the separate modules.

Also, each module can be done by competing teams. For example, two ore more teams can do the vision module, and each of the teams' vision modules can be tested with any one of a variety of competing audio modules, to end up with a best-of-best style of competition or coepetition.

Vision module: Design and implement a system to track location of object in scene, e.g. with the camera mounted above a desk, and looking down at the desk, track objects such as a person's hand or finger or stick hitting the desk. The vision tracking task should be done to display an output in realtime to show that the tracking is working.
Graphics module: display the results of the above vision tracking. The graphics module and vision module can be combined and worked on by the same team, or they can be separate modules if desired.
Audio module: Accepts input from a microphone, processes this input, and then sends the processed result to a loudspeaker. The audio module shall process the input based on physical location, as reported to it from the vision module. There shall be two modes of operation: continuous pitch variation from left-to-right, and quantized pitch, where the pitch is rounded off to one of 12 notes, to obtain a total of 12 frequencies, in Hertz, as follows:
```
A = 220.00
B = 246.94
C = 261.63
D = 293.66
E = 329.63
F = 349.23
G = 392.00
a = 440.00
b = 493.88
c = 523.25
d = 587.33
e = 659.26
```

Physical lab setup:

There are two input devices, camera and microphone:

The vision task is much easier if the camera is fixed, e.g. either the camera can be mounted on a tripod, looking at an object such as a desk surface, or the camera can be mounted above, looking down.
The microphone can be attached to, or touching, either the desk or the drum stick. The microphone can even BE the drumstick! Be careful not to hit it too hard if it is being used as the drumstick.
Some kind of guide helps. For example, a piece of masking tape with the notes "ABCDEFGabcde" written on it so that you know where the 12 notes are located, along the desk, will make it a lot easier to play.

Going further (optional)...

Typically the apparatus uses a fixed field. If you want to make it more challenging, try doing some object-tracking, and then you can make the camera mobile, e.g. as the camera moves around, the vision algorithm needs to track some reference marker(s) and maintain a moving frame of reference. If you do this well, you can use a hand-held or wearable camera.

This system can go anywhere. For example, the surface can be a sidewalk or other space, not merely limited to a desk:

Optionally, the surface can be a spinning cylinder instead of a desktop:

The Friction Idioscope: Polyphonic friction idiophone having continuously variable pitch: Public performance by composer Ryan Janzen:

Note the colored tape on two fingers of the musician. These are unique colors tracked by the vision system.

(Rightmost: Alternate playing style on the spinning cylinder)

(Above: audience participation: Teaching and demonstrating the idioscope...)