"LiqUIface" (LIQUID USER INTERFACE)

A WET USER INTERFACE SUCH AS IMMERSIVE MULTIMEDIATOR INPUT DEVICE WITH ONE OR MORE SPRAY JETS

Canadian Patent 2499784

Steve Mann, 2004

ABSTRACT:

Introduction:

Many traditional user-interfaces, human-computer interfaces, and the like, are cold, mechanical/tegical and lack an expressive continuous ``fluid'' and immersive form of interaction.

Some user-interfaces, such as proximity-based, or antenna-based musical instruments like the Theremin, or ``Doppler Danse'' (Steve Mann "Doppler Danse", Leonardo, Vol. 25, Iss. 1, 1992), achieve the desirable more continuous and immersive form of interaction but lack tactile feedback.

Likewise, ``air typing'' keyboards suffer from similar problems, as do many of the vision-based systems such as David Rokeby's ``Very Nervous System'' (a vision-based system that uses a camera as an input device to control virtual musical instruments, by tracking people's body position in space).

Playing these instruments is very difficult because they have no ``feel''. It is hard to play ``with feeling'' when you cannot feel the instrument.

To overcome these problems, the author has proposed a new kind of user interface, based on interacting with a fluid, to, for example, select from a discrete or continuous alphabet of symbols, using a body of water as an input device.

This new concept provides a more ``fluid'' as well as a more continuous and ``immersive'' multimedia input device that a person can feel.

It is possible with these devices to provide a Theremin-like musical instrument or other input device but one in which the user has some feel, that is provided by a fluid that is instrumental in the interaction. Additionally, the fluid can also be part of a closed-loop interaction, and the liquid user interface can be optically and visually engaging, as well as tactile.

Liquid user interfaces can be incorporated into hot tubs, public fountains, municipal swimming baths, the towers (platforms, such as 5-meter platform or 10-meter platform) at swimming baths, as well as in small portable devices that can be connected to a garden hose, or to a small pump to draw fluid from an ocean, lake, hot tub, bath tub, or the like.

One version of the liquid user interface allows a bather to press down on a spray jet of water, to play different musical notes, the notes depending on how far down the spray jet is pressed.

Another version creates a flat sheet of water that functions as a ``splash page'' to display a web page, projected onto the flat sheet of water, such that a bather can touch part of the sheet of water to select something from the web page.

Another version uses a pool as the splash page, with scolling multimedia matter projected onto the pool, or the bottom of the pool, so that a bather can enter the pool (possibly with the entire body, as, for example, from a 5-meter or 10-meter platform) to select something from the pool.

The liquid user interface allows the user to convey information in a very poetic, expressive, continuous, fluid way, and also for information to be presented to the user in a natural manner.

Single jet fluid user interfaces

Splash, spray, and jets of water tend to look very beautifully intense when backlit, such as when one looks at fountains when the sunlight is behind them, or when people splash into a pool with the sun behind the droplets of water. This is because the droplets behave like a lens, though poorly, in terms of optical quality, but good enough to concentrate the sun's rays over a range of angles where at least some of the water caustics (loci of points of what would be infinite brightness in simple theory) are visible slightly off-axis.

A bather blocks a portion of the jet 110, for example, with their hand 130, to prevent the jet from going beyond a certain height. Generally when a bather inserts his or her hand 130 into the jet, the water will hit the hand and crash back down mostly, with various droplets sprayed in the area.

A light source 120 serves to backlight the jet 110 with respect to an optical sensor 150, in order to measure the height of water column of jet 110 by way of processor 140. A baffle 170, either as part of light source 120, sensor 150, or a combination of both, or as separate elements, keeps light from shining from light source 120 into sensor 150 even though both are opposite jet 110, except, of course when and where jet 110 is spraying. This arrangement causes there to be almost complete darkness where there is no water, but almost complete whiteness where there is water. Thus the user's hand 130 will appear in silhouette, as a black outline, along with the user's body, and other objects, but the water jet, and all the droplets of water, will be bright white.

A satisfactory optical sensor 150 is an ordinary video camera, wherein processor 140 may be equipped with a frame grabber so that it can analyze the image of the backlit jet 110 and determine the highest bright spot, which corresponds approximately to the highest that the jet was allowed to go by hand 130.

A good kind of light source 120 is an aircraft landing light, or a PAR 36 or PAR 38 pinspot light, as are commonly used at rock concerts, and in theatrical lighting, to create dramatic strongly collimated light that emulates natural sunlight. In addition to providing an ideal light source for the computer vision system of sensor 150 and processor 140, such light creates a very pleasant ``summertime'' atmosphere that makes the liquiface comfortable to use in cooler weather, since the strongly collimated light has both an actual (concentration of heat rays) as well as psychological warming effect. Thus playing in the water jet is warm, and the living is easy, in the sense that a bather can feel nice and warm while playing.

In case there is actual sunlight that might illuminate background objects such as glare off windows in the background that could falsely activate the optical sensor 150, a lock-in amplifier may be used in the system to improve signal to noise ratio, or some other form of light modulation may be used, with controller 121. A satisfactory controller 121 is a bidirectional triode thyristor based control, such as by way of a triac-based light dimmer, although IGBT-based light controllers that can generate more arbitrary waveforms are more desirable. An insulated gate bipolar transistor (IGBT) is preferable as it combines the high current density characteristic of a bipolar junction transistor with the fast response and better output characteristic typical of an insulated gate field effect transistor (e.g. MOSFET). The ability to generate arbitrary waveforms is useful for signaling and modulation schemes.

In general, some form of light modulation, lock-in 160, puts a message signal onto the light, so that fluctuations in the light bear the message, whether it be a sinusoidal variation of light output that rides on a DC (Direct Current or other average) offset, or some other encoding. Ideally an adaptive encoding is used, as necessary, to modulate the light for good signal to noise ratio.

In the interest of modulation, a tungsten aircraft landing light, or the like, may be less desirable than a light source 122 that is based on Light Emitting Diodes (LEDs), especially since LEDs can be arranged in a linear array parallel to the jet 110, and also can be toed in to all point at optical sensor 150. This results in more efficient use of light as well as a better exploitation of the lenslike properties of the water jet 110. In so far as the jet 110 behaves optically similar to a glass rod, its properties may be best exploited with the arrangement of sources 122, as high brightness LEDs that have very narrow field of illumination (i.e. that concentrate most of their optical energy along a narrow axis) arranged to point as depicted by the arrows. Thus the lights furthest to the ceiling point downward, whereas the lights near the floor point up. This way, each section of jet 110 is perfectly backlit.

In typical usage, a bather may interact by pressing down a water jet 110, or by swinging the hand at the jet 110, or by taking a swipe at it, or even pulling a piece out of the middle of the jet 110. This action is sensed, and results in some outcome, typically given to the user by way of some feedback. The bather's interaction will typically select from a discrete alphabet of symbols, much like a ``QWERTY'' keyboard on a computer, or an ``ABCDEFGABCDEFGABC...'' keyboard of a piano. Additionally, this alphabet may be a multidimensional alphabet in the sense that each symbol may have meta information in it. On a computer keyboard, when we type the letter ``A'' there is no emotion carried with how hard and angry we hit the ``A'' or when. A piano carries more meta information with the key. Accordingly, the liquiface allows even more meta information than with the piano keyboard. In addition to velocity, force, displacement, and timing profile, the multidimensional alphabet selector of processor 140 can measure subtle nuances of the way in which the letter ``A'' is plucked from the column of water jet 110.

Once a symbol is chosen from the plurality of possible symbols, this symbol may then take action in feedback to the very input device that the symbol was plucked from. Unlike a computer keyboard, or even a piano keyboard, there is a programmable closed loop feedback system that modulates the very input medium.

Consider, for example, a simple task of adjusting water flow using the new input device. For example, the water jet 110 can simulate quantized states of height, and remember height, where the user can adjust the height of the jet, by hand. If the user wants the jet to run low, the user simply pushes the water jet down, and it stays down when the user walks away. If the user wants the jet to come back up, he or she walks over to it again, and grabs the jet 110 and pulls it up. In this application, the jet sprays up until it encounters the user's hand, and then stops. The system can detect the user's hand in a variety of ways, either directly by computer vision, or more preferably, by a better closed-loop process in which:

water jet 110 height is initialized to zero by setting control outputs from processor 140 to control inputs to a combination of nozzle 111 and controller 113 such that the flow is zero or sufficiently low; jet 110 height is incremented by applying ever increasing amounts of flow, by appropriate adjusting of outputs from processor 140 to control inputs to a combination of nozzle 111 and controller 113; the transfer function between a jet control input signal 115 (consisting of control outputs from processor 140 to control inputs to a combination of nozzle 111 and controller 113) and the height of the jet is adaptively modeled (having been determined previously but with adaptation to varying wind, varying water characteristics, and the like); a change in the transfer function characteristics is continuously checked for; if increments to jet control input signal 115 do not result in sufficient increase to the height of jet 110, then it is assumed that the jet is blocked, such as by hand 130; the height at which the jet is blocked to is continuously monitored, while continuously checking for unblocking of the jet; when the jet is unblocked, it is controlled actively to remain at the height at which it was last unblocked. This controlling is done in a closed loop fashion, by maintaining the height with jet control input signal 115 being adjusted to keep the jet at that height despite drift due to changes in water pressure, wind, and the like.

In a preferred version, whenever no bather is detected (i.e. no blockage by hand 130 is detected) the jet 110 rises and falls in a sinusoidally periodic fashion in order appear playful and enticing. In particular, many fountains have rising and falling jets which are found to be quite pleasing. For example, the architectural and artistic focal point of Canada's cultural and civic epicenter (known as ``Times Square North'') in Toronto's Dundas Square is Dan Euser's sculpture which consists of 600 ground nozzles (arranged in 20 grilles with 30 nozzles each) that spray water up in a rising and falling way to mimic the waves on a beach, or the pounding surf of the ocean. (In http://wearcam.org/dundas-square/ there is an explanation of Dundas Square's existing waterplay nozzle jet sequencer.)

This provides a soothing sound that masks traffic, while inviting people to play in the water.

Thus the liquiface can be used in similar kinds of places, to create the same kind of rising and falling surf, but while also being responsive to input from users. The liquiface allows people to sculpt the water, and have fun playing in the fountains while shaping the water flow through play.

In preferred versions, the rise and fall continues but with reduced amplitude, when jet 110 is blocked, and the continued oscillation of height of jet 110 is in the vicinity of the blockage, so that the rise and fall can be used to advantage as a way to more accurately measure the response effect of the blockage.

In an alternative simpler version, the jet 110 can simply be powered more than where blocked, as previously known by the transfer function between signal 115 and height. Thus it can simply then be known that blockage has occurred when the height is less than it should be for a given signal 115.

In either the preferred or simplified version, the signal 115 is preferably dynamically varied against the blockage of hand 130, to provide a time-varying tactile feedback signal to the user. This can be used to send back a ``buzz'' that the user feels upon the hand, much like the vibration of a silent pocket pager or cellular telephone vibrator.

This vibration can vary in pitch, amplitude, waveshape, and chirpiness, etc., as a way of providing user feedback as a variety of user felt symbols, either from a discrete ``dictionary'' or as a more continuously felt form of water expression.

Additionally, since the ground is wet, and since water that has been treated with salts, chlorine, bromine, or the like, is very conductive, a return path through the user may also be detected along with other additional optical properties, such as a change in the color of the jet 110. Especially if the jet 110 is laminar, it behaves like a fiber optic information conduit, and the flesh color of the hand is visible inside the jet, as an additional measurement signal 116. Thus processor 140 has various ways of detecting and measuring the presence, position, orientation, and the like, of hand 130.

Additionally processor 140 measures the way in which water is swept away by hand 130, so, for example, smashing through the jet 110 to push water to the north can result in different action than pushing east, west, or south. Pushing the water up and to the north can take different action than pushing it down toward the ground and to the north. Thus the direction of entry of the hand, as well as the direction that the water actually splashes, can affect the Wet User Interface (WUI), Fluid User Interface (FUI), more specifically, typically a Liquid User Interface, LUI.

In large installations like public fountains that are also important architectural landmarks, it may be desirable to have multiple such jets 110, each differently colored by lights inside nozzles 111. Thus a whole array of beautiful dancing fountains can be set forth that can be choreographed by automation that is adjustable by people playing in the fountains. Each jet can also be a separate symbol area for selecting from a discrete alphabet of symbols out of each jet, or out of the ensemble, or any combination thereof.

In this case, light source 120 may be a source that tracks and follows a bather, to backlight whichever spray jet 110 the bather decides to activate next. Followspot technology in which a spotlight follows a stage performer as he or she moves around, is well known in the art. Thus an automated followspot may be used as both a vision aid for the optical sensor 150, as well as to keep the bather warm, and illuminated, as might be desirable in an interactive art installation. Alternatively if it is desired for the vision light to be invisible, an infrared aircraft landing light, or the like, can be used. A satisfactory such light source 120 is a dichroic PAR 56 (Parabolic Aluminized Reflector size 56) infrared heat lamp or similar light as often used in security applications. This will still serve to keep the bather warm, and to provide illumination for the vision system including optical sensor sensor 150. Thus the bather can freely move around in a large waterplay area and interact with various jets.

For example, all six hundred of the jets in Dan Euser's masterpiece at Dundas Square could, in principle, be made to rise and fall in response to one person inserting their finger into one of the jets. Thus simply touching one small spray of water would result in a chorus of thunder from the other 599 nozzles. Children and adults alike would thus take a moment from their walk through the Square to stop and touch the water, and create dynamic art. This touch to the water could also affect momemtarily the billboards and giant pixelboard displays. While momentarily interrupting the advertising for art's sake, lost revenue could be made up for by the fact that more people would be looking up at the pixelboards because they would be truly interactive extensions of the water spray as their input. For example, pressing down on the nozzle jets could cycle through various ads, making the nozzles function like buttons on a TeleVision remote control. This would create a public interactive waterplay art installation in which the water jets become input devices, much like the keyboards and pointing devices of computers. An omnidirectional jet could also spray in various directions until blocked, and thus direction could replace heigt, or could be another parameter in addition to height, of jet 110. This can be used as a pointing device in place of a computer mouse or trackpoint, and can also be more expressive by including the two dimensions of cursor position in addition to other dimensions like the three dimensional space plus the fourth dimension of orientation, and more (including multidimensional hand position, orientation, etc., not to mention also the wonderful tactile feedback that the immersive nature of water spray provides.

Fountains could also be internetworked, i.e. fountains that are too big to safely play in (such as the fountains in front of the Bellagio Hotel in Las Vegas) could be controlled by a smaller waterplay fountain.

In this way a small child could choreograph the Bellagio fountains by playing in a smaller fountain.

Such a large and expansive show presented from an individual could function much like a karaoke machine, in the manner in which an individual person of ordinary talent could ``give'' an excellent and dramatic show or performance. To the extent that karaoke is defined as a ``method for the intoxicated to embarrass themselves'' (Wikipedia.org online encyclopedia) playing in the fountains can replace the fear of singing in public with the fear of being seen in a bathing suit or underwear in public. In this sense, interactive waterplay performance spaces could be installed in ``watering holes'' and other drinking establishments like restaurants, lounges, hotels, and bars.

FIG. 2 is a diagram depicting a splash screen or splash page 200, that consists of imagery projected onto a sheet of water that is sprayed from a flat nozzle 210. Here a sheet of music is projected and thus presented to the user, as streaming media, in addition to or instead of jets 110. As an addition to jets 110, splash page 200 may fall behind jets 110, as a wet (and thus immersive) projection surface that the user can read, look at, or choose to ignore (and even ``bump'' into, walk through, or stand in). The user can refer to the splash page 200 from time to time in order to help remember the words and notes of a song, such as the 1935 lullaby from Porgy and Bess (Gershwin, ``Summertime''). The notes in the lullaby are bounded from C to C, since the user has selected the key of C minor to match is or her vocal range. This selection has been made with hand 130 to block the height of jet 110C to a height that corresponds with the highest C note, the first note of the song. Thus blocking the water spray forms a liquid user interface into the streaming media.

In response to that selection, the processor 140 has caused the words of the song to display in a manner appropriate for a C minor key, so that the user can sing along with the song, displayed in a karaoke fashion. Additionally, the notes themselves are displayed in a similar way, so that the user can play the notes while singing. The notes are played using the Liquid User Interface, LUI, formed by water and the interaction with the water, i.e. a note is sounded based on when, where, and how forcefully the user touches the water. The nature of the water's path once it is deflected, also affects the way the sound occurs. Not only can the user ``pitch bend'' notes (such as near the end of the lullaby, on the word ``don't'' in ``hush little baby, DO-N'T you cry'' which bends down a minor second) but the user can also affect the nature of the sound by hand position.

In the preferred version, the hand position is sensed by the direction the water sprays off the hand 130, such that the sense of hand control is very intuitive because it is then consistent with the overall philosophy of the Liquid User Interface, LUI.

For example, if the user tips the hand 130 so it is angled up and to the right, the water jet 110C will splash against the hand and water will splash off to the right. Thus, in addition to sensing how high the jet went before it got blocked by hand 130, this direction of splash will be sensed by optical sensor 150, with processor 140.

To play the song, the user pushes the jet down to get lower notes, and lets up on it to get higher notes. The jet 110Ab is shown pushed down to affect a change in pitch downwards by a minor third from where it is in 110C. This change also operates as a closed-loop feedback system, so that pressing down on the jet 110C to jet 110Ab results in a feeling like a fret or similar disturbance at B-flat, Bb, along the way.

More generally, the liquiface includes various forms of tactile feedback, so that, for example, pushing down on jet 110 results in a tactile sensation that is, in this example, achieved as follows, operating in processor 140:

sense height position of hand 130 along the height of jet 110 by way of height sensor 240 which may comprise optical sensor 150 in conjunction with processor 140, or which may be a separate height sensor; compare height with an indexed list of virtual water fret positions; when one of these water fret positions is reached, provide extra stimulation of the hand 130 with the water, through stimulator 241, for as long as hand 130 is within a certain tolerance of the height position; repeat.

The sheet of music is projected onto the sheet of water which may also be a touch sheet, that functions like a touch screen, so that as the user touches the sheet, the coordinates of the place where the sheet is touched are sensed. This can work in addition to jet 110, or it can completely replace jet 110. When not using jet 110, the splash page 200 becomes the primary user interface.

There may also be a switching back and forth between the two modes of user interface, e.g. a novice user who wants the splash page 200 to stay, may interact with it, whereas by an appropriate gesture of pushing away at it with both hands, it goes away. Splash page sensors are present to detect when it is pushed away, either by both hands or the whole body of the user. Thus the splash page can be just an introduction, or for instructions, that goes away when the user is finished with it.

FIG. 3 is a diagram depicting a multi-jet wet-user-interface, Nine jets 310 spray water upward, tilted slightly toward the middle. A ring manifold 300, having a diameter, in the preferred versions, that ranges from 20 inches (approximately 51 cm) to 2 meters, has a Female Garden Hose Thread (GHT) connector 301 on a ``T'' fitting that supplies it with water in both directions. Each nozzle jet 310 is supplied from both directions with water. The entire manifold 300 and jets 310 may be supplied by fresh water from a garden hose, with runoff going to irrigation, such as when playing in a garden, or it may be supplied by water from a batter operated pump such as a bilge pump used in marine applications. Capacities of bilge pumps are usually specified in gallons per hour; preferred versions of the liquiface work well with bilge pumps in the capacity range of 500 GPH to 2000 GPH, with higher capacities being sometimes preferable for dramatic show, of the spray of the liquiface, but not usually necessary for good functioning. In particular, the preferred capacity is around 1000 GPH.

In a preferred version of small size (e.g. 20 inches, or approx. 51 cm diameter), the pipe size for the curved pipes of manifold 300 is 1/2 inch plumbing which is equivalent to 5/8 inch refrigeration (plumbing is specified as inside diameter but the refrigeration industry specifies by outside diameter). This size is suitable for being worn over the right shoulder, so that the high notes are to the right, and near the top, and the low notes are to the left and near the bottom, in front of the body of the user.

Jets 310 may be made by cutting an appropriately curved pipe into sections and then re-joining them with reducing ``T'' fittings. Suitable reducing ``T'' fittings are 1/2 inch through to 1/4 inch (5/8 inch through to 3/8 inch in refrigeration sizing). A piece of size 3/8 plastic toilet or sink hookup sleeve fits nicely into each opening in the reducing ``T'', with a good friction fit. Thus a module 311 may be built around the plastic sleeve, and inserted into each hole as needed. In this way, an entire module can be quickly replaced in the field. Module 311 is a flow sensor, and may also perform the role of an output device, such as flow control, or other stimulus to the user. At the very least, module 311 should measure the amount of flow, and thus facilitate a continuous fluid user interface. In this particular version, each jet is associated with a different note. Each note may be thought of as a symbol selected from a discrete alphabet of symbols, and each jet may be considered therefore as a symbol area, or a region around a symbol area, in which the symbol is selected by having the user enter this area. Movement between symbol areas results in the generation of an ordered list of symbols that are also annotated. The annotated ordered list uses annotation to record time of entry and exit to and from the area, and various attributes of how the entry and exit was made.

Each note sounds in amplitude that depends on how far down the jet for that note is pressed. For example, if pressing down the ``C'' jet, the C note will sound and the sound will grow louder as the jet is pressed further down. To play a C-Major cord, the C, E, and G jets are all three blocked together. To play a C note with a C-Major to accompany it, the C jet may be blocked entirely, and the E and G only blocked slightly, or the fingers may hover above the E and G, just lightly in the spray, whereas the finger may reach deeper down into the spray of the C jet.

It is preferable that the notes are activated by displacement rather than velocity, but if velocity is desired, the height value may be differentiated by processor 140. Since it is easier to take reliable derivatives than integrate reliably (due to the presence of baseline drift), the absolute height measurement of each jet is preferable to the velocity information.

The default setting for the instrument is also in displacement, and behaves much like a church organ, which is also easier to sing to than the more percussive and more ephemeral sound of a velocity based (and percussive) instrument like a piano.

The device may function as a direct user interface to a real organ such as a real pipe organ, or it may activate other synthesis devices by way of Musical Instrument Digital Interface (MIDI) output, serial output, wireless control, and the like. Because the manifold 300 is made of copper, it can advantageously shield the system, and thus the fact that copper is a common plumbing material as well as the most common electrical conductor, is advantageous. Internally a loop antenna 313 can still transmit through the copper since magnetic fields can there outwards propagate. Loop antennas, unlike dipole antennas, provide operation despite the copper shielding which serves to keep electrical noise out of the system.

Ordinarily, water from city water pressure mains is at much higher pressure than needed for the instrument. City water pressure is typically two to four atmospheres. One atmosphere is approximately equal to 10.3 meters of head, i.e. approximately equal to the maximum height of head that people enter swimming baths from (e.g. municipal swimming baths that have towers with 10-meter platforms). Thus water pressure is approximately two to four times higher than that experienced while bathing in the most extreme way at a pool (i.e. approximately 50 kilometers an hour impact with water after departing from the 10-meter platform).

To convert from the approximately 20 to 40 meter head, down to the lesser pressures needed for the apparatus, a flow control valve, or pressure regulator may be uaed.

However, it is preferable to recover that energy and use the energy to power the instrument, realizing the sheer magnitude of this energy that would otherwise go to waste. Thus an energy recovery module 302 may power the instrument.

Novice players may apply adhesive tape labels 312 to each such module, to label the notes. Alternatively liquid crystal displays in the modules may interactively display the notes as well as learning information for lessons, such as highlighting which note to play next.

The water jets may also be output devices either by illumination, color, or by tactile vibration, spray height variation, and the like. In a preferred version, all of the jets are green when they are active idle. To make the instrument easier to play, jets that are not used in a particular song may be shut off. Alternatively, it is preferable to keep all the jets running for aesthetic value, but only illuminate the ones that are part of a given song. For example, to play ``Amazing Grace'' (words, John Newton 1779, music, Carrell and Clayton, 1831) only six of the jets, namely C, D, F, G, A, and C, are needed. The others may be shut off, or their lights shut off, and a single green light may guide the user through the song, to light up the jet that the user should hit next.

In fact jets could go all the way around the whole circle, even in the back where it is difficult to reach, while only the front jets (easier to reach) would need to be used to play music. Alternatively, the space not used by jets at is used for indicia 303 such as trademark information e.g. as shown ``FROLICious FUNtain (TM)'' along with usage instructions, and the like.

Various modes such as teaching mode, and song to learn, are selected by holding down different combinations of jets at power up time. Unused chord combinations are used as symbols to type messages into a computer to select processor 140 operation. Water typing modes are selectable to type in song names, search parameters, etc., but the water typing is not so bad as mid air typing. It is known that air typing is difficult, like playing air guitar, since there is no feedback but water typing (or water guitar) are made easier by the feedback.

If learn mode is shut off, all jets glow green, until pressed down. As the hand enters the spray, the jet turns yellow, then orange, then red. This is by way of a 3-terminal LED that has red and green elements, and the LED also forms part of the computer vision system that sees the water spray flow diverted.

Thus the LED serves double duty as the light source for the vision system and the illumination. Since the illumination is nice and subtle it need not be visible to others, but can be if desired, by playing in a darkened room. In this way, teach mode can be hidden from others, so that in a liquid interface karaoke setting, only the player can see the prompting.

Alternatively, the apparatus of Fig~3 can be used as an interface to other equipment, such as a computer. For example, the apparatus may be used by a disc jockey to play pre-recorded music. By spinning the hand around in the circle of water jets, the virtual disk is spun to ``scratch'' or timewarp or modulate the music. Two such liquid user interface rings, i.e. two manifolds 300 may be used to simulate two turntables, to create a virtual mixing platform. Since many disc jockeys already perform in their boxers or briefs, and since many of the dance clubs have a ``foam party'' or ``beach party'' theme (e.g. in many clubs the electrical systems are already wet-safe) the apparatus of the liquiface may find many applications in such dance and performance oriented spaces.

FIG. 4 illustrates the vacuum exclusion principle of the multijet system of Fig~3. Hand 130 descends to partially block one of jets 310, thus reducing the amount of water that comes out of that jet. The lower the hand 130 descends, the less water can come out of the jet 310 that is under the hand. At least some of the water that would have come out that jet goes out the other jets. Typically blocking one jet results in increased flow out of the other jets. Additionally, each jet has a ``T'' fitting 400, so that when one jet is blocked water gushes out of the blocked T fitting side discharge 410. Note that ``T'' fitting 400 is not a reducing ``T'' fitting, although it may be spliced in by way of an additional reducing ``T'' fitting 499. Also it is important that jets 310, when not blocked deliberately by the user, do not offer significantly more resistance to water flow than discharges 411.

Interestingly, no water comes out of the other side discharges 411. In fact, the more jets that are blocked, the faster the water gushes out their side discharges and out of the other jets, but at the same time, an even stronger vacuum is created on the unblocked side discharges 411. Thus initially, where all the side discharges are under slight vacuum when none of the jets are blocked, the unblocked side discharges 411 are pulled under even greater vacuum when more flow comes out the unblocked jets, either because other jets are blocked, or when water pressure increases, or the like.

This system works very well, so long as the ``T'' fittings 400 are small compared with the size of the manifold 300. Various kinds of flow meters, pressure meters, or the like, attached to discharges, will work quite well. In a preferred version, the discharges point to the center, and a flow meter is used, because this allows the bather to get splashed by the discharges, and thus receive tactile feedback. In this way, blocking the jet with the finger or hand results in the body getting splashed by discharge. This often improves the ability of the player to become one with the machine of the instrument. A satisfactory flow meter is a vision system that uses a discharge lens property. Light sources 120 are blocked from shining into optical sensors 150 by baffles 170. Each of the nine discharges has one baffle 170, one light source 120, and one optical sensor 150. In this multijet version, an individual photoresistor is used for each discharge, rather than a single camera. The circular array of nine photocells (photoresistors) may be thought of as a nine pixel camera if desired, from a conceptual point of view. When water flows out through discharge 410, the spray forms a crude but sufficiently effective lens that light rays from source 120 reach sensor 150. Photocells of sensor 150 should point downward and the lights 120 should point up for 2 reasons:

ambient light tends to come from above, and thus downward facing sensors 150 will be less adversely affected by the ambient light that might otherwise result in false triggering of musical notes; light sources 120 have the advantage of being visible to the user when they are facing upwards.

A satisfactory photocell is a cadmium sulphide photoresistor such as the kind used in dusk to dawn electric eye lights. Such a photocell may be connected directly into the matrix of most musical keyboards to activate a note, since flow results in light diverted to an otherwise baffled photocell, and since light results in less resistance (more conductance), which is like pressing a key on a keyboard.

The same is true of computer keyboards, so the apparatus can be directly connected for water typing or playing music with little or no interface hardware or power supply needed by the input device itself, other than for light, which could, in principle, be just ambient light if the photocells were moved down to the bottom.

However, in preferred versions, for resistance to moisture effects, a lower impedance threshold is desired, and for other reasons (e.g. more bits of amplitude control) an active powered system is preferred. In some preferred versions, directional photodiodes or phototransistors are used for sensors 150. Typically a 7 bit precision is used to quantize the amount of flow, although greater precision and a lookup table are sometimes desired, to shape the amplitude response of the instrument comparametrically. The displays for note labels 312 on each note are preferably square computer displays, so they adapt well to a comparagram editor, for setting the note's amplitude response.

In other versions, an additional vision sensor overall such as an overhead camera for all nine jets, or an additional sensor on each jet, or a different design is used to measure direction of water spillage, slappage, etc., so that the jets can be played more expressively. For example, to play along while singing the word ``don't'' in ``hush little baby, DO-N'T you cry'' or ``standing'' in ``with mommy and daddy standing by'' (Summertime, 1935), one presses down on the the high C jet and sweeps the water to the left, toward Bb, prior to laying into the Bb from the other direction. The result is the nice sounding down-chirp that so expressively captures the closing words of the the lullaby.

FIG. 5 is a diagram depicting a multi-jet liquid user interface fully contained inside a copper pipe manifold 300. This provides a very simple aesthetic in which the instrument becomes a nice looking copper sculpture.

Sensors 550 are simply pressure switches rather than optical sensors. An acceptable sensor is a miniature version of something found in a Reznor duct furnace for checking to make sure the air is flowing through the duct before the natural gas is switched on. Any switch in the sensitivity range from 1 to 20 inches of water column will work quite well. The switches of sensors 550 and wiring 551 are inside the manifold 300. Holes 510 in front of each port of sensors 550. Water pressure supplied to the pipe forces water out all of the holes, creating a vacuum on all of sensors which keeps them from activating, except when a user wishes to block one or more of the holes in which case positive pressure activates sensor 550 to produce a user input. Preferably sensors 550 are back vented by vents 511 so they can see atmospheric air pressure as a reference pressure.

The resulting version with holes 510 can be played like a penny whistle, tin flute, or other similar wind instrument, except that it is a water instrument interface played by blocking water from coming out of certain holes. In particular, the processor 140 can be programmed to operate so that hole fingering is that of any preferred instrument of that type. Instead of the circular manifold 300, a straight manifold can be used, and the different size holes of a penny whistle or tin flute can be used, and thus preserve the familiar fingering of that instrument.

FIG. 6 shows how water may be diverted from main jets 610 to smaller side jets 611, so that a more immersive multimedia input device may thus be created. For example chords may be activated with one finger, by blocking multiple jets at the same time. By skipping by twos, harmonious groupings are possible so that sloppy fingering results in good sounds that harmonize well, in much the same way that a harmonica is designed so that sloppy playing results in good sound by blowing through adjacent holes to get harmonious sounds.

Thus in this version of the liquiface, harmonious groupings of of nozzle jets 611 facilitate easy chording.

FIG. 7 shows some examples of fingering positions for a popular song ``What shall we do'' (song of unknown authorship). Words to the song, chord suggestions, etc., or product information such as labeling (e.g. ``Playing in the fountains (TM)'' may be displayed in display field 730.

Multiple jets 611 can be simultaneously covered with one finger 710, to make a D minor chord. Finger 710 is shown as a solid line. By moving over and down, the second row of nozzles can be activated in similar grouping to get a C-Major chord with finger 720. A drawback of this design is that it is hard to get to the top row without affecting the bottom row slightly, especially when jets 611 shoot high.

Accordingly, preferably nozzle groups are brought closer together and re-arranged, so that fingertips can be inserted into the spray. Finger 711 plays a D minor chord, and any three jets in which there is one jet closest to the user plays minor. Any three jets with two toward the user plays major, such as the C Major of finger position 721. A display area 740 prints the words to the song and shows fingering positions for teaching mode.

FIG. 8 shows an example of a very simple version, more of an illustrative early version than the preferred version, since it shows some aspects of the liquiface in a way that is easy to understand. For simplicity (but not to suggest it is the preferred version) the input device of Fig~5 is considered for sensor 550 shown. Ordinarily, a naturally open or naturally closed switch sensor 550 would bridge over and give erroneous results due to conductivity of treated water. Thus to attain immunity to water conductivity, sensor 550 is has both its naturally open (N.0.) contact 809 as well as its naturally closed (N.C.) contact 800 in use, in addition, of course, to its common (C.) contact 805. These contacts are connected respectively to the ground 810, middle 815, and tip 819 of a stereo 1/4 inch plug 830 by wire cord 840. Cord 840 is preferably a round flexible black wire in which the ground is a shield around the other two wires.

A number of jacks (sockets) are provided for the insertion of a number of plugs 830. The number of plugs 830 is typically equal to the number of jets which is typically 9 for a simple instrument that can be played by children or inexperienced users, though more sensors may be used for more complicated pieces.

The 9 plugs 830 can be plugged into some of the sockets 850 on processor 140 to select a key in which to play. For example, to play in C Major, or D dorian minor, the nine plugs are inserted into the C, D, E, F, G, A, B, C, and D sockets 850. The 3-wire interface allows processor 140 to detect which notes are plugged in and to display this information such as on LED 861, or alphanumeric or computer display 870. One way for processor 140 to display its knowledge of which sensors were plugged in, is for it to display some possible songs that can be played in the key so selected.

Light source 861 will illuminate when the input line 865 is pulled low by plug 830 connection from middle 815 to ground 810. This connection will also very decisively short the coil of relay 862, to definitely keep it off.

At this point, blowing into port 820 of sensor 550 will test the system and play a note, so even without the availability of water, the instrument can be tested by blowing into the holes of each note. Blowing into port 820 will cause contact 805 to disconnect from contact 800 and then to connect to contact 809. This will lift the coil of relay 862 to energize it, along with energization of the LED 860. Preferably there are similarly two LEDs inside each jet 310, and preferably the LED 860 that is on when the jet is blocked is red, and the LED 861 that is on when the jet is not blocked is green. When the jet changes color to red, it will still be visible through the flesh of the user, since flesh is more red in color than green. Moreover, in transmission, flesh is very red (i.e. it is somewhat translucent in the red). Thus the finger on the blocked jet 310 will be visible in red to affirm as a form of visual feedback that the instrument is working and has responded. This is useful when using a music synthesizer with slow attack, so that the user can know exactly when a note is actuated, i.e. that the jet 310 has been blocked or depressed enough to be considered a note-on, prior to even hearing the note.

It is essential to have a break before make type of switch, to avoid shorting the power supply, but most pressure switches are of this type. Preferably the switch can be modified to remove hysteresis or deadband, so that it can function as a velocity sensing switch, so that processor 140 could determine how fast the water jet is blocked, to adjust the amplitude of the musical note in response to how hard each jet 310 is hit. However, this feature is not shown in the simple relay embodied in processor 140, in order to make the diagram simple. In implementation this velocity is found (calculations in processor 140) by computing the time between break and make.

FIG. 9 depicts a platform 910 for immersive multimedia, with a fluid user interface in which the user's entire body, not just his or her fingers, is used in the immersive multimedia input device. This version of the liquiface may be installed at a municipal swimming bath 900 where there is a tower, or at locations without a tower, since it is also possible for people to enter from a springboard, or even to enter just by jumping off the side of the pool (0-meter platform) or to interact by frolicking in the pool, or with similar immersive multimedia in a lake or ocean.

In the version shown, the user 901 climbs the tower, up onto the platform 910. A satisfactory platform is the standard 5-meter platform (for approximately 1 second in the air, and 36 km/hour speed of entry into the water user interface) or 10-meter platform (for approximately 1.4 seconds in the air, and 50 km/hour speed of entry into the water user interface) that may be found at many municipal swimming baths, university pools, and the like, as are often referred to as ``olympic pools'' (because entering a pool from a height of 10 meters is an olympic event, as well as a form of recreation for children).

A fun and playful splash screen or splash page 200 is projected from a light source 120 hung from the bottom of the platform, as projected rays 920 that span all or some of the pool area. Most platforms are cement with railings made of structural pipe fittings with size 8 being the most common size of structural pipe fitting found on platform railings. Various fittings commonly used in the theatrical lighting industry may be used to temporarily attach light source 120 to the bottom of the platform with appropriate rigging, using Alvin pipe clamps (e.g. from Alvin Industrial Sales in Canada). Standard safety procedures for rigging are used, i.e. safety chains on the light source in case it works loose over the years when a temporary one-day installation might be kept for 10 or 20 years in change of mind. A satisfactory light source for this version of the liquiface is a high power data projector, or a laser based vector graphics projector. The projector projects streaming media such as a scrolling sign of splash page 200, with the words rolling down the musical scale, so that user 901 can select a key in which to later play the music. The user 901 selects the key by departing from the platform.

When departing from a platform, bathers typically insert their hands into the pool 999 first, so that the water hits them from above. In this way the user's body is upside down at time of impact, so that, all things being relative, in human-centered coordinates, the water in the pool pours down on top of the user. This water-from-above results in an experience similar to (though much more extreme) a shower, where water falls down on top of a person.

In this case, usually the hands 130 of user 901 will be the first body part to hit the water (hands are usually extended to cut through the water to avoid getting hit on the head with the water).

Optical sensor 150 has a field of view that includes rays 950 to see some or all of the pool 999 water surface and below, and is arranged to detect when and where the user's hand 130 hits the water. This point of contact selects from the splash page 200 in a manner similar to that shown in Fig.~2, except that the wall of water or sheet of water of Fig. 2 is now laid out flat and is the surface of the pool 999.

In addition to being a splash page for streaming media, the pool 999 also functions as an immersive multimedia environment, because the sensor 150 can continue to observe user 901 in descent into the water, and the manner of entry can be used to select or affect options, or can be otherwise used as a fluid user interface (i.e. as an input device) to a computer processor 140 or other input system or systems of the liquiface.

The top of the platform 910 may also be used as a display surface 911, to display messages for the bather, such as cautionary notes if the system observed that bather in a suboptimal entry on a previous try, or to display emergency messages, since it is hard to hear lifeguards, etc., from way up on the platform. When surface 911 is not being used for emergency messaging, it may display fun product information such as shown, ``The key to good music is to play in the water (TM)''.

In one application, this version of the liquiface may be used to set the key of another instrument in a water park, for example. Thus a user can use the 10 meter platform as an input device to choose the key that a nearby fountain will then play in. This creates a fun and playful way of having an input device for setting parameters for playing music.

Additional multimedia spaces include areas around the pool. For example, when the system detects the presence of a bather on the tower (i.e. on the way up) or up on the platform, a cautionary message on deck surface 922 may be projected to warn other bathers not to enter the pool at that time. This feature may be added simply by extending the field of coverage of light source 120 so that it includes rays of light to ray 921.

In addition to the splash screen of splash page 200, other playful elements may be included. For example, a fish-based screen saver may operate at idle times, or interactive gaming elements 902 may be displayed on the bottom of the pool. Various games include ``catch a fish'' in which a user needs to land on a fish, as well as ``avoid the fish'' in which a user needs to not land on a fish. The latter game is preferable to the former, to teach bathers the safety skills of avoiding collisions with other bathers.

Splash pages 200 may be projected on the bottom of the pool, or on the surface, or simultaneously on both, or on various intermediate mid-water areas, through the use of focus, and the like. A large aperture projector can have limited depth of field, and when a black background with light colored lines is used, it can focus on the bottom without affecting the surface, or vice versa. Two projectors, one for surface, and one for bottom can also be used together. Thus surface game elements such as element 903 may be combined with bottom game elements 902.

To enhance surface visibility the bubbler feature of most pools can be switched on or modulated. Many swimming baths have bubble jets to reduce the severity of impact when bathers land poorly, and these bubbles could be modulated as projection surface. To make the immersive multimedia interactive, the bubble jets can be dynamic, with the splash pages 200. Also, if visibility of the bottom is desired, the bather can be tracked, and a burst of bubbles delivered just before the bather hits that water.

This results in loss of visibility of the bottom on the descent, but this is not such a bad thing. Bathers generally learn that looking down into the water, whether in head first or hands first entry, often results in two black eyes and a badly bruised face. Thus it is even desirable in the liquiface to blank out the splash screen 200 (such as by turning off light source 120) as soon as a bather departs from the platform.

Where light source 120 is part of the vision system for sensor 150, and the blanking feature is desired, the light may change to infrared or other invisible light source during the blanking interval, or for the whole time so as not to rely on visible light.

The baths are a very social place, and particularly the towers, since the sequentiality of bathing is there mandated by safety, so that bathers line up to use the platforms, and there is time for idle chat while standing in line. Additionally, the serialization of bathing (sequentiality) gives rise to a phenomena in which bathers are each on display upon the elevated platform, one at a time.

The liquiface can thus be used for adding fun, games, music, or other multimedia elements to such ritualized or social bathing.

The pool 999 need not be limited to a rectangular olympic style pool. For example, a round pool could be built, with an offset platform that hangs over it like the tone-arm on a record player. A projection of a spinning roulette wheel could then be the streaming media of splash page 200, such that user 901 becomes the roulette ball. In this way, a person could place a bet by entering the pool. Processor 140 determines the landing time and place of the first part of the bather's body, and the spinning of the roulette wheel then would stop exactly when user 901 hit the water. By continued display of a stationary roulette wheel, the user and others could wait in suspense until the water ripples from the splash of the bather fade out, to reveal a clear image of where the bather hand landed on the virtual wheel. This adds the thrill of the platform to the thrill of gambling, and turns a fun and silly game like roulette into a fun and silly and splashy game.