Video mediation

The compute-power required to perform general-purpose manipulation of color video streams was, at the time of the experiments (1994) too unwieldy to be worn in comfortable clothing (although more recently, a clothing-based apparatus has been constructed to facilitate general-purpose reality mediation without relying on broadband wireless communications). In particular, a remotely-located compute-engine was used by establishing a full-duplex video communications channel between the wearable apparatus and the host computer(s). A high-quality communications link (called the `inbound-channel') was used to send the video from the camera(s) to the remote computer(s), while a lower quality communications link (called the `outbound channel') was used to carry the processed signal from the computer back to the head-mounted display (HMD). This apparatus is depicted in Fig 3.

  

Figure: Implementation of tetherless computer-mediated reality for use as a personal visual assistant. The camera sends video to one or more more computer systems over a high-quality microwave communications link, called the `inbound channel'. The computer system(s) send back the processed image over a UHF communications link called the `outbound channel'. Note the designations ``i'' for inbound (e.g. iTx denotes inbound transmitter), and ``o'' for outbound. The term `visual filter' refers to the process(es) that mediate(s) the visual reality and optionally insert virtual objects into the reality stream.

Ideally both channels would be of high-quality, but the machine-vision algorithms were found to be much more susceptible to noise than was the wearer's own vision. Originally, communication was based on antennas that the author had installed on various rooftops, but presently, the mediation may also be achieved completely on local (worn) processors.

The apparatus (Fig 2) permitted one to experience any coordinate transformation that could be expressed as a mapping from a 2D domain to a 2D range, and the apparatus could do this in real time (30frames/sec = 60fields/sec) in full color, because a full-size remote processing engine was used to perform the coordinate transformations. This apparatus allowed experiments with various computationally-generated coordinate transformations to be performed both indoors and outdoors, in a variety of different practical situations. Examples of some useful coordinate transformations appear in Fig 4.

  

Figure: Living in coordinate-transformed worlds: Color video images are transmitted, coordinate-transformed, and then received back at 30 frames per second - the full frame-rate of the VR4 display device. (a) This `visual filter' could someday allow a person with very poor vision to read (due to the central portion of the visual field being hyper-foveated for a very high degree of magnification in this area), yet still have good peripheral vision (due to a wide visual field of view arising from demagnified periphery). (b) This `visual filter' could some day allow a person with a scotoma (a blind or dark spot in the visual field) to see more clearly, once having learned the mapping. Note the distortion in the cobblestones on the ground and the outdoor stone sculptures.

In order for the `personal visual assistant' to be useful, it will need to be small, lightweight, tetherless, and unobtrusive. Many of the design issues have already been dealt with, but much remains to be done. For example, although the current body-worn multimedia system has the capability to mediate the video stream locally, eye tracking capability built right into the head-mounted display would greatly advance the research toward a system that will hopefully someday be of widespread benefit to the visually challenged.