Lightvectoring (painting with lightvectors):

Anti-homomorphic vectorspaces as visual art

Steve Mann
mann@eecg.toronto.edu
University of Toronto,
10 King's College Road,
Toronto, Ontario,
Canada, M5S 3G4

Abstract

The research described in this paper arises from the author's work in designing and building a wearable graphics production facility which he has used to create a new kind of visual art over the past 15 or 20 years. This work bridges the gap between computer graphics, photographic imaging, and painting with powerful yet portable electronic flashlamps. Through the use of an anti--homomorphic filter, principles of anti--homomorphic superposition and anti--homomorphic vector spaces are used to create a new kind of image content within a community of users sharing a commonly altered visual perception of reality. The apparatus for altering the visual perception of reality forms a useful interface to collaborative production of artwork that is viewable in realtime on the Internet while it is being created.

Keywords:

Comparametric Equation, Comparametric Plot, Image processing, Personal Imaging, Wearable Cybernetics, Wyckoff principle, photography, lightspace, photoquantigraphic imaging

Photographic Origins of Wearable Computing and Augmented/Mediated Reality in the 1970s and 1980s

The Reality Mediator (RM) has, in recent years, received considerable attention in military applications as well as in industry. However, the original Personal Imaging application of the RM (cite mann97, IEEE Computer) was an attempt to define a new genre of imaging and create a tool that could allow reality to be experienced with greater intensity and enjoyment than might otherwise be the case.

This effort also facilitated a new form of visual art called Lightspace Imaging (or Lightspace Rendering) in which the author chose a fixed point of view for the camera, and then, once the camera was secured on a tripod, the author walked around and used various sources of illumination to sequentially build up an image layer-upon-layer in a manner analogous to paint brushes upon canvas, and the cumulative effect embodied therein. The author's early 1980s attempts at creating expressive images using the personal imaging system he developed in the 1970s and early 1980s are depicted in Figure 1a and Figure 1b.
CAPTION: A common criticism of the camera is that it reduces our enjoyment of reality because we become so busy working the camera that we neglect to appreciate reality itself. However, a goal of Personal Imaging is to create something much more like the sketch pad or artist's canvas than like the camera in its usual context. The images produced as artifacts of Personal Imaging are somewhere at the intersection of painting, computer graphics, and photography. (Figure 1a) Notice how the broom appears to be its own light source (e.g. self-illuminated), while the open doorway appears to contain a light source emanating from within. The rich tonal range and details of the door itself, although only visible at a grazing viewing angle, are indicative of the affordances of the Lightspace Rendering method. (Figure 1b) architecture offers a unique perspective, which can also be illuminated expressively. (C) Steve Mann, 1984.

Throughout the 1980s, a small number of other artists also used the author's apparatus to create various lightpaintings. However, due to the cumbersome nature of the early WearComp hardware, etc., and the fact that much of the apparatus was custom fit to the author, it was not widely used over an extended period of time by others. However, the personal imaging system proved to be a new and useful invention for a variety of photographic imaging tasks.

To the extent that the artist's light sources were made far more powerful than the natural ambient light levels, the artist had a tremendous degree of control over the illumination in the scene. The resulting image was therefore a depiction of what was actually present in the scene, together with a potentially very visually rich illumination sculpture surrounding it. Typically the illumination sources that the artist carried were powered by batteries. (Gasoline powered light sources were found to be unsuitable in many environments such as indoor spaces where noise, exhaust, etc. were undesirable.) Therefore, owing to limitations on the output capabilities of these light sources, the art was practiced in spaces that could be darkened sufficiently, or, in the case of outdoor scenes, at times when the natural light levels were least.

In a typical application, the user positions the camera upon a hillside, or on the roof of a building, overlooking a portion of a city, usually having an assistant oversee the operation of this camera. The user may then roam about the city, walking down various streets, and use the light sources to illuminate various buildings one-at-a-time. Typically, in order that the wearable or portable light sources be of sufficient strength compared to the natural light in the scene (e.g. so that it is not necessary to shut off the electricity to the entire city to darken it sufficiently that the artist's light source be of greater relative brightness) some form of electronic flash is used as the light source. In some embodiments of the personal imaging invention, an FT-623 lamp (the most powerful lamp in the world, with output of 40kJ) is used, housed in a lightweight 30 inch highly polished reflector, with a handle which allows it to be easily held in one hand and aimed (Figure 2) CAPTIOM: Wearable Computing and Augmented/Mediated Reality for Computer Supported Collaborative Photography}: A portable electronic flashlamp is used to illuminate various buildings such as tall skyscrapers throughout a city. The viewfinder on the helmet displays material from a remotely mounted camera with computer generated text and graphics overlaid in the context of a collaborative telepresence environment. An assistant at the remote site wears a similar apparatus with a similar body-worn backpack-based processing system.

Lightvector amplification

The communications infrastructure is established such that the camera is only sensitive to light for a short time period (e.g. typically approximately 1/500 of a second), during the instant that the flash lamp produces light. In using the personal imaging invention to selectively and sequentially illuminate portions of a scene or object, the user will typically point the source at some object in the scene in front of the camera, and issue a command through the wearable computer system. A simplified diagram of the architecture is depicted in Figure 3: (a) (b) CAPTION: Artist's ``paintbrush'' for Computer Supported Collaborative Photography}: The artist issues commands to a remote camera using a data entry device while monitoring the resulting pictures and overlaid text+graphics on a head mounted display. Here a simplified diagram is used to illustrate signal routing. (a) When the artist issues a command by switch closures (S), a signal is sent through an INBOUND communications channel, depicted as transmitter I.Tx, to the central base station (b) and is received by the inbound receiver, denoted I.Rx. This initiates frame capture (depicted by solenoid S) with a computer system located at the base station. At the correct instant during frame capture, a signal (depicted by flash sync contacts X) is sent back by the camera's outbound transmit channel O.Tx, to the artist (a) and received by the artist's light source synchronization receiver, O.Rx. This activates FLASH through its synchronization contacts denoted X. Light then emerges through the OPENING and illuminates the scene at the exact instant during which the camera's sensor array is sensitive to light. A short time later, the image from the camera base station (b) is sent via the OUTBOUND channel to the artist (a) and is displayed on the artist's head-mounted display, overlaid with a calculated summation of previous differently illuminated images of the same scene and appropriate graphics for manipulation of the summation coefficients.

The receiver at the camera is typically embodied in a communications protocol, which in newer embodiments of the invention runs over amateur packet radio, using a terminal node controller in KISS mode (TCP/IP). In the simple example illustrated here, the RECEIVER activates shutter solenoid S; what is depicted in this drawing is approximately typical of a 1940s press camera fitted with the standard 6 volt solenoid shutter release, while in actual practice there are no moving parts in the camera, and the shutter is implemented electronically. The camera is sometimes designed so that it provides a sync signal in advance of becoming sensitive to light, so that pulse compression may be used for the synchronization signal.

The wearable computer is generally distributed throughout a heavy black jacket, and the artist will typically wear black pants together with the jacket, and hold the light source depicted in Figure 3a, using a black glove, although this is not absolutely necessary. Accordingly, the housing of the lamp head will often be painted flat black.

In this manner a comparatively small lamp (small compared to the scale of a large city, e.g. a lamp and housing which can be held in one hand) may illuminate a large skyscraper or office tower in such a manner that the lamp appears, in the final image, to be the dominant light source, compared to fluorescent lights and the like that might have been left turned on upon the various floors of the building, or to moonlight, or light from streetlamps which cannot be easily turned off.

Typically, the artist's wearable computer system comprises a visual display which is capable of displaying the image from the camera (typically sent wirelessly over a data communications link from the computer that controls the camera). Typically, also, this display is updated with each new exposure. The wearable computer is generally controllable by the artist through a chording keyboard mounted into the handle of each light source, so that it is not necessary to carry a separate keyboard. In this manner, whichever light source the artist plugs into the body-worn system becomes the device for controlling the process. An example of an input device built into the handle of a smaller electronic flashlamp appears in Figure 4. CAPTION: Typical ``Keyboard'' and ``Mouse'': The portable electronic flashlamps used in conjunction with the invention are each equipped with a data entry device and cursor pointing device (useful in computer supported collaborative photography). Here a 1970s system is shown with five microswitches operable by the right hand while simultaneously holding and aiming the lamp. Thus it is possible that one can be walking and entering data at the same time, or even be climbing a ladder or rope, and stop to enter data. The pointing device (joystick) is operated with the left hand so that when ``typing'' and pointing, both hands are occupied. However since the pointing device is not used frequently the apparatus is usable with one hand most of the time. The lamp pictured in Figure~\protect\ref{fig:623} has a similar user-interface (``keyboard'' for right hand and ``mouse'' for left hand), except that the left hand device is also built in proximity to a separate handle/grip to facilitate two-handed grasping of the lamp in conditions of high wind. (The 30 inch reflector acts like a sail, so it needs to be held with both hands in windy weather.) Otherwise the two lamps have the same user-interface. Consistency of user-interface was an important human-factors consideration.

Typically exposures are maintained as separate image files overlaid on the artist's screen (head mounted display) together with the current view through the camera. The exposures being in separate image files allows the artist to selectively delete the most recent exposure, or any of the other exposures previously combined into the running ``sum'' on the head mounted display (``sum'' is used in quotes here because the actual entity, a summation in homomorhic vectorspace, will be described later. Additional graphic information is also overlaid to assist the artist in choice of weighting for manipulation of this ``sum''. This capability is quite useful, compared to the process of painting on canvas, where one must paint over mistakes rather than simply being able to turn off brushstrokes or adjust the intensity of brushstrokes after they are made. Furthermore, exposures to light can be adjusted either during the shooting or afterwards, and then re-combined. The capability of doing this during the shooting is an important aspect of the personal imaging invention, because it allows the artist to capture additional exposures if necessary, and thus to remain at the site until a final picture is produced. The final picture as well as the underlying dataset of separately adjustable exposures is typically sent wirelessly to other sites so that others (e.g. art directors or other collaborators) can manipulate the various exposures and combine them in different ways, and send comments back to the artist by email, as well as by overlaying graphics onto the artist's head mounted display which then becomes a collaborative space. In very recent embodiments (1990s) this has been facilitated through the World Wide Web. This additional communication facilitates the collection of additional exposures if it turns out that certain areas of the scene or object could be better served if they were more accurately described in the dataset.

Lightstrokes and Lightvectors

Each of a collection of differently illuminated exposures of the same scene or object is called a lightstroke. In the context of Personal Imaging, a lightstroke is analogous to an artist's brushstroke, and it is the plurality of lightstrokes that are combined together that give the invention described here it's unique ability to capture the way that a scene or object responds to various forms of light. From each exposure, an estimate can be made of the actual quantity of light falling on the image sensor, by applying the inverse transfer function of the camera. Such an estimate is called a lightvector.

Furthermore, a particular lightstroke may be repeated (e.g. the same exposure may be repeated in almost exactly the same way, holding the light in the same position, each time a new lightstroke is acquired). These seemingly identical lightstrokes may collectively be used to obtain a better estimate of a lightvector, by averaging each of the lightvectors together to obtain a single lightvector of improved signal to noise ratio. This signal averaging technique may also be generalized to the extent that the lamp may be activated at various strengths, but otherwise held in the same position and pointed in the same direction at the scene. The result is to produce a lightvector that captures a broad dynamic range by using separate images that differ only in exposure level (cite 

@article{comparam,
  author    = "S. Mann",
  title     = "Comparametric Equations",
  journal   = "{IEEE} Trans. Image Proc.",
  year      = 2000,
  volume    = 9,
  number    = 8,
  note      = "ISSN 1057-7149",
  month     = "August",
  pages     = "1389-1406"}
where the mathematical theory behind this art is described in detail)

Other practical issues of painting with lightvectors

Another aspect of the invention is that the photographer need not work in total darkness as is typically the case with ordinary lightpainting. With a typical electronic flash, and even with a mechanical shutter (as is used with photographic film) the shutter is open for only 1/500 sec or so for each ``lightstoke''. Thus the lightpainting can be done under normal lighting conditions (e.g. the room lights may often be left on). This aspect of the invention pertains to both traditional lightpainting (where the invention allows multiple flash-synched exposures to be made on the same piece of film, as well as to the use of separate recording media (e.g. separate film frames or electronic image captures) for each lightstroke. The invention makes use of innovative communications protocols and a user-interface that maintain the illusion that the system is immune to ambient light, while requiring no new skills beyond that of traditional lightpainting. The communications protocols typically include a full-duplex radio communications link so that a button on the flash sends a signal to the camera to make the shutter open, and at the same time, a radio wired to the flash sync contacts of the camera is already ``listening'' for when the shutter opens. The fact that the button is right on the flash gives the user the illusion that he or she is just pushing the lamp test button of a flash as in normal lightpainting, and the fact that there is really any communications link at all is hidden by this ergonomic user interface.

The invention also includes a variety of options for making the lightpainting task easier and more controlled. These include such innovations as a means for the photographer to determine if he or she can be ``seen'' by the camera (e.g. indicates extent of camera's coverage), various compositional aids, means of providing workspace-illumination that has no effect on the picture, and some innovative light sources.

Computer Supported Collaborative Art

Finally, it may, at times, be desirable to have a real or virtual assistant at the camera, to direct or advise the artist. In this case, the artist's viewfinder which presents an image from the perspective of the fixed camera also affords the artist with a view of what the assistant sees. Similarly, it is advantageous at times that the assistant have a view from the perspective of the artist. To accomplish this, the artist has a second camera of a wearable form. Through this second camera, the artist allows the assistant to observe the scene from the artist's perspective. Thus the artist and assistant can collaborate by exchange of viewpoints, as if each had the eyes of the other. (Such a form of collaboration based on exchanged viewpoint is called {seeing eye to eye}\cite{mann260}.) Moreover through the use of shared cursors overlaid on this exchanged viewpoint, a useful form of computer supported collaborative photography has resulted.

The artist's camera is sometimes alternatively attached to and integrated with the light source (e.g. flash), in such a way that it provides a preview of the coverage of the flash. Thus when this camera output is sent to the artist's own wearable computer screen, a flash viewfinder results. The flash viewfinder allows the artist to aim the flash, and allows the artist to see what is included within the cone of light that the flash will produce. Furthermore, when viewpoints are exchanged, the assistant at the main camera can see what the flash is pointed at prior to activation of the flash.

Typically there is a command that may be entered to switch between local mode (where the artist sees the flash viewfinder) and exchanged mode (where the artist sees out through the main camera and the assistant at the main camera sees out through the artist's eyes/flash viewfinder).

More recent productions

Wearing a graphics enabled computer system and power source, and walking through the city streets after dark, illuminating various buildings with powerful yet portable hand--held arrays of flashlamps, the author continues to produce new images. These systems, with the appropriate signal processing, allow entire cities to be illuminated, or other large scale subject matter such as the
Brooklyn Bridge, to be captured in a new and expressive light. The wearable photographic system turns the body into a measurement apparatus to collect data about how subject matter responds to light. This process of creating visual art is often a collaborative process involving a fixed base station operated by one person and at least one other person (usually the author) walking around and illuminating the subject matter of a picture in various ways. The base station is connected to the internet, and also serves as a gateway for the wearable portion to obtain network access, through a temporarily rigged antenna stand.
CAPTION: Setting up the base station in Times Square} (a) A portable graphics computer and image capture camera are set up on a tripod, with antennas for wireless communication to the wearable graphics processor and portable rig. Various other apparatus is visible on the table to the right of and behind the imager tripod. (b) One of the items on the table is an additional antenna stand. To the left of the antenna stand (on the easel) is a television which displays what I call the ``lightvector painting'' as it is being generated. This television allows the person operating the base station to see the output of my wearable graphics processor as, for example, I walk through Times Square illuminating various skyscrapers with the hand held flash array shown in the next figure.

Each time the author moves to a new location and takes aim to light up subject matter, a new ``light vector'' is generated using the dataset from that particular lamp location and lamp orientation. Each such lightvector provides a description of how the subject matter responds to light from that particular location. Each lightvector is analogous to a brushtroke in a painting, and it is the collection of a large number of such lightvectors that make up an entire lightvector painting.

Using multiple lamps, as an art form, allows greater control over the system. Six lamps are connected to the wearable computer to provide a powerful yet hand--held and wearable photographic lighting studio system with wearable multimedia computer used to illuminate various subject matter from a variety of different locations in space. The number six was chosen for dense hexagonal packing, but having the seventh lamp (in the center) removable and removed for a space in which the author could look out to view the subject matter being illuminated. The portable nature of the apparatus allowed it to be carried around in a collaborative computer--mediated space.

The backpack based rig used to light up various skyscrapers in Times Square produces approximately 12 kJ of light into six separate lamp housings, providing better energy localization on a per--lamp basis, and giving a 100 ISO Guide Number of approx. 2000. The two antennae on the headgear wirelessly link the computer eyeglasses to the base station shown in the previous figure.

Within the computer mediated world, it was thus possible to make the lights of times square weak by comparison to the small battery powered handheld rig.

Other examples of lightvector paintings can be found in:

Collaborative projects

More recently, the author has taught his new art form to some students at the University of Toronto, and taught these students the art of building similar rigs.

Thus twenty years later, the art has emerged from a cumbersome solo effort, to a highly mobile team of visual artists, scientists, and engineers, able to quickly collect and process lightvector spaces. Gone are the days of the heavy cumbersome gear.