Chapter 1

Chapter 1: MannGlass ("GlassEyes"), SpeedGlass, "GOOGlass", and "The Veillance Contract"

By Steve Mann, 2013 September 26

Meet to discuss, Mondays 12h10 - 14h00 in WB219 (information office = GB149; my office = SF2001)

1.0 Introduction

Ubiquitous mobile/portable imaging and computing is now a reality, and wearable computing (e.g. Digital Eye Glass) is just becoming commonplace. We are thus entering an era in which our everyday lives are being or will soon be mediated by vision-based computers.

Many of us already carry a camera and computing device (smartphone or cameraphone) most of the time. These devices will soon take the form of Digital Eye Glass to help many of us see better, and improve the quality of our lives. This technology will be especially welcome to the growing population of older individuals with failing eyesight, or to those who have trouble remembering things.

As technologies become more personal, the Digital Eye Glass will help us remember names and faces (e.g. the Wearable Face Recognizer [Mann 1996]), find our way (digital maps overlayed with Augmediated Reality), and provide us with improved safety (e.g. personal security), health (sensing, etc.) and well-being.

From my long-standing personal experience (more than 35 years of inventing, designing, building, and experimenting with wearable computing and Digital Eye Glass since my childhood -- first as an amateur scientist in my youth, and later in my professional life), I have made some interesting observations. As General Chair of the IEEE ISTAS13, I was able to bring together leading thinkers and doers from around the world, to create and define the new field of research: Veillance.

1.1 WearCam: The Wearable Camera

My grandfather taught me to weld when I was 4 years old. It was an experience that was at once both exhilaratingly-fun and also frightening. Frightening because, in order to protect one's eyes, it is necessary to wear a full helmet or put one's head into a hood fitted with a piece of very dark glass.

Now we see through a glass, darkly

Both eyes see through this single piece of glass. Through this darkglass everything except one tiny little speck of incredibly bright light (the electric arc) appears almost completely pitch black.

Throughout my childhood, in the 1960s and 1970s, I experimented with television -- a then new technology -- as well as electric circuits, computation, and the like, all as a way of seeing better -- something I felt was a basic human need.

This led me to the invention of HDR (High Dynamic Range) imaging: now we can see, through the Digital Eye Glass, a dynamic range of more than 100 million to one, from the blinding bright tip of the welding electrode to the deepest darkest shadows.

Here is what I see through my Digital Eye Glass while welding (screen grab from realtime live video):

Perhaps this was the first time in human history that a person was able to see the entire welding process, from the bright electric arc all the way into the deepest, darkest shadows, processed in real-time together with realtime AR (Augmediated Reality) computer graphics overlays.

A brief history of digital cameras and HDR (High Dynamic Range) Imaging

The resolution of cameras has increased from thousands of pixels to megapixels, but image quality (e.g. dynamic range) at each pixel has not kept pace.

The first digital camera was invented (by Steven Sasson of Kodak) in 1975, and had 10,000 pixels (based on a Fairchild 100*100 pixel sensor), and 4 bits-per pixel per channel (1 channel, i.e. greyscale).

Today, mobile and portable camera phones have resolutions up to 41 Megapixels (e.g. Nokia PureView 808), and some cameras are in the gigapixel range, i.e. spatial resolutions that are more than 100,000 times what they were in 1975. Thus many cameras can "see" the detail (e.g. "read" small print on a newspaper) thousands of times better now than then.

But they still could not (until the invention of HDR) match the human eye for dynamic range (i.e. being able to simultaneously sense in low-light and really bright light).

Remarkably, the bit depth of modern cameras is typically only twice what the world's first digital camera was 38 years ago (8 bits compared to 4 bits).

Recently I have addressed this problem by inventing and steadily improving something called HDR (High Dynamic Range) Imaging.

Robertson et al. recently wrote:

"The first report of digitally combining multiple pictures of the same scene to improve dynamic range appears to be Mann." [Mann 1993]
-- [Estimation-theoretic approach to dynamic range enhancement using multiple exposures, JEI 12(2)], p220, right column, line 26.
HDR is now used by many commercially manufactured cameras, including Apple's iPhone (which implements HDR internally), as well as many surveillance cameras and other specialized imaging devices.

1.2 WearComp: The Wearable Computer

Back in the 1970s our family was one of the first few in our city to have a computer -- back then home computing was all but unheard of. At the time, many computers were still programmed using keypunch machines and punched paper cards. I envisioned the computer as part of a wearable photographic imaging system.

Back in the 1970s my childhood experiments in computing involved wearing the computer together with a helmet-mounted television system/station to see the world photographically, through a viewing glass on my helmet:

I even put "rabbit ears" (antennas) on my wearable computer so that it could also function as a wearable television station with wireless communications capability. I could stream live video so others could look through my eyes and provide remote expert advice or guidance in, or simply to share photographic experiences. I called this "Seeing Eye-to-Eye".

MannGlass and SpeedGlass (3M)

This was around the same time that the SpeedGlas, the first electronic welding helmet, appeared in 1981.

Speedglass darkens uniformly everywhere and thus doesn't help the wearer see better than traditional fixed-shade darkglass during the actual welding.

MannGlass adapts the image, improving visibility during actual welding. These explorations in Digital Eye Glass later led to my invention [Mann 1993] and patenting (U.S. Patent 5,828,793) HDR (High Dynamic Range) imaging a technology now used in or with most digital cameras.

(The "MannGlass" Digital Welding Glass replaces the darkglass with a computational glass.)

I spent much of my childhood as an inventor and amateur scientist, tinkering, creating, and exploring.

Now beyond my simple need to see the welding arc, I began to look at my everyday life through this photographic and computational glass, using a variety of "wearable computing" contraptions I invented, designed, built, and wore.

By the early 1980s I had miniaturized my Digital Eye Glass from the 1970s helmets into an eyeglass device suitable for creating an "Augmediated Reality" in my everyday life.

The MIT Wearable Computing Project

A decade later, in 1991, I was accepted to MIT (Massachusetts Institute of Technology) and I brought my invention to the MIT Media Lab, and ended up founding the MIT Wearable Computing Project. See to hear the historical account in the words of Nicholas Negroponte (Director of the MIT Media Laboratory) himself:
"Steve Mann was... building wearable computers in high school, and I think its [a] perfectly good example... here's a young man that brought with him an idea... And when he arrived here alot of people sort of said wow this is very interesting... I think its probably one of the best examples we have of where somebody brought with them an extraordinarily interesting seed, and then it sort of... grew, and there are many people now, so called cyborgs in the Media Lab and people working on wearable computers all over the place."

Upon arriving at MIT, I helped others there build wearable computers, and by 1997 I had published an article in IEEE Computer defining Wearable Computing as a discipline [Mann 1997 (PDF)(HTML)].

Many people from around the world took an interest in this work, including Mark Spitzer and a handful of others who built upon these ideas and created a commercial embodiment of the Digital Eye Glass (but without the camera). I showed them how they could build a camera into their MyVu product. It wasn't long after that, that MyVu was acquired by Google.

After finishing at MIT in 1997, I took a faculty position at University of Toronto. A number of my students from Toronto ended up working at Google, and a product manufactured by Google now somewhat resembles my wearable computer:

(MannGlass 1978-1980; Google Glass)

(MannGlass 1998, sleek-and-slender wire-frame design, published 2002 [Mann 2002])

Bearable Computing

This field of research could be broadened from "wearable computing" to "bearable computing" or "body-borne computing" -- the broader term "bearable" suggesting the technology can be borne by the body which may mean (1) carried (e.g. by hand), (2) worn, or (3) or implanted in the body.

As an example, consider the eyeborg camera implant, Canadian Patent 2313693, listed by Time Magazine among The 50 Best Inventions of 2009:

Wristworn Computing

Another example is the wristwatch computer, Canadian Patent 2275784, written 1999, describing the wristwatch computer I invented in 1998, and presented to the world at IEEE ISSCC 2000:

ISSCC: 'Dick Tracy' watch watchers disagree

By Peter Clarke
EE Times
(02/08/00, 9:12 p.m. EST)

SAN FRANCISCO -- Panelists at a Monday evening (Feb. 7) panel session at the International Solid State Circuits Conference (ISSCC) here failed to agree on when the public will be able to buy a "Dick Tracy" style watch for Christmas, with estimates ranging from almost immediately to not within the next decade.

Steve Mann, a professor at the University of Toronto, was hailed as the father of the wearable computer and the ISSCC's first virtual panelist, by moderator Woodward Yang of Harvard University (Cambridge Mass.).

Not surprisingly, Mann was generally upbeat at least about the technical possibilities of distributed body- worn computing, showing that he had already developed a combination wristwatch and imaging device that can send and receive video over short distances.

The work on the wristwatch computer was also published as the feature article in (and appeared on the cover of) Linux Journal, July 2000:

Sixth Sense

While still at MIT, I experimented with wearable gestural interfaces to mediate and augment physical reality using computational information and natural hand gestures, such as pointing [Mann 1997 (PDF)(HTML)]. The early prototypes required colored tape on my fingers, to make it easier for the wearable camera to recognize the gestures, but by 1998, I'd developed a neckworn augmented reality pendant that no longer required finger-tape.

This device contained both a data projector and camera, and could be used for gesture-sensing self-interaction [Mann 2001], or for collaborating with others [Mann 2001b].

I envisioned digital information superimposed on the real world as a kind of "sixth sense", rendered visible by synthetic synesthesia, so I named this invention "Synthetic Synesthesia of the Sixth Sense" [Mann 2001] [Gearly 2002].

11 years later, another student and his professor presented this "SixthSense" work at TED, and it was among the top 20 most viewed TED talks of all time:

See also and

1.3 Spaceglass: Extramissive Digital Eye Glass

Having created four generations of Digital Eye Glass [Mann 2012], we have now developed a more advanced form of truly 3D gesture-based interaction:

[Mann 2013]

Unlike the typical GUI (graphical user interface) found in desktop computing, or mobile devices, our system requires only simple gestures, such as the "Outstretched-Hands" gesture in front of any subject matter, as shown above. This enables seamless interaction between the physical world and Digital Eye Glass. For example, reaching out to a tree, the tree "answers back" with information about itself. Although the Landscaping Department has a database of all trees planted, and thus the knowledge could be obtained "top-down" (i.e. from the authorities), our application allows individuals to acquire, disseminate, and share knowledge in a "bottom-up" way, i.e. through realtime online social networking. The 3D camera in the eyeglass collects information about the environment, enabling individuals to be producers of -- and not merely consumers of -- information. Thus the proposed Digital Eye Glass answers the call of "haccessibility", i.e. the DIY (Do-It-Yourself) spirit of community-based knowledge acquisition during and about everyday life activities, like enjoying nature.

Presently, as Chief Scientist of Meta (makers of Spaceglasses), I'm working with my PhD student, Raymond Lo, who is now Meta's CTO (Chief Technology Officer).

Spaceglasses embody, for example, the Wearable Face Recognizer [Mann 1996], and many other features of WearCam and WearComp, but with true 3D accuracy.

1.4 Technology for Humanity

Many inventions come from a research lab, but wearable computing, Digital Eye Glass, and HDR are inventions that came from my everyday life.
In this respect, these technologies are very much "in and of the real world".

I was very much inspired by the tagline of the world's largest technical society, the IEEE. Their tagline is "Advancing Technology for Humanity":

I've always felt this importance of "Technology in SERVICE of HUMANITY" and many of my inventions and ideas are driven by this ethos -- not just ideas, but ideals!

So, back in the 1970s, when I saw people bent up like pretzels, hunched over their keypunch machines or computers, I thought the computer should be the thing that's "bent up" ---- the computer should be twisted and wrapped around the body and head ---- i.e. wearable computing, not the other way around.


And when I saw the GUI and "desktop metaphor" I came up with something else, for which I coined the term "Natural User Interface" back in 2001 [Mann 2001]. Digital Glass gives us a natural interaction with the real world rather than being hunched over a desktop metaphor!

1.5 Surveillance is the Veillance of Hypocrisy

For more than 35 years I've been inventing, designing, building, and wearing computers in my day-to-day life, and this has uncovered some strange paradoxes about society in general.

In the 1970s people would often cross the street to avoid me, but by the 1980s I'd finally developed a form of the technology that was much less offputting, and suitable for use in everyday life. However, despite the everyday acceptance, there were still specific members of society that had problems with it.

In 1985, 28 years ago, I was stopped by a security guard while getting onto the subway. I began to try to understand why security guards, in particular, were among those who were so strongly against Digital Glass.

It was mostly shopkeepers, security guards, and other authority figures who objected to my computerized seeing glass.

Ironically, it was those members of society who install and use surveillance cameras -- those were the people who seemed to object to the Glass --- the very same people who were building a world of watching -- were afraid of being watched!

At the time, my computer system didn't have the memory or storage capacity to capture and record much, if anything, of what I saw.

But nevertheless, having been physically assaulted by security guards afraid that the Eye Glass might be recording them, I began to try and understand why ordinary people accepted it more readily than those in positions of authority who had such an adverse reaction to it.

Back in the 1970s and 1980s I was probably the only person walking around wearing a computer with wireless communications, but these days many people are walking around with a small computer, such as a smartphone or other device that includes a camera, computer, and wireless data link.

And I'm no longer the only one facing these issues. Last year when I was physically assaulted by McDonalds staff, in Paris, France, I did a little research and found that I wasn't the first. A travel agent had previously been assaulted by a McDonalds staff for photographing the menu in this Paris location.

Privacy rights of signage

Many people today also use smartphone applications that translate signs from a foreign language, or use their phones to read QR codes.

There is a certain irony in the recent proliferation of "no photography", "No cameras" and "No Cellphone in store, please" signs along with QR codes that give pre-purchase product information.

The QR code is like a sign that says "Cameras required", often seen together with signs that say "cameras forbidden".

Existential Contraband

Most of the places that forbid cameras are quite full of cameras. For example, at the US Consulate, when picking up my daughter's passport, they had a rule that no electronic devices were allowed in their building. But (as a "cyborg") I "am" an electronic device. So I was both required to enter to pick up my daughter's passport, and also forbidden from entering because I myself, am an electronic device. By their rules, I was existential contraband == contraband by my mere existence.

I've sometimes pointed out this irony to the shopkeepers and establishment owners: "Cameras are forbidden but you have quite a few of them here", to which they reply, "but those are SURVEILLANCE cameras".

As if to somehow suggest that surveillance cameras are not cameras:


So I asked:

If I streamed my Eye Glass video to the police, it would continue to function as a seeing aid, but now the police could also look trough my Glass and catch suspects. This would make my seeing aid into a "surveillance camera". Since "surveillance cameras are not cameras", streaming the video to the police would make my seeing aid into something that is no longer a camera. Since it would then no longer be a camera, would I be allowed to wear it so long as I continued to stream the video to the police?

What is sur-veillance?

"Surveillance" is a French word which means "watching from above" or "watching over". "Veillance" means "watching" == from the French verb "veiller" which means "to watch". "Sur" means "over" or "from above", as in "sur tax" or "sur charge".

Random House Dictionary defines "surveillance" as:

"a watch kept over a person, group, etc., especially over a suspect, prisoner, or the like: The suspects were under police surveillance."

Under assault by these security guards I felt like the suspect or prisoner in this definition. Whatever I had invented was the opposite of surveillance, so eleven years ago I formulated a new philosophical theory for a reciprocal to surveillance, which I named "sousveillance", from the French prefix "sous" which means "under", "below", or "from below", as in "sous-chef".[Mann 2002][Mann 2003].

Now there are hundreds of books and papers on sousveillance, e.g.:

This work is informing privacy policy worldwide[Freshwater, Fisher, and Walsh 2013][Ganascia 2010][Bakir 2010][Weston and Jacques 2009], and adding new directions to previously surveillance-centered policies.

So now there are two veillances: surveillance and sousveillance. My six-year old daughter made a couple of nice drawings that illustrate these two veillances:


Why do buildings and cars always have the right to wear cameras?

Children see things in an amazingly clear way. For example, my daughter also asked:
"Why do buildings and cars have more rights than people?" (e.g. why are buildings and cars allowed to wear cameras when people are not?)
"why is merchandise more valuable than people"? (e.g. why is merchandise allowed to protect itself with cameras when people are not? We must protect the welfare and well-being of the candy on the shelf!)

Obviously its really the shopkeepers and building owners and motorists that have these rights, but the question -- simply posed with a child's clarity, as above -- makes for interesting thought and discussion.

Remarkably, for example, cars are never forbidden from "wearing" a dashboard camera even in a drive-in movie theatre!
I challenge the reader to come up with even one single example of a situation in which a car has ever been forbidden from "wearing" a camera!

When those under surveillance also have cameras:

Less than half a year ago, a New York police informant was caught planting crack cocaine, and then photographing the "evidence", to falsely incriminate a shopkeeper, in a shop when the police informant thought no one was looking. But the shop's surveillance cameras caught the police agent planting the evidence, from two different camera angles.

When the shopkeeper, Donald Andrews, was arrested for posession of crack cocaine, his shop's surveillance video was presented and he was then released.

In this case, his surveillance cameras were doing some sousveillance.

Imagine what might have happened if the only recording of this incident was one made by the police cameras used to photograph the "evidence". Andrews would have been sent to prison for 25 years!

When only one party (e.g. the police and their agents) keeps a recording of an event, there is an inherent conflict-of-interest. There is a possibility that the one party (e.g. the police) might alter, or simply omit or delete some of, the recordings, in order to increase their numbers of arrests and convictions.

What if the drugs were planted in a backpack?

What if the police (or their agents) had planted the drugs in his backpack rather than in his shop?

The backpack would become a "crackpack". And when this sort of thing could happen in a country like Singapore, where drug posession is a hanging offence, it could be quite easy for police (or their informants) to "crackpack" someone they don't like, and then send them to the gallows when they "discover" the drugs.

People like Donald Andrews are able to defend themselves from false accusations because they have their own recording of what happens to them while they are inside their own buildings or shops. But when Donald Andrews or anyone else is outside his shop, should he still not have the same right to capture his own memories for the purpose of proving his own innocence?

Architecture of One

We can think of clothing as a building built for a single occupant. This "Architecture of One" [Mann 2001] thinking might help us understand the "reasonableness" of protecting that building with a security camera, not unlike the security camera found in an automobile or multi-person building.

To deny people the right to protect themselves in this way is to create a one-sided veillance that invites fraud, falsification, and omission by the one party that is otherwise the sole keeper of history.

This one-sided nature of surveillance came through when I was physically assaulted by McDonalds staff.

I asked McDonalds to investigate their assault against me, and they refused to consider their surveillance videos. They only wanted to consider their employee testimony. In this sense, the one-sided nature of their surveillance concentrates evidence into the hands of one entity, and therefore could create a possibility of fraud through this inherent conflict-of-interest.

Therein lies the hypocrisy of surveillance. Surveillance is the veillance (watching) that says "we're going to watch you but you're not allowed to watch us".

Thus surveillance is the veillance of hypocrisy.

Sousveillance -- the new word that I made up -- is the opposite of surveillance.

Surveillance is to Sousveillance as Hypocrisy is to what?

The opposite of hypocrisy is integrity.

Sousveillance, therefore, might be regarded as the veillance of integrity.

1.6 The Veillance Contract

While using my Glass to magnify some small print on food labels at T&T Supermarket in Toronto, I was told that such activity was forbidden. I asked to speak to the manager who said such camera-based magnifying devices were forbidden.

Business establishments like T&T Supermarket are actually in violation of the law because they forbid the use of a seeing aid in their establishment.

I wasn't recording anything until I was told such activity was forbidden. Once told to stop recording (which I wasn't doing), I started recording, so that I would have evidence of their illegal activities.

I've looked at human rights approaches to veillance, e.g. having governments step in and enforce existing laws against forbidding the use of a seeing aid, or visual memory aid, on the basis of Human Rights Law that allows accessibility to people with all levels of ability. Indeed, it is easy enough to wear a hidden camera. It is simply the part that helps people see better (the display) that creates the objection. Even with a large neckworn camera like Sixth Sense, I've seldom had problems. In this day-and-age of hidden spy cameras, it is all but impossible to stop people from wearing cameras -- their effect is to stop people from wearing Digital Eye Glass specifically.

I've also considered liabalization: An organization forbidding Digital Eye Glass is mandating a change in the way we see (or don't see) the world, and therefore could be liable for injury if a person cannot see where he or she is going (or suffers a sudden and unprepared-for change in the way the world is presented visually). For example, I've got a form for people to sign, accepting liability if I should slip and fall because of a sudden change in prescription (from digital eye glass to none) mandated by their organization.

However, these approaches rely on government intervention ---- something some people are in favour of, but others (especially libertarians) are against. Some believe a business owner should be able to say "this is my store and cameras are not allowed here for any reason".

So, I have a different answer == one that would even appeal to nearly everyone, even the libertarians.


An integrity protocol for the cyborg age

Let me now present the "Veillance Contract" by way of a simple analogy or comparison to a more traditional contract:
Let us consider an example in which you, the reader, are negotiating a business contract with a businessman named Alphonse ("Al"), who happens to be the President and CEO of Acme Corporation.

Now suppose Acme Corporation has a "no photography" and "no recording" policy.

You and Al agree to form a business partnership and you both draft up a 100 page contract together in his office. The last page is the signature page, that you and Al both sign.

Suppose you are forbidden from having a copy of the contract. Al says that you can take as long as you want to read and understand the contract, but you're not allowed to make any recordings or copies of it. He even allows you to bring in your legal counsel so long as you all agree to Acme's "no recording" policy.

Now let's suppose there's something you don't like about what's on page 5 of the contract. You and Al retype some of the wording on page 5, until you're both happy with it, and then print out a new page 5. Maybe your legal counsel has further problems and Al agrees to a couple of other minor changes.

Then something on page 10 needs revision to make your legal counsel happy.

Finally, Al inserts a couple of replacement pages, and you and Al both sign the singature page (page 100).

Al leaves the stack of papers on his desk and you leave without a copy of the contract. Let's suppose, perhaps a week later, Al staggers into his office with a gallon jug of whiskey, half drunk, and knocks the desk over, scattering the papers about his office. After sobering up, he sorts through the papers and finds 3 copies of page 5, and two copies of page 10. Not remembring which is which, he picks one of each and puts the contract pages into his safe. A couple of years later, Al dies of liver disease, and his successor comes across the contract you signed with Acme Corporation, and sues you for contract infringement. Eventually you get a copy of the contract through discovery (the right to compel the other party to disclose documents that establish a Cause of Action in Civil Procedure). By this time, the contract no longer looks like what you remember signing.

The lesson in this story is that if you're forbidden from receiving a copy of the contract, there's an increased incentive for fraud, or error, e.g. the one and only party that has custody of the records, could make some changes, or maybe simply some careless errors or omissions.

That is why a contract should not be valid unless both parties are allowed to receive a copy of it.

Contracts are binding whether written or oral, but until recently we've not had an easy way to document oral contracts.

Now the veillance contract is a contract for the digital and new media age. There is a certain "social contract" or "ad idem" mutual understanding about how we should and do conduct ourselves in the world.

So my proposal is very simple:
The Veillance Contract
If party "A" wants to record the activities of party "B" and also wants to forbid party "B" from having or making any recording of the activity, then any recording that "A" makes should be inadmissible as evidence in a court of law.

I call this principle the "Veillance Contract". So the shopkeeper or business owner who asks the government to stop meddling in his or her affairs, gets exactly what they asked for ---- they shouldn't come crying to that same government when they want to use that government's courtrooms and prisons to prosecute shoplifting crimes, for example, where they've forbidden customers from recording their own evidence for use in their own defense.

For example, T&T Supermarket should not be able to use their video surveillance recordings as evidence in court, because, (by prohibiting customers from making their own recordings), they've, in effect, destroyed (or prevented the collection of) a person's own evidence the person might use to exonerate themselves.

1.7 Absolute Surveillance Corrupts Absolutely

Surveillance is the centralized veillance of a single authority, whereas sousveillance is the distributed veillance of the many.

Traditionally surveillance cameras looked like this:

Zabou, a street artist, gives us a glimpse of what sousveillance might have looked like then too:

Surveillance is the veillance of secrecy:

But now surveillance cameras are often concealed in mysterious domes of wine-dark opacity, so people can't see which way the cameras are "looking", or how many cameras are present in the dome:

Surveillance, whether created by government or by private industry, tends to be secretive. If one asks about details of the security camera systems in a government building or business estalishment, the answer is usually evasive.

What if we followed the example set by our goverments and industry leaders?

So imagine if we followed the example of our governments and industry leaders, and their ideals of secrecy, i.e. if we were to each wear a spherical dome around our head so that nobody could see which way you were looking:

But sousveillance draws its strength not from secrecy, but from the opposite of secrecy: openness.

Sousveillance is the veillance of openness

Sousveillance, by its nature, tends to be open and distributed, where videos that document corruption and wrongdoing are widely distributed on social networking sites.

Data integrity is often protected by distributed storage, e.g. instead of recording data in only one location -- where it can be lost accidentally or deliberately -- we spread the data around and make multiple backups in different locations, distributed among multiple individuals. That way if one person, whether through malice (e.g. sabotage), or simply carelessness, happens to cause data loss, others can restore it.

The opposite of data integrity is data corruption.

Thus surveillance is also the veillance of corruption:


The word "corruption" here has a double entendre: it refers to computer/data corruption as well as political/police/human corruption. Thus its opposite is either integrity (i.e. data integrity) or honesty (e.g. an honest politician or an honest police officer).

Sousveillance counterbalances surveillance, by completing the Veillance Contract to facilitate an increased possibility of integrity, openness, and honesty.

1.8 Completing the Veillance Contract

Let us end with two examples: one of data corruption due to centralized storage; and another of data integrity due to distributed storage and dissemination.

For our first example, consider the Brazilian electrician, Jean Charles de Menezes, who was shot dead (8 times) by police on the London subway system. It was a case of mistaken identity (the police thought he was a terrorist but it turned out they had the wrong person). What's most salient is that there were four CCTV surveillance system recordings seized by police who claimed all 4 recordings were blank.

But subway workers had already looked at the recordings before the police had time to seize them, and the subway workers had seen that they were not blank.

This raised quite a bit of controversy in the press:

[Row over 'blank' CCTV tapes at station, by Rosie Cowan and David Hencke, The Guardian, Tuesday 23 August 2005 00.08 BST, and many other news publications]

The second example comes much closer to home. I live with my family on Dundas Street West, where we all ride the "505" streetcar to school each day.

Less than 2 months ago, a tragic incident happened on that same streetcar, with a teenager named Sammy Yatim.

Police arrived to find the boy alone with a knife on the streetcar. The driver had aleady safely departed from the streetcar, leaving the boy alone. Police surrounded the streetcar, and from a safe distance, with guns drawn, asked him to put the knife down.

The boy froze, perhaps in fear, and did not move or drop the knife.

A policeman standing outside the streetcar (from a safe distance) shot Sammy 3 times. Then after the boy fell to the ground, the policeman shot him another 6 times, even though the police were at a safe distance and vastly outnumbered the boy.

Sammy was also tasered later, after he was already dead.

Like most of our streetcars, this streetcar had four surveillance cameras in it. Here's a picture of one of the surveillance cameras on the 505 streetcar, which I took only 26 days before the shooting:

Jean and Sammy were both shot 8 times, under the surveillance of four cameras. In Jean's case the only recordings were surveillance recordings. In Sammy's case, there was both surveillance and sousveillance: bystanders on the street had cellphone cameras that captured the entire shooting incident.

It was only after that video was brought to the attention of the public, that there was a public outcry, that there was a full investigation.

It has often been said that the police should have been wearing surveillance cameras. However, as evidenced here, the police themselves were not investigated until sousveillance took hold.

Sousveillance is Crowdsourced Veillance

It was the presence of sousveillance -- and its open nature (in contrast to the secrecy of surveillance), through crowdsourcing and social networking, which brought the situation to the pubic eye. In Jean's case, no charges were brought to the officers. In Sammy's case, the officer was charged with murder.

If you see something, say something, with pictures and video

While opposed by some security guards, police, and other members of the establishment, sousveillance may be the answer to a balanced veillance, completing the other side of the Veillance Contract. So if you see something, like the fire exits illegally chained shut at the Finch subway stop in Toronto, say something, with pictures and video:

1.9 Implementing the Veillance Contract

How might we implement the Veillance Contract?

One approach would be realtime subject access to surveillance video. We're already tracked by surveillance cameras, smartphones, and the like. Suppose any one of us could submit a "subject access" request from our smartphone, for any surveillance video taken of us? Denial of such a request could automatically invalidate the surveillance video (e.g. enter the establishment into non-compliance status so that their surveillance videos would later be excluded as evidence in any court of law). This would create an incentive for large entities to obey subject access laws.

Such a system would create a Veillance Contract rendered valid if both parties were allowed to keep a copy of the "Contract" (the video surveillance recordings), or rendered invalid if the owner of the surveillance video refused to immediately stream a copy of the surveillance video to the requestor.

In areas where older technologies (slower wireless networks) are still in place, the requestor might simply request a realtime MD5 checksum of the surveillance video be broadcast to him or her, to validate it. Failure of a business owner or other surveillance entity to provide the checksum would break the Veillance Contract, rendering the surveillance data illegal and inadmissible in a court of law.

Social Contract or Social Conscription?

The "social contract" arises when individuals choose to give up some of their individual freedoms to submit to an authority in return for the protection of their remaining individual rights [Jean-Jacques Rousseau 1762].

In olden times people might choose to live in a walled city, or the like, and submit to the authority of a king or other ruler, in return for protection. But in today's society, the "social contract" is more of a "social conscription". And likewise, in the matter of veillance, let us choose the Veillance Contract as an alternative to the one-sided "surveillance conscription".

The integrity pendant

Another way to implement the Veillance Contract is through the wearing of a sousveillance device. One such device, called "ALIBEye" [Mann, Ali, Lo, and Wu, 2013], is a pendant that provides a lifeglogging experience that can be used as an alibi, in a court-of-law, for example, to prove one's time and place.

One way of mitigating the one-sided nature of surveillance is to hold those doing the surveillance accountable. This can be done by sousveillance (keeping one's own recording of what happens). In this case, it is useful to timestamp one's own sousveillance recordings so that they can be used as an alibi in any accusation of wrongdoing. Another way of mitigating the one-sided nature of surveillance is to timestamp the surveillance data. This can be done without the consent of -- and even against the will of -- the authorities conducting the surveillance. An example of this is the display of content into the surveillance data. For example, if you simply walk around with a copy of the front page of a newspaper, and display it to every surveillance camera you see, you've greatly reduced the ability of the authorities to backdate the surveillance video.

Wearing a small 2D bar-code (e.g. neckworn display or wristwatch computer) that updates with a computational hash function of the day's news headlines, for example, can serve a similar effect:

Ideally a combination of these two methods (recording and displaying) can be used, in which one records one's own sousveillance video, while also displaying to one's self AND to other cameras (other sousveillers and the surveiller), a hash of today's latest news, and then timestamping that sousveillance recording. The use of a visual recording to provide an alibi requires that the time of recording is bounded both before AND after the recording takes place. Standard cryptographic timestamping of a hash of a digital recording provides evidence that the recording took place before the timestamp; i.e. the signed hash depends on the recording. We now describe approaches to bound the time of recording in the other direction, providing evidence that the recording took place after a certain time, preventing replay attacks, and preventing temporal falsification in BOTH temporal directions.

Alibi veillance can be used for both surveillance and sousveillance and in various combinations of these veillances.

Prior work: Timestamps are all one-sided

A digital timestamp can prove to the world that a certain set of bits existed before a certain time, namely when the timestamp was created; examples are formalized in IETF's RFC 3161, RFC 3339, and ANSI's X9.95.

Alibi Veillance: Another side to timestamping

To create an alibi veillance system, three elements are required: (A) an a priori unknowable fact that is widely observed, ideally in a precise digital form; and (B) a way to embed A into a real-world scene, and record the embedding, in a way that is very difficult to falsify; and (C) a verifiable timestamp service, to which a hash of the recording can be immediately submitted to.

Steps A and B bound the time in one direction, and step C bounds it in the other temporal direction, thus resulting in a bi-directional bounding of the alibi veillance.

Alibi displays

A simple embodiment of this idea is to display a hash of current events, such as a QR code. This can be done, for example, on a GNU Linux wristwatch that runs a "cron job" to automatially display the material now and again, so as to be visible to the wearer of a sousveillance eyeglass, or other wearable camera that can view the wristwatch (as well as other cameras in the environment, including surveillance cameras). These cameras are able to view the wristwatch that runs a "cron job" to automatially display the material displayed on the watch now and again.

ALIBEye is an app that uses Gen-5 Glass to project a pattern (visible or infrared) onto the scene and captures recordings of that pattern, as shown below:

The barcode can also include a hash of a recording of the wearer's own eye, iris pattern, gaze, eye movements, etc., to further biometrically establish their presence at the scene. The resulting complex interaction is difficult to falsify. ALIBEye requires little to no attention from the user. It provides continuous automatic updating to the latest temporal code, and provides the safety and security benefit to others, not just the wearer.

Sousveillance is not anti-surveillance, and, in fact, ALIBEye can even facilitate a cooperation between sousveillance and surveillance so that neither party can falsify the veracity or time-stamp of their veillance recordings.