Understanding what a camera measures

Abstract

A great deal of mathematical processing of pictures has been developed, to create the very active field of image processing, but with little or no regard to what the numbers coming out of a camera actually mean. The purpose of this paper is to ask, and attempt to answer, the fundamental question “what does a camera measure”. Once we understand what a camera measures, we can introduce a new kind of image processing that works in lightspace rather than in image space.

1 What does a camera measure

Cameras are often present in computer vision systems that are used to measure various quantities [?], yet an often overlooked question is “what does a camera itself measure”?

A camera actually acts as an array of sensors that each measure light. In an electronic camera these sensors are typically the outputs from the pixels.

Let us construct (either as a thought experiment, or as a do experiment that can actually be carried out) a simple one pixel camera. Once we understand what one pixel measures, we can then better understand what can be measured from many pixels working together.

Perhaps the simplest way to construct our one pixel camera is to use a light sensor. The cheapest and most common kinds of light sensors are usually photoresistors, such as the cadmium sulphide photocells found in the small controllers that automatically turn streetlights on after dark.

Cameras measure electromagnetic energy, but unlike an ideal radio receiving antenna that one might connect to a spectrum analyzer, a camera measures electromagnetic radiation only within a certain region of the electromagnetic spectrum. Typically cameras are sensitive to electromagnetic energy falling within or close to the visible, portion of the spectrum and a camera’s sensitivity to light is not very flat across the spectrum over which it is sensitive. Thus a camera is certainly not a radiometer, and the measurements that it makes are certainly not radiometric. Cameras have spectral sensitivities that are different from that of the human eye, so cameras are not photometers either. But the output from a camera can still be quantified irrespective of its unique spectral response function, so cameras are referred to as quantimetric devices [?].

Although cameras do not output a linear measure of light, with the right computations, we can derive such a linear measurement, so that the camera itself can be used as a measurement instrument. The quantimetric unit, typically denoted by the letter q, is usually made relative to some reference value, q_{0}, so that it can be expressed as a ratio, or as a logarithmic ratio (often in decibels).

In this simple experiment to create and understand a one-pixel camera we will consider quantimetric analysis of the Radio Shack Cadmium Sulphide photocell because it is readily available as a test sample (Radio Shack part number 276-1657). However any surplus photocell, or even a real camera, may be used.

1.0.1 Construction of a one pixel camera

Photoresistors are devices in which resistance is a function of incident light. Typically an increase in light results in a decrease in resistance. Most are known to obey an empirical law:

R = R_{0}q^{−γ},  (1)

where γ is usually a positive constant less than one. The fact that γ is less than one indicates that the photocell tends to compress dynamic range. Cameras also compress dynamic range in a similar way. The Radio Shack 276-1657 CDS cell has γ = 0.75, allowing us to calculate the ratio of resistance change for a change in the quantity of light. (See Chapter 3 of the University of Toronto Physical Sciences PSCB01H3S course text, http://www.utsc.utoronto.ca/ quick/PSCB01H3S/Manual/).

Let us assume that a camera is an instrument for converting light into numbers. We will thus construct a camera from the photocell by simply connecting it to an ohm meter. A satisfactory ohm meter might be the MICRONTA 22-201U which has a moving coil meter movement, as shown in Fig. ??.

The needle swings to the right when the resistance is less. Like most similar ohm meters, ∞ is at the far

Figure 1: Typical resistance meter used in photocell experiment.

left and 0 is at the far right, with intermediate values in between.

It is more convenient to consider conductance, which is the reciprocal of resistance, and to define the camera’s response function, f, as conductance of the photocell. Equation (??) now becomes:

f(q) = βq^{γ},  (2)

where β = 1/R_{0}. The quantimetric function, f, increases with increasing quantity of light, q. Thus the needle on the meter will be at the far left when f = 0 = q, and will swing more to the right as f and q increase.

We wish to determine f as a function of q (e.g. suppose that we did not already know the relationship between f and q). Suppose that we have just a lamp and the camera (photocell plus ohm meter), together with a piece of black cardboard we can use to cover half of the lamp, but that we have no other measurement instruments. We assume that the lamp is round or has some other shape that makes it easy to cover up exactly half of it.

We are therefore able to cover half of the lamp, and then point the lamp at the camera (sensor), and take a reading f(q_{0}), (See Fig ??(a)), and then uncover the lamp while leaving it exactly in the same place. Uncovering the lamp exactly doubles the quantity of light received by the camera (sensor), so that we then know what f(2q_{0}) is. (See Fig ??(b).) Although we do not know the absolute quantity q_{0}, we do know the relative quantity 2q_{0}/q_{0} = 2, e.g. we do know the fact that we have exactly doubled the quantity of light, and we can record a set of ordered pairs (f(q_{0}), f(2q_{0})). We can put the lamp at various different distances from the

Figure 2: Photocell experiment: for each of various distances between the lamp and the camera, a reading is taken with the lamp exactly half covered, versus with the lamp not covered. Other than the camera, lamp, and black card (BARRIER), no other devices (such as rulers or other measurement instruments) are needed in order to characterize the response of the photocell to light.

camera (sensor) and for each such lamp position, we can generate an ordered pair (see Fig ??(c) and (d)). Let us suppose that we take ten such ordered pairs of measurements, e.g. continuing with (f(q_{1}), f(2q_{1})) and so on, all the way up to (f(q_{9}), f(2q_{9})). We can plot these ordered pairs on a graph, as shown in Fig ??. This is just like an (x, y) plot, except that the axes are actually functions, f(q), and f(2q). The first axis is f so it is convenient to use the next letter of the alphabet, g, after f in order to denote the other axis. Thus we have an (f, g) plot — a plot of a function against a plot of a function of a function, where g = f(2q).

We can fit a curve through the points. The solid line shows one possible curve, namely a straight line of slope approximately 1.68. The dotted line shows another possible curve. Repeated measurements, however, lead us to believe that the relationship is simply g = 1.68f, as shown by the solid line.

This is simply a plot of the photocell’s response function f(q) against a contracted (squashed in) version of the same function f(2q). Such a plot is called a comparametric plot.

The notion of a parametric plot is familiar to almost everyone. A parametric plot of a circle, for example, is simply a set of ordered pairs (rcos(θ), rsin(θ)) where the parameter θ takes us counterclockwise around the circle. A comparametric plot is just a special kind of parametric plot, where both axes pertain to the same function but at different rates.

Figure 3: Datapoints from photocell experiment: ordered pairs of points taken by reading the camera with the lamp uncovered and exactly half covered.

1.0.2 Solving comparametric equations

From the comparametric plot shown in Fig ?? we have determined that g = f(2q) = 1.68f(q). This equation, g = 1.68f is called a comparametric equation. In general, solving a comparametric equation g(f(q)) = f(kq), for some comparametric ratio k (in this case k = 2) means determining a family of possible functions f(q) that satisfy this equation.

In our specific case, therefore, we wish to know what possible functions f(q) give a straight line (of slope 1.68) when plotted against themselves contracted (by a factor of 2).

Fig ?? depicts two plots, one being a smooth function f(q) who’s comparametric plot is a straight line of slope 1.68, and the other being a quasi-periodic function who’s comparametric plot is also a straight line of slope 1.68. Both of these functions have the same comparametric plot. Both are solutions to the comparametric equation g = 1.68f.

The quasi-periodic function illustrates that any function can be specified on, for example, the interval from q to 2q, and then merely replicated into the interval to the right scaling by (2∗, g∘), e.g. by stretching to twice the width and composing to g() of the height, in this case, merely multiplying by 1.68 times the height, since in this specific case, g is linear. This recipe can be applied recursively in both the left and right directions. Therefore, in general, solutions to comparametric equations are not unique. However, we might choose a function that is semimonotonic, with semimonotonic slope, semimonotonic curvature, (and so-on, including possibly further derivatives), of which f(q) = βq^{γ} is

Figure 4: Two examples of possible solutions to the comparametric function g = 1.68f. One solution, plotted as the thick line, is given by f(q) = q^{0.75}. Another solution plotted as the thin line, consists of a triangular wave, square-shaped wave, and round shaped wave between q and 2q so that we can see how the waveform has comparametric periodicity. In each “period”, the function is stretched by a factor of two as we go to the right (or contracted by a factor of two as we go to the left), and composed by a factor of g() as we go to the right, or g^{−1}() as we go to the left. This phenomenon is called comparametric periodicity (“comperiodicity”). Thus there is a comparametric uncertainty in the solution to a comparametric equation. In this simple example, the comperiodization is a simple multiplication by 1.68 for each comperiod, as denoted by the triangular amplifier symbol, that maps, for example, point P_{n} into point P_{n+1}.

the preferred general solution to g = 2^{γ}f.

1.1 Directly solving a comparametric equation by unrolling while collecting the data

We started by knowing the response function of the photocell, e.g. knowing the solution of the comparametric equation, and then confirming that this known function was in fact a solution.

Now suppose we did not know the solution, and did not know how to solve the comparametric equation, in general.

In this case, we can construct a simple way of simultaneously collecting comparametric data, and arriving at a numerical solution to the comparametric equation, namely, to obtain samples of the function f(q).

Refer back to Fig ??(a) where half the lamp is blocked with the black cardboard, to obtain f(q_{0}). Now suppose we unblock the lamp, as shown in Fig ??(b) to obtain f(2_{q_{0}}). The important next step is to now cover exactly half the lamp again and then bf move the lamp toward the photocell until the observed meter reading is exactly the same as it was when the map was not covered. Thus the situation as shown in Fig ??(c) will be that q_{1} = 2q_{0}.

The procedure is repeated. Uncover the lamp to obtain f(2q_{1}) = f(4q_{0}). Then cover it again, and move