Light Field Photography

The American company Lytro makes cameras based on light field photography. The first version, a tiny box, made headlines but didn’t have much impact. Now Lytro tries again with the Lytro Illum, an improved model about the size and shape of a typical “prosumer” camera. One defining feature of light field photography is the ability to refocus an existing picture, and you can try that for yourself on Lytro’s Living Pictures page: click a picture, then anywhere within the picture to bring that part into focus.

While the Illum announcement was once again widely publicized, the articles I’ve seen were rather vague on the underlying technology, as is the (rather messy) Lytro website itself. The best way to learn about the technical and scientific details is Lytro founder Ren Ng’s 2006 dissertation, Digital Light Field Photography (PDF hosted by Lytro). Thankfully, Ng offers easy-to-understand and generously illustrated overview sections along with the obligatory mathematical formulae. So if you have a scientific (and photographical) bend of mind you’ll want to read the dissertation yourself. The technology is both fascinating and splendidly explained by Ng, so it’s well worth your time.

In the rest of this post, I’ll try to highlight the most important aspects of the dissertation from a practical viewpoint. Note that I’m neither a mathematician nor an expert in optics, so I’m depending on Ng’s simplified explanations myself!

Recording Light Fields

I grabbed two of Ng’s plentiful illustrations to briefly present the basics of light field photography (LFP). The first image (Figure 2.3 in the dissertation) shows a cross-section of a conventional camera. Imagine this slice of camera as recording a one-dimensional image, with x as the axis of pixel coordinates. Since every pixel gathers all rays projected from the same point of origin in the focal plane, assuming the lens is precisely focused, we can introduce a second axis u that records the point at which each ray for a given pixel x intersected the lens.

Normal Camera Recording

For a normal camera this second dimension is meaningless, as each output pixel simply sums the light from all input rays. However, a “plenoptic” camera that is capable of recording a light field divides its sensor into an array of microlenses, each gathering all light coming from the main lens and projecting it onto a small group of actual sensor pixels. The second picture (Figure 3.1 in the dissertation) visualizes the effect for the cross-section of a plenoptic camera, again producing a one-dimensional image with x as the axis of output pixels. Now the u axis is itself divided into sections, each corresponding to a microlens. That is, for each output pixel x the plenoptic camera distinguishes various contributing input rays along u – directional information that is lost in a normal camera.

Plenoptic Camera Recording

To obtain a conventional photography from a plenoptic camera, we can simply sum all vertical input rays u for each pixel x, which is exactly what a normal camera does implicitly. But the additional directional information enables us to perform different sums. As it turns out (please read the dissertation for the gory mathematical details), tilting the vertical “column” that defines an output pixel corresponds to changing its focus, and more complex deformations correspond to lens aberrations. Summing the microlens “cell” data from different adjacent columns, with some appropriate weighting, therefore enables algorithmic focus shifting and correction of lens aberrations.

Photographic Benefits

So now we have a light field recording. What are the benefits compared to a conventional picture?

Focus Correction — This was the original motivation for Ng’s research. Unfortunately, it turned out to be the least important use of LFP by the time Lytro actually commenced production. Today, the amateur photographers most likely to struggle with focus problems simply use smartphone cameras with effectively infinite depth of field, as noted below.

Interactive Focus Shifting — Lytro’s sample images nicely demonstrate this effect. The big disadvantage is the requirement for an appropriate interactive display. Moreover, artistic photographers will want to permanently fix the focal plane at their discretion. Amateurs once again won’t care and will just use their infinitely focused smartphone pictures.

Aberration Correction — The aberrations of current lenses are already handled by software (in or out of camera), but LFP’s great flexibility could enable simpler and cheaper lens architectures. Moreover, it can correct for aberrations not induced by lenses, e.g. atmospheric turbulence in telescopy.

Greater Aperture — The Lytro Illum uses a remarkable f/2 aperture throughout its entire 9.5–77.8 mm (30–250 mm equivalent) zoom range. That’s possible because LFP compensates for the shallower depth of field and increased aberrations normally incuded by greater apertures. Consequently, cheaper lenses can be used with less light, longer zoom, and/or faster shutter times.

Holographic Recording — Since LFP knows each pixel’s focal plane, it can derive its spatial distance from the lens and so construct a holographic relief. This neat feature drives industrial and scientific LFP use, but remains irrelevant to consumers until we have convenient holographic displays to match.

Macro Photography — The magnification required for macro photography immensely compresses the depth of field. Obtaining a correct focus is tricky, and capturing the entire object may require digitally combining multiple exposures with different focal points. LFP could at least greatly simplify macro photography, if not entirely obsolete the laborious multi-exposure procedure.

Inherent Limitations

When I first heard about LFP I was quite enthusiastic and thought it might obsolete conventional lenses entirely. Sadly, this turned out not to be the case. LFP has two important limitations that qualify the benefits listed above. These are discussed in chapter 3 of the dissertation.

Lower Resolution — To obtain directional information, multiple pixel elements on the image sensor must be grouped under one microlens, effectively forming a single output pixel. Consequently, the sensor’s effective output resolution shrinks with the size of each microlens pixel group. Increasing the latter improves refocusing power but reduces maximum image quality. Eventually the image is “focused” everywhere thanks to high directional resolution (many pixels per microlens), but that “focus” is actually rather blurry due to low spatial resolution (few microlenses).

Ng notes that image sensors can theoretically achieve a higher resolution than anyone knows what to do with, so this tradeoff shouldn’t matter in the long run. However, smaller sensor elements reduce light gathering power and increase errors, up to a limit imposed by diffraction. The Lytro Illum offers only 5 MP effective spatial resolution which is slightly laughable for a big $1,500 camera (my Sony NEX-7 has 24 MP). Unfortunately, an LFP sensor that reaches good resolution along both spatial and directional axes is bound to be rather gigantic.

Conventional Casing — Ng cites previous work that dashed my hopes for doing away with conventional lenses and camera casings. Arrays of microlenses without a main lens have already been constructed by various research groups, but cannot match the image quality of similar arrays mounted behind a conventional main lens. The reason is once again diffraction, and thus a fundamental physical limitation. It seems existing smartphone cameras already offer the best feasible compromise of size and image quality. LFP is not an alternative for minimizing size and weight.

(Lack of) Market Success

Why hasn’t Lytro’s first model caught on? The obvious answer is that it was overtaken and obsoleted by smartphone cameras. These have virtually unlimited depth of field thanks to their tiny focal length (explanation), meaning they never have trouble focusing on anything. Since modern smartphone cameras also offer good resolution and even somewhat decent low-light performance (aided by post-processing), and since casual photographers don’t normally care about aesthetic bokeh, a consumer-grade light field camera was irrelevant right out of the gate.

So will the new Lytro Illum fare better? Repositioning for the (semi-) professional market was the only option available to Lytro, but unfortunately this market emphasizes maximum resolution and sharpness where the light field technology is forced to compromise. On the other hand, finding the perfect lens for the perfect aperture and depth of field is part of the art and sport of photography, so to speak, and it’s not clear whether the audience for a $1,500 camera would even accept aid in this regard.

Unless light field sensors can match the perceptible quality of conventional sensors so closely that big camera makers are willing to make the switch, I don’t think the technology as it stands has much of a future outside of industrial applications. However, there should be a niche for macro photography which is LFP’s one stand-out ability that’s both highly useful and hard to replicate conventionally. We might see a few good specialized cameras for that purpose.

2016-04-06: As I expected, Lytro failed to gain any foothold in the consumer photography market and now has exited the field altogether (DP Review summary, Imaging Resource commentary). The company will focus on industrial equipment for virtual reality recording, which is really a much better match for light field technology.