It sounds like an invitation to enter The Twilight Zone, but real life, of course, is already lived in three dimensions. Traditional photography and other imaging systems, however, can only show us two-dimensional pictures—pictures that can fool even trained eyes. For instance, experts in wartime deception have known for nearly a century that flat or inflatable decoys look from afar much like real tanks or bunkers in optical (photographic) or radar images. Differentiating between real tanks and painted decoys can mean the difference between victory and defeat on the battlefield.

Finding better ways of bringing 3D realism to military surveillance and reconnaissance underpins a MITRE research project called "Three-Dimensional (3D) Sensor Exploitation." MITRE has a long history of supporting improvements in radar technology. For instance, as chief engineer on the Air Force's airborne Joint STARS (Surveillance Target Attack Radar System) program, we've done much of the systems engineering work on the platform's SAR technology. This is a continuation of that history.

Not Like the Movies

In the movies, radar (or sonar) seems so simple: an operator sends out a signal, gets a blip on a screen, and learns exactly where the object he's seeking is located. But it's considerably more complex than that, explains Walt Kuklinski, principal software systems engineer and head of the project.

"A radar at a single point returns information about where objects are in only one dimension, range, or distance from the radar," he says. "Standard radar measures the time of flight of the pulse transmitted from the platform [such as Joint STARS] to the target, which creates a reflection that returns to the radar. With a single pulse in a single location, you have no idea what or where the object is, except that it's on the spherical shell of a given radius.

"Synthesizing an image with SAR involves taking thousands of one-dimensional range profiles as the radar platform moves," he says. "We then apply the same type of mathematics used to produce 3D medical images like MRIs—the physics is similar. We can resolve the uncertainty in one of the two remaining dimensions and produce images to get a more complete picture of the scene. Now, if we modify the flight path of the radar and operate it so that we get views of the scene from all different angles, we can begin to produce 3D images."

Covering All the Angles

The first part of the research aims at improving the detection process—to find out where the objects really are. This involves pulling together images from both SAR and hyperspectral sensors and fusing the data. (Hyperspectral sensors take measurements at hundreds of distinct wavelengths, many beyond the visible light spectrum—thus helping us distinguish, say, a mylar balloon masquerading as a tank from an actual metal tank.) Correlating all these sensor readings to locate objects on the ground isn't easy. Finding a computational formula for the final outcome, called "data registration," has frustrated many researchers over the years. The MITRE team, however, has made significant strides toward solving the data registration problem, which in turn provides the foundation for the rest of the project.

Kuklinski describes the work as "very state of the art." Since the MITRE team began its research, in fact, the Defense Advanced Research Projects Agency (DARPA) and the Air Force have proposed a full 3D SAR imaging work program and have distributed test-data sets of SAR images to the wider research community (such as military research labs, universities, and other federally funded research and development centers) for their use.

"What we'd done here is very similar to what DARPA is proposing," he says. "You take the fundamental physics of the problem and simulate what a real radar sensor would see in a particular scene. The data set we've been using to develop our 3D images is exactly the same type that DARPA has distributed. We're at the leading edge of what's happening—though our study was on a more limited scale. We conducted full simulation studies by taking targets, simulating what radars would see over a large number of viewing angles, then taking the data and synthesizing SAR images. This is where the field is advancing."

There are similarities between the SAR work and medical imaging, but Kuklinski notes that medical imaging offers two big advantages over military radar. First, the medical "target" (the patient) sits still; second, he or she is rarely camouflaged. Military targets have an unfortunate tendency to both move and be hidden or disguised in some way.

"Ideally, to form a complete 3D image, you need the sensor to take measurements at every possible viewing angle," he says. "In military situations, it's very hard to get a complete set of views from all possible angles, so our challenge is to develop the math that can make up for our inability to get sensory data from an entire target in a given situation. But we've made progress." Enough progress that Kuklinski and his colleague Andrea Kraay were able to report their latest findings at the prestigious International Society for Optical Engineering 2004 Defense & Security Symposium.

A Tank Is Still a Tank

Once researchers have found computationally efficient ways to build three-dimensional images, there are a number of uses for the improved pictures. Kuklinski's team is also focusing on the area of automatic target recognition (ATR).

"Let's say you're trying to make a decision in a battle during a rainstorm," he says. "Is a vehicle leaving the location a school bus or a flatbed truck filled with soldiers? Because of the clouds, you can't get an optical image, so you have to use radar. To improve your ability to say with certainty what you're seeing, having a 3D image would be a big help. But even with 3D, you would still need a skilled image analyst on hand to decide what you're seeing. That's where an automatic tool could take some of the pressure off a pilot or other decision maker."

Team member Mike Jeffris, a lead signal processing engineer, is developing algorithms that use 3D data from multiple sensors to improve ATR capability. This is the "exploitation" stage of the project, when the 3D image formation capability is put to use.

The main problem, Jeffris says, is that you can lose a lot of information about a target when collecting a two-dimensional image. Such imagery depends on the viewing geometry, target pose, and target articulation. "For instance, the image of a tank can vary dramatically as its turret rotates and as the tank moves through the battlespace. Its basic 3D geometry remains unchanged, however, and this is at the heart of our ATR toolset."

He's focusing on developing mathematical tools for 3D imaging that are based on the target's 3D geometry. "That way," he says, "a tank with a turret pointed in any direction or assuming any pose will still be recognizable to the automatic target recognition process as a tank."

"Automatic target recognition systems already exist," adds Kuklinski. "But many of them are designed to work with pre-existing templates, which identify objects through an embedded set of patterns. Templates usually have limitations, however. Most of them rely on seeing the object at a certain orientation—from the top, from the side, and so on. The problem is that objects often come at you in random orientations. Our goal is to be able to take a template of a target at one orientation and use one relatively straightforward set of calculations to determine what the target is, rather than looking at millions of rotations of a target and comparing them to the radar data. This is where Mike's work is really different."

Benefits from Screening to Manufacturing

Kuklinski envisions non-military uses for the 3D recognition technology as well. Homeland security, in particular, could benefit from the research. For example, it's currently possible—at least theoretically—to defeat luggage screening systems in airports and other checkpoints. If someone trying to sneak weapons or contraband through a scanner knew precisely at what angle the scanner views the object, it might be possible to fool the machine.

"Current screening images are usually 2D, but without too much trouble, you could take additional measurements and get a 3D image of the contents of a package," Kuklinski says. "With MITRE's recognition algorithms, we could improve the automatic classification of objects. We could have 3D templates of things such as guns, knives, gas canisters, and scissors stored in a screening system. Then, regardless of the orientation of the object, we'd be able to automatically recognize them with higher accuracy." Other areas that might benefit from this research include inspections during manufacturing and robotic vision.

Ironically, just as our brains can do things that even the most sophisticated computer can't, much of this work seeks to replicate what comes naturally in our three-dimensional world.

"Humans perform object recognition very well, of course," says Kuklinski. "Unfortunately, it's a difficult task for machines, but we believe we're advancing the state of machine understanding through this work."

—by Alison Stern-Dunyak

Related Information

Articles and News

• Seeing More with Nonlinear Visualization
• Fooled Again? Developing Counter-deception Decision Support

Technical Papers and Presentations

• Three-Dimensional (3D) Sensor Exploitation