Improving Our Depth Perception in Augmented Reality

Accurate depth perception is critical to future military, medical applications of AR

Sports fans have come to expect some of the extras they see on their TV screen, such as the yellow lines that appear on a football field highlighting where the ball needs to go for a first down. Similarly, NASCAR fans can find their favorite driver in the pack because of those superimposed car numbers on the screen.

"Anyone who's seen a football game has seen the virtual first-down line and it looks like it's really something painted on the field. It's a very compelling graphic. It really looks like it's there," says computer scientist J. Edward Swan, II.

But most people are probably not familiar with the technology that makes this all happen.

Augmented reality, or "AR," is used to superimpose computer-generated virtual objects on our view of the real world.

With support from the National Science Foundation (NSF), Swan and his team at Mississippi State University (MSU) are working to improve depth perception in the augmented reality environment.

 "Ed Swan's work on perceptual issues for AR goes back more than a decade and includes co-authoring a seminal paper in which computer graphics attributes were shown to be effective in providing a sense of the depth of occluded objects for mobile AR using a heads-up display," explains Lawrence Rosenblum, program officer for graphics and visualization within the NSF Directorate for Computer and Information Science and Engineering. "As AR continues to develop, with many of the ideas for mobile AR now being ported to PDAs and improved hardware for medical and other relatively stationary applications, his ongoing AR perception research is paving the pathway for working AR systems."

One challenge is aligning an object in the real word so that it precisely lines up with virtual objects. While exactness in millimeters may not be all that critical for a TV sporting event, think about a soldier relying on this tool during battle or a doctor using this technology in an operating room. The virtual graphics might be located inside the patient's body.

"Imagine an application where you have a surgeon trying to align a scalpel in relation to virtual graphics that are also in view," says Swan.

"The surgeon would look at the patient and be able to see into their skin. So, instead of looking to one side and seeing the scanned information on a monitor, they would look at the patient, but it's as if they're looking into the patient, as if the patient has become somewhat transparent at that spot," continues Swan.

Think about it as a sort of X-ray vision!

The most advanced lab that is actually applying augmented reality to medical applications is located at the Technical University of Munich, which Swan visited in 2011. There are no scalpels or heart monitors in Swan's lab at Mississippi State, but there is an augmented reality haploscope.

"A haploscope is a precision device used by vision scientists to present a carefully controlled image to each eye, so it's a stereo image into the right and left eye," he says.

In Swan's lab, volunteers use the haploscope to position virtual objects and real ones. Graduate student Gurjot Singh explains to the volunteers that they will see an object that looks exactly like a physical one, but it will be computer generated. Their goal is to align those objects.

"The purpose of the head-mounted display device is to display an image in stereo. I have two transparent screens in front of my eyes, which display a stereo screen. This thing on my head is a tracking device, which tracks my head when it moves, and the device sends information to a computer through this wire. And when the computer receives this information, it draws the scene on this screen based on this information, and my location in space. That's how, when I move my head, you can see the scene is updated on the screen," explains Singh.

He says the equipment has improved dramatically in just the past few years.

"The last device used to be so heavy. The optics were made of glass. These are made of plastic, and are much brighter than our last device. It gives a more compelling immersive experience than our last one," says Singh.

Accurate depth perception is vital to military applications of augmented reality. Swan previously worked at the Naval Research Lab, on a project to develop a mobile augmented reality system.

"The idea is that soldiers would walk around with an augmented reality display on their head. They would be able to see the real world with their actual vision and we would be able to show them graphics that, if we did it right, would look just like they were also real objects in the world," says Swan.

On a battlefield, soldiers must quickly assess what's happening on the ground. As AR improves, they will increasingly need to combine that situational awareness with commands or warnings sent through their goggles or helmet.

"You don't want to fill someone's field of view with graphics that are going to block too much of their view of the real world," notes Swan.

Ryan Ismert is general manager for augmented reality at Sportvision, the company that's developed this technology for major league sports broadcasters. And for TV viewers, just like for a soldier or a surgeon, simplicity is the key.

"It is presented as a natural part of the game, explaining something that may otherwise be hard to see, like the first down marker or the path someone took for a reception route," says Ismert.

Augmented reality is also the tool used to paint those country flags on the bottom of swimming pools to distinguish swimmers during the Summer Olympics.

"Visual reasoning adds an extra layer of understanding and engagement to sports, but industry may also benefit from augmented reality tools in everything from aircraft maintenance to capping an underwater oil well," says Ismert.

Swan agrees that industrial applications of AR could be widespread.

"Boeing was the first company to try using augmented reality as part of their manufacturing process, in the early 1990's. One of Boeing's chief scientists recently said that augmented reality was going to be increasingly used at Boeing and at other high-tech manufacturing companies in the coming decades," says Swan.

Miles O'Brien, Science Nation Correspondent

Marsha Walton, Science Nation Producer

Seeing Beyond the Visual Cortex

Research could lead to new rehabilitative therapies when visual cortex is damaged

It's a chilling thought--losing the sense of sight because of severe injury or damage to the brain's visual cortex. But, is it possible to train a damaged or injured brain to "see" again after such a catastrophic injury? Yes, according to Tony Ro, a neuroscientist at the City College of New York, who is artificially recreating a condition called blindsight in his lab.

"Blindsight is a condition that some patients experience after having damage to the primary visual cortex in the back of their brains. What happens in these patients is they go cortically blind, yet they can still discriminate visual information, albeit without any awareness." explains Ro.

While no one is ever going to say blindsight is 20/20, Ro says it holds tantalizing clues to the architecture of the brain. "There are a lot of areas in the brain that are involved with processing visual information, but without any visual awareness." he points out. "These other parts of the brain receive input from the eyes, but they're not allowing us to access it consciously."

With support from the National Science Foundation's (NSF) Directorate for Social, Behavioral and Economic Sciences, Ro is developing a clearer picture of how other parts of the brain, besides the visual cortex, respond to visual stimuli.

In order to recreate blindsight, Ro must find a volunteer who is willing to temporarily be blinded by having a powerful magnetic pulse shot right into their visual cortex. The magnetic blast disables the visual cortex and blinds the person for a split second. "That blindness occurs very shortly and very rapidly--on the order of one twentieth of a second or so," says Ro.

On the day of Science Nation's visit to Ro's lab in the Hamilton Heights section of Manhattan, volunteer Lei Ai is seated in a small booth in front of a computer with instructions to keep his eyes on the screen. A round device is placed on the back of Ai's head. Then, the booth is filled with the sound of consistent clicks, about two seconds apart. Each click is a magnetic pulse disrupting the activity in his visual cortex, blinding him. Just as the pulse blinds him, a shape, such as a diamond or a square, flashes onto a computer screen in front of him.

Ro says that 60 to nearly 100 percent of the time, test subjects report back the shape correctly. "They'll be significantly above chance levels at discriminating those shapes, even though they're unaware of them. Sometimes they're nearly perfect at it," he adds.

Ro observes what happens to other areas of Ai's brain during the instant he is blinded and a shape is flashed on the screen. While the blindness wears off immediately with no lasting effects, according to Ro, the findings are telling. "There are likely to be a lot of alternative visual pathways that go into the brain from our eyes that process information at unconscious levels," he says.

Ro believes understanding and mapping those alternative pathways might be the key to new rehabilitative therapies. "We have a lot of soldiers returning home who have a lot of brain damage to visual areas of the brain. We might be able to rehabilitate these patients," he says. And that's something worth looking into.

Miles O'Brien, Science Nation Correspondent

Jon Baime, Science Nation Producer

Navy Medicine Announces New Eyeglass Frame

From Navy Bureau of Medicine and Surgery Public Affairs

WASHINGTON (NNS) -- Navy Medical Logistics Command (NMLC) announced Jan. 19 that all active duty and Reserve personnel, including recruits will soon have a new standard issue eyeglass frame available.

Since 1990, military personnel and recruits have received standard issue S9 eyeglass frames, often jokingly referred to as "birth control glasses" or simply "BCGs." Not any longer.

"We are happy to announce that the New Year brings with it a new frame option for all personnel serving on active duty and in the Reserves," said Capt. Matt Newton, commanding officer of Naval Ophthalmic Support and Training Activity (NOSTRA) in Yorktown, Va. "Service members have told us that they like the appearance of the new frame. We are confident this frame will increase the likelihood that military personnel will continue to utilize their eyeglasses beyond boot camp."

Effective Jan. 1, the current cellulose acetate spectacle frame provided at all Armed Forces initial entry training sites began the transition from male and female, brown "S9" spectacles to a new, unisex, black "5A" frame.

The change stems from a study which was directed by the Military Health System's Optical Fabrication Enterprise (OFE) and coordinated by NOSTRA in order to find a suitable frame to add to the standard issue inventory. Selected samples were submitted to U.S. Army Public Health Command for review, and three frames were identified for user tests. Tests were conducted at Recruit Training Center Great Lakes, Ill.; Advanced Infantry Training, Camp Geiger, N.C.; Fort Sam Houston, Texas; Fort Knox, Ky. and U.S. Coast Guard Recruit Training Center, Cape May, N.J.

Surveys assessed functionality, durability and cosmetic appearance and the 5A frame was selected as the best option.

Initial deployment of the 5A frame will occur at all Armed Forces initial entry training sites. Within six months, the 5A frame will be made available to all active duty and Reserve service members with full implementation expected to be completed over a two-year period.

Retirees are currently eligible to receive standard issue S9, S91A and Half-Eye frames, and there will be no change to this authorization. However, over the next two years, the OFE will study the feasibility of providing 5A frames to retirees.

The OFE was established by Congressional mandate in 1996, with the U.S. Navy Surgeon General charged with managing the program. Upon the closure of the Army Optical Fabrication Laboratory at Fitzsimmons Army Medical Center, NOSTRA became a joint production lab with Army opticians augmenting Navy and civilian production staff. The OFE is guided by the Optical Fabrication Advisory Board which represents the Surgeons General of the Army, Navy and Air Force. A sampling of OFE initiatives includes managing the military Frames of Choice program, standardization of military combat eye protection inserts, introduction of a new submariner frame, and operational support with the deployment of the new M50 gas mask insert. For the last several years, the OFE has produced approximately 1.5 million pairs of spectacles and optical inserts annually for authorized military personnel.

NMLC is responsible for designing, executing and administering individualized state-of-the-art solutions to meet customer medical materiel and health care requirements. NMLC supports the U.S. Navy with acquisition and logistics systems training, health care services strategies, operational forces support, medical equipment and logistics solutions, acquisition management, deployable platforms and eyewear fabrication.

Navy Medicine is a global health care network of 63,000 Navy medical personnel around the world who provide high quality health care to more than 1 million eligible beneficiaries. Navy Medicine personnel deploy with Sailors and Marines worldwide, providing critical mission support aboard ship, in the air, under the sea and on the battlefield.

Vision Center of Excellence Promotes Eye-injury Research, Care

By Donna Miles
American Forces Press Service

WASHINGTON, Jan. 9, 2012 – Next month will mark a major milestone in advancing care for wounded warriors suffering debilitating eye injuries with a ribbon-cutting at the Walter Reed National Military Medical Center in Bethesda, Md.

The Department of Defense/Department of Veterans Affairs Vision Center of Excellence will officially open its new headquarters at the Walter Reed facility, providing an expanded physical presence for a growing collaboration between the two agencies.

The goal, explained Army Dr. (Col.) Donald Gagliano, its executive director, is to promote research and initiatives to prevent eye injuries and better diagnose and treat those suffering from them.

Although often overlooked, eye injuries are one of the signature wounds of the wars over the last decade, Gagliano noted.

Just how prevalent these injuries are isn’t clear, he said, although officials estimate that 13 to 22 percent of all casualties between 2002 and 2010 have suffered eye injuries or trauma.

These injuries often go underreported on the battlefield, particularly when caused by explosions that inflict other, highly visible and frequently life-threatening wounds. Gagliano estimated that eye-injury rates soared as high as 29 percent among casualties before the military began mandating the use of ballistic glasses for deployed troops.

“The reality is it is very difficult for us to know exactly what the prevalence of eye injury is, because it is often intertwined with other polytrauma,” Gagliano said.

What is known is that the same flying fragments and high-energy waves that tear into body tissue and inflict traumatic brain injuries also take a severe toll on the eyes. Service members are suffering eye injuries unlike those in civilian trauma cases, and more severe than those from past conflicts, Gagliano said.

So one of the first goals of the Vision Center of Excellence, stood up under the 2008 National Defense Authorization Act, is to establish a registry to determine the prevalence of eye injuries and track wounded warriors’ care and rehabilitation through both the DOD and VA systems.

This, Gagliano said, will provide an important starting point for advancing eye protection and care throughout the force.

The law that established the center formalized a partnership already being forged between DOD and VA to provide better care for wounded warriors. Even its makeup -- Gagliano and half of the staff are from DOD, and his deputy, Dr. Mary Lawrence, and the other half of the staff from VA -- promotes collaboration as they pool expertise and resources, he said.

“It is very unique in structure, and that is what allows us to function across both systems as effectively as we have,” Gagliano said. “It helps us bring together the entire vision-care team of both the Defense Department and the VA to function effectively as an integrated team.”

In standing up the center, its founders opted to maximize rather than replace existing resources. “Early on, we elected to be a center that would work with and through the existing system rather than being a single place,” Gagliano explained.

As a result, the center has operated with facilities and office space in Washington as well as at Madigan Army Medical Center near Seattle.

The opening of the new headquarters next month will provide an official home to the center, but Gagliano said it will continue to draw on existing capabilities throughout DOD and VA, including VA’s network of 13 blind rehabilitation centers.

To ensure these entities operate as effectively as possible, eye surgeons and eye-care providers from both agencies come together each month for a worldwide ocular trauma videoconference, Lawrence said. Participants -- at forward operating hospitals in Afghanistan, at Landstuhl Regional Medical Center in Germany, at military treatment facilities stateside and at VA polytrauma centers -- come together to share experience and explore ways to improve the care they provide, she explained.

“This is an amazing worldwide group of providers,” she said. Based on their inputs, she added, “a lot of interesting process improvements ideas have surfaced that can be put into good use immediately.”

Another priority for the Vision Center of Excellence is expanding the research base about eye trauma, Gagliano said. That’s critical to improving care, he explained, because civilian institutions, including the National Eye Institute, have limited research about the types of eye injuries being seen in the combat theater.

Gagliano expressed hope that the center’s collaboration with colleges, universities and research bodies worldwide ultimately will improve the research, prevention, diagnosis, treatment and rehabilitation of military eye injuries.

At the Vision Center of Excellence, “we are leading the nation in trying in trying to determine the best ways to address these issues,” he said.