Improving Our Depth Perception in Augmented Reality

Accurate depth perception is critical to future military, medical applications of AR

Sports fans have come to expect some of the extras they see on their TV screen, such as the yellow lines that appear on a football field highlighting where the ball needs to go for a first down. Similarly, NASCAR fans can find their favorite driver in the pack because of those superimposed car numbers on the screen.

"Anyone who's seen a football game has seen the virtual first-down line and it looks like it's really something painted on the field. It's a very compelling graphic. It really looks like it's there," says computer scientist J. Edward Swan, II.

But most people are probably not familiar with the technology that makes this all happen.

Augmented reality, or "AR," is used to superimpose computer-generated virtual objects on our view of the real world.

With support from the National Science Foundation (NSF), Swan and his team at Mississippi State University (MSU) are working to improve depth perception in the augmented reality environment.

 "Ed Swan's work on perceptual issues for AR goes back more than a decade and includes co-authoring a seminal paper in which computer graphics attributes were shown to be effective in providing a sense of the depth of occluded objects for mobile AR using a heads-up display," explains Lawrence Rosenblum, program officer for graphics and visualization within the NSF Directorate for Computer and Information Science and Engineering. "As AR continues to develop, with many of the ideas for mobile AR now being ported to PDAs and improved hardware for medical and other relatively stationary applications, his ongoing AR perception research is paving the pathway for working AR systems."

One challenge is aligning an object in the real word so that it precisely lines up with virtual objects. While exactness in millimeters may not be all that critical for a TV sporting event, think about a soldier relying on this tool during battle or a doctor using this technology in an operating room. The virtual graphics might be located inside the patient's body.

"Imagine an application where you have a surgeon trying to align a scalpel in relation to virtual graphics that are also in view," says Swan.

"The surgeon would look at the patient and be able to see into their skin. So, instead of looking to one side and seeing the scanned information on a monitor, they would look at the patient, but it's as if they're looking into the patient, as if the patient has become somewhat transparent at that spot," continues Swan.

Think about it as a sort of X-ray vision!

The most advanced lab that is actually applying augmented reality to medical applications is located at the Technical University of Munich, which Swan visited in 2011. There are no scalpels or heart monitors in Swan's lab at Mississippi State, but there is an augmented reality haploscope.

"A haploscope is a precision device used by vision scientists to present a carefully controlled image to each eye, so it's a stereo image into the right and left eye," he says.

In Swan's lab, volunteers use the haploscope to position virtual objects and real ones. Graduate student Gurjot Singh explains to the volunteers that they will see an object that looks exactly like a physical one, but it will be computer generated. Their goal is to align those objects.

"The purpose of the head-mounted display device is to display an image in stereo. I have two transparent screens in front of my eyes, which display a stereo screen. This thing on my head is a tracking device, which tracks my head when it moves, and the device sends information to a computer through this wire. And when the computer receives this information, it draws the scene on this screen based on this information, and my location in space. That's how, when I move my head, you can see the scene is updated on the screen," explains Singh.

He says the equipment has improved dramatically in just the past few years.

"The last device used to be so heavy. The optics were made of glass. These are made of plastic, and are much brighter than our last device. It gives a more compelling immersive experience than our last one," says Singh.

Accurate depth perception is vital to military applications of augmented reality. Swan previously worked at the Naval Research Lab, on a project to develop a mobile augmented reality system.

"The idea is that soldiers would walk around with an augmented reality display on their head. They would be able to see the real world with their actual vision and we would be able to show them graphics that, if we did it right, would look just like they were also real objects in the world," says Swan.

On a battlefield, soldiers must quickly assess what's happening on the ground. As AR improves, they will increasingly need to combine that situational awareness with commands or warnings sent through their goggles or helmet.

"You don't want to fill someone's field of view with graphics that are going to block too much of their view of the real world," notes Swan.

Ryan Ismert is general manager for augmented reality at Sportvision, the company that's developed this technology for major league sports broadcasters. And for TV viewers, just like for a soldier or a surgeon, simplicity is the key.

"It is presented as a natural part of the game, explaining something that may otherwise be hard to see, like the first down marker or the path someone took for a reception route," says Ismert.

Augmented reality is also the tool used to paint those country flags on the bottom of swimming pools to distinguish swimmers during the Summer Olympics.

"Visual reasoning adds an extra layer of understanding and engagement to sports, but industry may also benefit from augmented reality tools in everything from aircraft maintenance to capping an underwater oil well," says Ismert.

Swan agrees that industrial applications of AR could be widespread.

"Boeing was the first company to try using augmented reality as part of their manufacturing process, in the early 1990's. One of Boeing's chief scientists recently said that augmented reality was going to be increasingly used at Boeing and at other high-tech manufacturing companies in the coming decades," says Swan.

Miles O'Brien, Science Nation Correspondent

Marsha Walton, Science Nation Producer

Mapping the Genomes of Crocodiles and Alligators--It's Not for the Faint of Heart!

Adventurous field work expands knowledge of evolution and could help save endangered species

David Ray never turns his back on his research, and with good reason! "If it can't bite you, it's not interesting," he jokes.

Ray and his team study alligators, crocodiles, bats and flies, among other creatures. There's no handbook for learning how to capture an alligator or a crocodile. "Oh, it's great. I mean, there's just a thrill," says Ray, an evolutionary biologist at Mississippi State University (MSU).

With support from the National Science Foundation (NSF), this multidisciplinary team from several universities is mapping crocodile and alligator genomes. Reptiles resembling these animals have existed for around 80 million years and they are among the first reptiles to have their DNA sequenced. The research could expand our knowledge well beyond crocodilians to other reptiles, birds, and even dinosaurs.

"Birds and crocodiles, though you wouldn't think it from looking at them, are each other's closest existing relative," notes Ray.

"The group currently assembled by David Ray and others includes scientists with expertise ranging from crocodilian systematics and population genetics to pure molecular biology to the fields of bioinformatics and comparative genomics," explains Lou Densmore, chair of the Biological Sciences Department at Texas Tech University. "Although just 10 years ago, the thought of such a study was beyond the wildest dreams of any of us, we are now sitting on the threshold of the most ambitious crocodilian genetics and genomics research ever attempted."

Catching a 'croc' or 'gator' is usually done at night from a boat or a canoe. These animals have a layer of tissue in their eyes called tapetum lucidum, which reflects back red. So, when a researcher's headlamp spots that red color, the team heads in that direction.

"You approach the animal as quietly as you can, and preferably from the front so that you can just basically get the breakaway snare to go over the snout," says Ray. "Of course, the animal doesn't like that, so it thrashes and then you've got potentially a 10-foot animal that wants to eat you on a rope!"

"When they've exhausted all their energy, you can handle them relatively easily. Then, we will go to a sinus on the back of the neck and draw however much blood we need, and then it's time for release. The key is to keep control of the head. That skull is like a brick and if it whips around and knocks you, it can hurt you pretty badly. Always keep a hand on it," he warns.

The Crocodilian Genomes Project has benefited from the input of a bona fide movie star. Errol, the Australian saltwater crocodile whose genome is being sequenced by the group, has been featured in a number of films--most notably the 2007 thriller Black Water. "I never thought I'd get the opportunity to work with crocodiles or celebrities," jokes project co-investigator Daniel Peterson, associate director of Genomics at MSU's Institute for Genomics, Biocomputing & Biotechnology. "Now I can say that I have had the rare privilege of working with a celebrity crocodile."

Learning more about the genetic makeup of crocodilians could help efforts to save some endangered species, such as the very odd-looking Indian gharial (Gavialis gangeticus), which is now down to just a few hundred animals. Scientists could possibly identify the most diverse animals in the gene pool and then breed them. "The more we can understand how their DNA is put together, the more likely we are to understand how to keep them from going extinct," says Ray.

That is one of the most exciting aspects of the research for Lou Densmore. "By the time the next genetic sequence analysis of this genome is complete, we will not only know exactly how the gharial fits into the evolutionary history of the Crocodylia, but we will also have the data needed to pursue a 'comparative -omics' approach that will help explain the remarkable cranial morphology that has caused such controversy in interpreting its phylogenetic placement in the order," explains Densmore.

Two other team members, biologist Fiona McCarthy, who teaches in the College of Veterinary Medicine at MSU, and Carl Schmidt, an associate professor in the College of Agriculture and Natural Resources at the University of Delaware, take the assembled sequences, identify genes, and provide standardized gene nomenclature and functional annotation.

"My main research focus is providing functional annotation so that researchers are able to more easily get from data to knowledge, and it is wonderful to work on a sequencing project where functional information is factored in from the start," says McCarthy. "Add on top of that, all the really interesting biology, such as temperature regulation of sex determination, tooth development in crocs and birds, linking reptiles and birds together in an evolutionary sense, and you get a lot of very interesting insights into fundamental biology."

"Incorporating some of these insights into my teaching ensures that I have examples that students won't soon forget," she adds.

At the University of Florida, team member and associate professor of biology Ed Braun is also a co-investigator, along with microbiology professor Eric Triplett, on a separate NSF grant to create a curriculum that is based on the research.

"Crocodilians really have the potential to capture the imagination of students since they look like living dinosaurs. Involving students in the annotation and analysis will open their eyes when they see the similarities to and differences from the real living dinosaurs--birds. Understanding crocodilians is critical for understanding birds. Despite their obvious differences, reconstructing their common ancestor will require information from both groups of organisms," says Braun.

Up to now, most of the vertebrate genomes sequenced and analyzed have been from mammals. "Thus, most of what we know about genome evolution is very mammalian-centric," notes Ed Green, assistant professor of biomolecular engineering at University of California, Santa Cruz. "We're now coming to learn that the reptilian world has evolved more slowly, from the rate of divergence at the level of chromosome rearrangements to how fast individual bases change. On the one hand, this makes things easier for genome assembly, but it also requires that we revisit a lot of assumptions and models that were made when we only had data from mammals."

When they're not fishing for 'crocs' and 'gators,' Ray's team might be tracking down bats for their research on transposable elements or so-called 'jumping genes.' These genes can copy themselves and literally jump around in a DNA sequence. Better understanding of them could lead to improved genetic therapies.

"Bats are the second largest group of mammals in terms of number of species. Transposable elements, which are very common in some groups of bats, alter composition, but perhaps more importantly, regulation of genes when they insert themselves," explains Richard Stevens, associate professor of biology at Louisiana State University. "These genetic changes could be important in the diversification process and may provide key insights especially in terms of understanding mechanisms that generate diversity of species-rich groups, such as bats."

"These transposable elements contributed many of the regulatory elements that tell a gene when to turn on and turn off. So, the fact that these things can move from place to place lets us understand better how genes are regulated," adds Ray.

The team is also investigating 'jumping genes' in flies and the group's research may contribute to a new tool for medical examiners and crime scene investigators. Those experts have long used blowfly eggs and larvae to help determine time of death, but a lot of fly species and their young look alike.

"It's critical that you actually know which species you're dealing with or you're going to get the time of death wrong. Our idea is that we use these transposable elements as genetic markers. Then we can narrow down which species we're dealing with and, therefore, get an accurate time of death," says Ray.

Miles O'Brien, Science Nation Correspondent

Marsha Walton, Science Nation Producer.