Researchers Teach Computers To Perceive 3D from 2D

hamilton76 writes to tell us that researchers at Carnegie Mellon have found a way to allow computers to extrapolate 3 dimensional models from 2 dimensional pictures. From the article: "Using machine learning techniques, Robotics Institute researchers Alexei Efros and Martial Hebert, along with graduate student Derek Hoiem, have taught computers how to spot the visual cues that differentiate between vertical surfaces and horizontal surfaces in photographs of outdoor scenes. They've even developed a program that allows the computer to automatically generate 3-D reconstructions of scenes based on a single image. [...] Identifying vertical and horizontal surfaces and the orientation of those surfaces provides much of the information necessary for understanding the geometric context of an entire scene. Only about three percent of surfaces in a typical photo are at an angle, they have found."

leaning tower

[ZivZoolander]

Wonder how this will handle those optical illusion photos. like me nocking over the leaning tower of pisa, or holding hte statue of liberty.

Directly applicable to the car racing AI grand....

[ChrisGilliard]

...challenge. I think Carnegie Mellon wants revenge against Stanford for beating them in the 2006 DARPA grand challenge. Maybe 2007 will be Carnegie Mellon's year to win the grand challenge. If this happens, we're only a hop skip and a jump to having these things drive us around (esp on freeways).

Imagine the Possibilities

[Valthan]

One could concievably take a pictures of a city, upload them to this program, stich the pieces together and then import it into a game world. How awesome would it be to actually be able to run around a city(say Toronto) and do things you always wanted to do... (dropping a penny off of the CN tower and having it hit someone :D)

Typical photos?

[doti]

Only about three percent of surfaces in a typical photo are at an angle

What typical photos are those? No faces, people, trees or any organic thing?
No cars? No roofs?

That's been possible for years...

[Penguinisto]

It's called Canoma. Problem is, it's been limited in scope, and the original company that wrote it (MetaCreations) went out of business ages ago: It still exists as an orphan that Adobe has been sitting on, however [canoma.com].

(MetaCreations also produced Poser, Bryce, and Carrara. - all three of which are still alive and in use by the 3D hobbyist market).

/P

Using multiple camera angles...

[jsharkey]

Last year I worked on an Artificial Intelligence project [jsharkey.org] to recognize objects from several video angles. It takes 2D images (from camera video) and turns them into a 3D path.

It uses a super-neat concept called "Geometric Hashing" which can be used to recognize an object regardless of size, rotation, or even partially-obscured regions.

Play with it yourself!

[cranesan]

http://www.cs.cmu.edu/~dhoiem/projects/popup/index .html [cmu.edu]

Looks like some of the software they wrote to do this has been GPL'ed.

Re:Well...

[jackbird]

I've used Photomodeler and Canoma, and made camera mapped environments in 3D software by hand for years. It is incredibly nontrivial. it is a lot of blood, sweat, tears, handpainting, and a not-so-terribly good result. Some typical problems:

• Camera barrel distortion
• chromatic abberations
• hot colors in high-contrast areas of digital photos
• JPEG compression artifacts
• specular highlights and reflections
• lens flares and blooms from those specular highlights and reflections
• clipped/out of gamut areas
• occluding objects like trees, parked cars, signs, telphone poles, pedestrians, trashcans, newspaper vending machines, etc., etc., etc.
• occluding objects like other buildings in aerial photos
• only being able to shoot certain details from awkward angles
• not being able to shoot certain details from any angle at all
• horrendous texture stretching
• perspective problems with concave/convex detail like window ledges, cornices, awnings, etc., etc., etc.
• stuff you forgot to photograph
• different lighting conditions when you go back out to shoot the stuff you forgot to photograph
• unavailable architectural drawings
• paper architectural drawings
• poorly-reproduced paper architectural drawings from 1912
• architectural drawings that bear no resemblance to the conditions onsite
• CAD files aligned to state survey coordinates so large that the single-precision floats in most 3D software starts scrambling the model due to rounding errors.

  as I said, nontrivial.