3D Virtual Reconstructions From Microsoft

Lord Satri writes "New around the corner, Microsoft Live Labs' Photosynth, will 'take a large collection of photos of a place or object, analyzes them for similarities, and displays them in a reconstructed 3-Dimensional space.' There's a demonstrational video and a 'smart photos' example page. From the site Very Spatial: 'The word is that Photosynth will be available for free, at least at first, but no word yet on an exact release date.' I must admit, seems like Photosynth may offer interesting features with an clean interface. This tool will directly compete with Stitcher, and to some extent, Google SketchUp. The virtual world reconstruction tools market is getting crowded, and competition is good. Microsoft doesn't yet have software to tie a photo library with Windows Live Local (Google does), but don't be surprised if it comes to life."

Dear Aunt...

If this software is half as good as the famous: "Dear Aunt, let's set so double the killer delete select all.", then we at least should be able to use it to create Escher like visual paradoxes, but if anyone is hoping to seriously convert a few pictures of themselves into 3d models, they may find themselves in a Dali like nightmare.

Re:Dear Aunt...

I agree with Erich Schubert's blog post (http://blog.drinsama.de/erich/en/2006073102-micr o soft-vapourware [drinsama.de], found via http://planet.debian.org/ [debian.org]) that it's more vapour than some E3 "rendered in-game" footage. Citing the UWash Photo Tour (http://phototour.cs.washington.edu/ [washington.edu]) whose technology ultimately makes up PhotoSynth, the processing power required is of the scale of two weeks to place 597 of 2635 images using a 3.4 GHz P4. I doubt that too many home computers will have the grunt to do that on a reasona

Google Earth From User Photos?

This basically looks like Google Earth based on user photos and not satellite photos. I find it quite interesting, but a little too much of a gimmick right now. Pixel zooming into a picture is NOT the same as diving into the scene and looking around like its a virtual world.

http://religiousfreaks.com/ [religiousfreaks.com]

Re:Google Earth From User Photos?

This isn't really pixel zooming, it "zooms" by determining which of the pictures of the scene it has is closest to the view selected by the user, and switching to that one, rather than zooming in on a specific image. So if you, for example, select to view the head of a statue from a picture of that statue, it looks for a picture of the head of statue, then views that. It's pretty neat.

Re:Google Earth From User Photos?

Man-made objects should be pretty easy for software to put together. We tend to build things in a non-random fashion. I'd be a lot more interested to see how well this software works with natural objects like trees (blowing in the wind, no less) than it works with buildings.

Real Estate

This software could revolutionize buying real estate remotely. Imagine, an agent goes in with a cheap digicam and takes a bunch of shots of the house they're selling. They load them into this software which creates a 3D, navigable model of the house, which someone can browse via a browser plugin.

Sure, this has been around for a while with VRML, but it was complicated and costly for an agent to do. From the looks of this software you can use normal photos as a base. Anyone could create 3D tours with this.

Re:Real Estate

If you're a real-estate agent, it's not really that big of an investment to buy a Kaidan 360 lens. [photo.net] It only takes a few seconds to take a picture with, and you only have to snap one picture. Their software builds a quicktime VR of the environment. In my opinion, a quicktime VR gives a better presentation of an environment than what was seen in the MS software anyway. The only benefit to the MS stuff is that it will let you zoom in on particular features you photographed more close-up, where I believe that Qu

Re:Real Estate

The only benefit to the MS stuff is that it will let you zoom in on particular features you photographed more close-up

It doesn't just let you zoom in (which, by the way, Quicktime VR can do too); it lets you look at the scene from any arbitrary perspective. It's the difference between just standing still and looking around in Quake and actually running around the level.

The world is not static

What about cars and people that change from picture to picture? It's not possible to match them. Does their reconstruction algorithm try to erase them, or to merge them?

Escher

What happens if you throw some Escher drawings at it?

Obligatory Blade Runner

Deckard: Enhance 224 to 176. Enhance, stop. Move in, stop. Pull out, track right, stop. Center in, pull back. Stop. Track 45 right. Stop. Center and stop. Enhance 34 to 36. Pan right and pull back. Stop. Enhance 34 to 46. Pull back. Wait a minute, go right, stop. Enhance 57 to 19. Track 45 left. Stop. Enhance 15 to 23. Give me a hard copy right there.

Re:Obligatory Blade Runner

But keep in mind Microsoft's speech recognition woes..

A more likely transaction might be: "Enhance 224 to 176. Enhance, stop. Stop. STOP. Dear aunt? Move in, stop. Pull out. No, pull OUT. Pull out. Pullllll out. DAMN IT. Dear aunt? Track right. No, RIGHT. Whoa.. okay... left. Pull left. LEFT. STOP. FUCK!"

Re:Obligatory Blade Runner

I never quite understood how that photo-magnification/voice-recognition machine that Deckard uses could reveal an object that's behind ANOTHER object in the original (apparently 2-dimensional) photo.

I don't know how much processing power it would take, but if I were going to write software like that, it would:

- Build up a basic 3D model of the room based on what's visible in the photo.
- "Sketch in" the missing parts using a combination of interpolation and looking at cast shadows.
- Map the known colours fro

Not as 3D, but be sure to check out

I have to say, this would be pretty neat - take all of the images on Google Images for instance and be able to take a high-def virtual tour of places around the world.

Be sure to check out PlayAnywhere [on10.net] too - another neat tech that's being made over at Microsoft Research.

What are the legal implications?

Looks like panarama software on crack. Lots of legal implications I would think - depending on how the photo's are shared or linked - since it is taking photo's that you may or may not have shot and combining them all together - the question might be "who owns the final composite?".

Looks amazing though - can't wait to see it come out.

Cool use of technology

Upload Natalie Portman.

Then fly over her 3D body in realtime. Excellent!

Wikipedia, not MSN Encarta

Their website shows wikipedia [live.com], not MSN Encarta :)

I didn't take all of my happy pills this morning

But this isn't even alpha. It's blogoware - concepts and a video trailer.

Wake me up when it's over.

Don't use the word "tie"

Microsoft doesn't yet have software to tie a photo library with Windows Live Local (Google does), but don't be surprised if it comes to life."

OK, I won't be surprised.

I also won't be surprised when slashdotters gush and fawn over Google's product, then go ape-shit over Microsoft "tying" a software product to Windows Live Local.

Good for Investigations

I can see where this would be a big help in investigations, journalistic, scientific, criminal, etc. Reconstructing a 3-D scene would help understand where people and things were when something happened.

Today there are mic's placed in some high crime areas that identify a gunshot and where it happened. Cameras placed at strategic locations would complete the "picture".

The next step - video

The next logical step (as the algorithms improve, hardware gets faster, and demand grows) will be to do the same with video. See http://www.bigfootencounters.com/files/mk_davis_pg f.gif [bigfootencounters.com] to see a cursory example of how motion picture data can be used to build a persistent environment.

Another poster earlier in the thread speculated that a real estate agent could photo a house to make a virtual tour. Even better, maybe, would be to just carry a high def video camera of some sort through the house, waving it around to get at least a little bit of footage of everything. With that data, an intelligent program could composite a 3D representation with even fewer blackout spots. Combine this with an accelerometer/gyro field that gives a non-software correlation to the video stream, and it's essentially bulletproof.

In the form demonstrated, this is a fantastic heavy duty software solution, but physical tracking data would both make this job easier and improve the quality.

I suspect that in the near future we will see the following technologies made ubiquitous in cameras:
1. GPS
2. Tilt/Compass
3. Accelerometer/motion tracking for video.

Items 1 and 2 would enable any camera to provide very accurate geo-located data. #3 with video gives you tracking where GPS fails plus the super accurate tracking data needed to take this to the next level.

"But Chairboy, you tool, why would the camera companies go to the expense?"

The features listed have become incredibly cheap (both in cost and power consumption) over the past few years. Within a couple years, it'll probably be hard to NOT have them in one of the shared chipsets the camera manufacturers use, and at that point, why fight it?

I wonder how it scales?

I'm rather curious to see how well their approach scales. For example, what if you just dumped all the 1,853 photos of Times Square from Flickr [flickr.com] into their interface? Scaling even more, in the future could one use this to aggregate all the photos in a particular city, or even have a Google Earth-like interface aggregating photos from all over the globe and integrating it with satellite data? There's some interesting computational problems with arise in trying to find correspondence between that many visual features.

I'm also like to see if they can deal with pictures taken at different times of day. I'm guessing it's still too difficult to actually adapt a day image to a night image, so it'd probably just end up treating photos taken at different times of day as different scenes.

Re:Open source sticher? Nasa?

Yes, there is this software: http://user.cs.tu-berlin.de/~nowozin/autopano-sift / [tu-berlin.de] It is GPL'ed but the problem is that it is using a patented algorithm (SIFT features) so it is not free to use in commercial applications without paying I guess.

Re:Open source sticher? Nasa?

I use Hugin Panorama Tools [sourceforge.net]. I think it works really well, though I don't have a lot of experience with other stitchers. It can do panorama stitching, and also correct for barrel distortion and such.

Re:Open source sticher? Nasa?

There was a long discussion of pano tools, both free and commercial, over at dpchallenge.com [dpchallenge.com] a while back. That link is to the first page where the discussion starts, way down at the bottom. From that thread, it would appear that, while a major PITA to install and learn to use, Hugin produces results that are typically at least as good as most of the major commercial tools and are far better than many of them.