Photosynth Demo

A couple of days ago Microsoft labs released a demo of their new Photosynth software on the web. Photosynth allows the aggregation of social picture networks (a la Flickr) into a completed image in addition to allowing a level of depth to image browsing previously unavailable. There is also a very impressive video of the demo available.

Huh?

Photosynth allows the aggregation of social picture networks (a la Flickr) into a completed image in addition to allowing a level of depth to image browsing previously unavailable.

That appears to be syntactically tolerable English. Semantically, though, WTF?

Re:

Let me translate.

Pretty pictures.

Re:Huh?

Photosynth allows the aggregation of social picture networks (a la Flickr) into a completed image in addition to allowing a level of depth to image browsing previously unavailable.

That appears to be syntactically tolerable English. Semantically, though, WTF?

This lets you take all sorts of pictures of your room, and will automatically assemble them into a 3D environment. It will assemble your photos to look like an RPG, instead of a slideshow.

Using the example in the video...there are hundreds of online collections of people's photos of Notre Dame cathedral. Each photo is of a different part of it, from a slightly different angle.

This software takes all those different photos and assembles them into a 3D representation of Notre Dame cathedral, where you can look at any of the individual photos.

In addition, if someone identifies one of the saints in a statue on the cathedral, when you take a photo of it and your photo is added to the collection with the software, your photo will also have that saint identified--thereby enhancing the data contained in your photo.

- RG>

Re:Huh?

I don't think this technology has that much to do with social picture networks in particular, I'm not sure using it to index images is all that compelling. What would be more useful is inputting some images from different angles (or a video) and getting back a .3ds texture-mapped geometric model. Reconstruction of gometry from imagery has been a big research topic for ages but I'm not aware of any effective, user-friendly software to do it.

Re:

Yeah, it's pretty decently cool, too. Personally, I thought the magazine and the car ad with highly detailed information "printed" really small was as interesting a concept at anything-- it looked like it might provide a reading experience that would make sense for an online magazine, and the small print bends the concept of your printable space in an interesting way. So long as there are sufficient hints that the tiny text was there, it would allow you to put a lot of information into a small "space".

Th

Now that's a real summary!

Photosynth allows the aggregation of social picture networks (a la Flickr) into a completed image in addition to allowing a level of depth to image browsing previously unavailable.

Slashdot summary entices the accumulated aggravation of social comment communities (a la Digg) into a aggregated juxtaposition while interspersing levels of irritation heretofore unimaginable

Some impressive things

Unlike the first set of posters I managed to get over my self importance and watched a couple of seconds of BMW ads to see the actual video.

I liked the initial viewing of large quantity of hi-res images and the smooth zoom. The aggregation of many thousand flickr images of the Notre Dame (including one of a poster on a wall) into a 3-D image was fantastic.

C

One step forward!

At least now someone at Microsoft seems to know _what_ to buy, this is some pretty amazing technology. I just hope that someday it will be available to other OS'es too.

Re:One step forward!

One thing that amazes me of Microsoft is how, having so many bright people at MS reasearch, most of their stuff is so bad, and/or lacks innovation. (I know part of this came from some other company they bought, but some of it is original from MS, I've read a paper related to this technology).

Every single paper I've seen from MS research is great. Well done!

(from someone developing computer vision on linux)

Re:One step forward!

Microsoft didn't buy Photosynth. It bought Seadragon. The Photosynth client is indeed built on Seadragon's client, but the idea behind Photosynth (which was a joint University of Washington/Microsoft Research project called PhotoTourism) significantly predated the Seadragon acquisition, and there was a working client. When Microsoft decided to reimplement the client as a technology preview, that's when the Seadragon team and client came into the picture.

That said, Seadragon's technology is great. It's a fantastically smooth way to browse arbitrarily large images or collections of images, and it was a good acquisition indeed.

(I was on engineer on the Photosynth team.)

The Humane Environment

This zoom-ability of the first part has a lot in common with the ideas behind Jef Raskin's The Humane Environment http://en.wikipedia.org/wiki/Archy/ [wikipedia.org].

The second part, however, shows marvellous stuff. Especially if what I think he did, was search for patterns in images, and compare those for unique objects to collect a library of images of a single object.

This guy and supposedly his group shouldn't work for Microsoft in my opinion, but would perhaps feel more at home in a fundamental science laboratory. But I think my opinion on this is slightly partial.

B.

It's here! Web 2.0 is HERE!!!

I can honestly say, without hyperbole, that this is the first time all those promises of what the web can really do - interconnectivity, automatic synaptic contextual linking, user generated content, and god-damned cleverness - have finally come together into something which is un-fucking-believable!!

All those next-stage, new-wave, super-hyped ideas that generated enough excitement to get a survivable user-base just kind of passed me by, because they only ever seemed to be minor amplifications of what we already had. But this... this is something totally new. And utterly, utterly incredible!

I'm so excited by this it's making me feel sick! TECHNOLOGY! INTERWEB! I take it all back - forgive me for my lack of faith! I LOVE YOU!

And by the way, that "content only limited by how many pixels are on the screen" idea has been a long time coming, and I'm deeply happy that someone's solved it. I could never understand why we use raster-imaging for computer games because it's a squillion times quicker than ray-tracing, but nobody had applied the same idea to other applications. Now I feel justified in wondering, and I'm so pleased with the result!

Re:It's here! Web 2.0 is HERE!!!

I could never understand why we use raster-imaging for computer games because it's a squillion times quicker than ray-tracing, but nobody had applied the same idea to other applications.

I don't think that basic rasterizing engines are the limit. The limit is that the source data for all these pictures are tens or hundreds of gigabytes (and in the future, conceivably terabytes). Somewhere in the assembly and cross-correlation of all this data, they have to be generating LOD's (levels of detail) and dynamically loading / managing MIP-maps to keep the loaded dataset to a reasonable level. This is the hard part since "reasonable level" for loaded imageset size is probably currently a couple hundred megabytes or much less. You can probably load more data into RAM but try maintaining a 60FPS refresh with a gigabyte of textures - especially on a laptop or basic computer.

Once you've done this you can use a variety of display techniques... the main reason to use basic texture-mapping / flat rasterization is that sources are photos which are basically a pre-lit "flat" textures.

However, if you can generate a 3-D model and can separate lighting / color information (perhaps using combinations of day and night pictures or varying lighting from different photographs), it would be then possible to perform simple ray-tracing or other hybrid renderers -- think how cool it would look to have a dynamic artist's sketchpad with these images "penciled" in realtime. There are already high-frame-rate (near-realtime) ray tracing demos already out there for CELL and X86 that render moving images at a lower-res for higher-interactive frame rates and then when not-moving, render high-quality image stills that are quite impressive.

Amazing Software, Lackluster Comments

This software is absolutely amazing, especially when you consider the programmatic side of this. People bashing this without actually watching the video AND playing with the operating demo are really missing out. You don't have to like it but at least have a reason that shows some form of intelligence. Not just "the intro was poorly done".

Covered on /. a year ago...

Right here [slashdot.org].

Data aggregation

Near the end of his presentation, the guy sums up the technology as taking all of these separate images from various sources on the net and figuring out how they all interlink to present a larger, more coherent picture. He got applause.

My first thought was about the U.S. government's "total information awareness" project, where they're trying to take lots of separate pieces of info (which are already available to law enforcement) and interlinking them all together to provide a more coherent picture... but most people consider that to be evil.

Granted, the government isn't doing it with vacation photos, but the idea, of finding pieces of data that are related and finding out *how* they're related, is the same. The difference in people's reaction to it, I can only attribute to the fact that people see the photosynth guy as good, and the government as evil. But I don't agree that the goodness or evilness of an action is solely determined by the goodness or evilness of who's doing it. The U.S. gov't tries this and fails. It expects that it can invade foreign countries and install friendly governments and torture people because it's "the good guys", yet the soviet union did those same things during the cold war and we admonished them for it because they were "the bad guys".

So, where am I going with this rant? My point is this: You can't blame somebody for connecting the dots. In fact, that seems to be one of the things that we, as humans, are particularly good at. So, if you think that this photosynth thing is fine, then I think you've got to grant that the TIA project is fine. Now, you could argue that some particular bits of information shouldn't be available, but the piecing it together to form a more coherent picture... I can't come up with an argument against it that I consider defensible. Sure, it makes me uncomfortable, but that's not an "argument".

Vast Desktop...

Actually, as I looked at the demo, I couldn't help feeling like all that virtual space was looking like a damn nice desktop environment. Nevermind the part of the demo with a flat-on scrolly-zoomy desktop, as nice as that would be (Seems obvious in a way too... And wouldn't it be nice if Leopard had that instead of "Spaces" ?). But imagine the notion of opening up an application and instead of just popping up a new window it creates a new space - within the desktop virtual space - and brings you into it. You can always pull back and move around to another window or workspace, but while in it you'd be totally immersed.

I dunno, I just like the notion of immersive environments, especially for conceptual learning. I think we're going to see a prevalence of this kind of interface in the near future.

Re:Does anyone have an actual video of the demo?

Then you closed the window about 10 seconds before the demo started. Keep watching.

Re:Does anyone have an actual video of the demo?

This system was demoed a while ago, I think at siggraph. There are some videos on the original university of washington PhotoTourism page. [washington.edu]. Also here's a repost of the video on youTube. [youtube.com]

Also there's microsoft's page [live.com], which has the demo (I don't think that's new either). It seems to have some longer videos

Non-newness and marketing hype aside, this software is frickin' awesome. It lets you view and tag photos organized in a 3D environment that reflects where the photos were taken. It should be particularly useful once cameras have GPS built in.

I imagine the reason the software is still in the demo phase is because it's very difficult to take a large number of photos and reliably figure out where they were all taken from. For the demo purposes, Microsoft probably hand corrected a lot of the placements. Even so, everyone I've shown this too thinks its often (even non-slashdot readers!)

Re:Does anyone have an actual video of the demo?

I was one of thee engineers that worked on the first release Photosynth. It's a great team, and it was a super fun project.

I can tell you that we did not tweak any camera positions by hand. The only real "editing" we did was to eliminate pictures that just didn't correlate well, generally because they didn't have enough feature points in common with the rest of the photos. We didn't tweak any camera positions, but the camera positions (i.e. the locations of the orange camera frusta when you have frusta turned on) are a best estimate, which is subject to some error. Same goes for the projection planes.

What's great about Photosynth is that from the perspective of anyone outside the computer vision community, it appears to be magic. Enough so that lots of the blogosphere was convinced that we somehow "authored" the 3D point clouds. Nope. It's more or less an automatic (albeit somewhat prolonged) process. The hard work is done as a big preproceess, then the client consumes largely precomputed data.

It'll be cool to see Photosynth in action in BBC's upcoming How We Built Britain piece that was announced on Live Labs [live.com] today.

I did a video interview [onten.net] about Photosynth a while back which is targeted at a non-technical audience but still might be of interest. (And I wrote the music for the original video [live.com] at Live Labs.)

We know you tried this at least once...

Clearly then, you'd be the one to ask... What happens when you enter "Jessica Alba" into Flickr and use photosynth?

Re:press release

I don't get the point of that part either, but keep watching. A couple minutes into it he moves on to the real meat of the demo, and it's pretty astonishing. I won't spoil it except to say that if I'd seen it in a sci-fi movie I'd probably have dismissed it as very cool-looking but totally unrealistic.

Re:

In the interests of openness, would you mind publishing these calculations of yours? I'm sure we'd like to see your quantification of the open-source development process, particularly for software as complex as this evidently is. Thanks.

Re:Interesting

In the interests of openness, would you mind publishing these calculations of yours? I'm sure we'd like to see your quantification of the open-source development process, particularly for software as complex as this evidently is.

• Day 0: Someone registers a project on Sourceforge, commits main.c to CVS.
• Day 3: It's noted on Digg, Reddit
• Day 30: Slashdot links to project, hails it as "the Photosynth killer", misspells project leader's name. Commenters gloat about M$'s lack of innovation, speculate on the throwing of chairs in Redmond, argue about atheism and gun control.
• Day 33: Slashdot dupes story, misspells "Photosynth", "killer" and "the".

The Software is AWSOME! However the delivery...

I decided wade through the hype/ads/blah, and came across a really cool piece of software. It takes thousands of flickr images stitches them into a 3-dimensional mosaic, all just through software. No special on-site 3d imaging hardware, just a program compiling everyday images of something. It does this through some very advanced image recognition. If you can brave the ads, it IS worth it.