Searching by Image Instead of Keywords

Content based image retrieval (CBIR), the technique to search for images not by keywords, but by comparing features of the images themselves has been the focus of much research ever since the web emerged. Consider for instance adding CBIR to Google Images, where you would be able to search for images similar to a query image instead of using keywords. A research project at Penn State University has recently been applied to the biggest aviation photo database in the world with close to 800,000 images. You can search for images similar to a photo already in their database (click "View similar photos") or submit your own query image. Some queries generate better results than others but CBIR is certainly here to stay and will be standard in many image applications of the future.

Think of the greatness to society!

I can't wait to put a nipple into it!

Arm jokes...

and set for goatse!

Location?

What an awful beach [airliners.net].

Wow

I was just thinking about this the other day. I think content-based image search is one of the Next Big Things. Cameras are so ubquitous now (for better or worse), but having to rely on metadata to give meaning to images requires lots of effort up front.

It will be interesting if we ever get to a stage where we can just search for a random object (or person) in a database of photos. Then you could take pictures of everything with an always-on camera and if you need to find where you put your car keys, just do a search.

Re:Wow

If you are only interested in searching for images on your own computer, have a look at imgSeek. http://imgseek.python-hosting.com/ [python-hosting.com]

It's been around for some time now. You can not only use an existing image to search, but also do a rough sketch. Check the screenshots: [sourceforge.net]

Nice complement to what has been presented in this article.

Brilliant!

This that I have an image to search with...which is why I am searching for images to begin with - to find an image.

They would need major manpower to maintain this db

This is just asking for trouble. As most of you would probably imagine, some self-proclaim "comdeian" would post either porn pictures, or pictures that resembles porn body position.

They would need a team of outsource Indian workers to go through each picture one by one!

I am not Indian but...can I apply for the image filtering job?
I said this first, I should get the job ;) .

Top Search

Something with two circles and dots in the middle of each circle.

security issues

Forget slashdotting Airliners.net, how long before the TSA shuts down that website? The trainspotting hobby has already died off following terrorism fears, I can't help but think that other enthusiast sites like Airliners.net will be next.

The ever popular 'Breast' option...

Hmmm...

+"34b"

-"puffy" ... Profit! (Oops, or something, grabbing for a Kleenex)

Similar images

Some Applications of Our Research
1. Airliners.net
A site with almost 1,000,000 aviation images.


Wow !!! I tested their Sample search [airliners.net] and all the results were aeroplane photos !!! Ok, ok the site only has airplanes but still ..:)

On a more serious note the alogorithms seem to look for similatity in the colors and lighting rather than the subjects (for example it shows the interior of a cabin in photos similar to a whole plane in the sky. To really see its effectiveness we need to test in in the real world (google images) . The 'artisticly revealing' photo you always liked ... now you should be able get similar pr0n (^H^H^H^H I mean art) with these algorithms

The Human Brain

Isn't this kind of how the human brain works to identify objects your eye hasn't seen before?

IANABP, I am not a bio-physicist but it seems very much like artificial intelligence to me.

And for 'lynx' users...

... the search engine will support ASCII art image searches.

Some relevant research papers

There's a bunch of interesting papers out there on content-based image analysis and retrieval. Below is a sampling from my bibtex file. Does anyone else have others they'd like to share?

* Finding Naked People [hmc.edu] (Fleck et al, 1996)

* Video google: A text retrieval approach to object matching in videos [ieee.org] (Sivic & Zisserman, 2003): web page demo here [ox.ac.uk]

* Names and Faces in the News [columbia.edu] (Berg et al, 2004)

* FACERET: An Interactive Face Retrieval System Based on Self-Organizing Maps [springerlink.com] (Ruiz-del-Solar et al, 2002)

* Costume: A New Feature for Automatic Video Content Indexing [www.irit.fr] (Jaffre 2005)

One more: automatic film character retrieval

I forgot one more, where specific faces were automatically retrieved from feature-length movies and Fawlty Towers:

Automatic Face Recognition for Film Character Retrieval in Feature-Length Films [cam.ac.uk] (Arandjelovic & Zisserman, 2005)

The objective of this work is to recognize all the frontal faces of a character in the closed world of a movie or situation comedy, given a small number of query faces. This is challenging because faces in a feature-length film are relatively uncontrolled with a wide variability of scale, pose, illumination, and expressions, and also may be partially occluded. We develop a recognition method based on a cascade of processing steps that normalize for the effects of the changing imaging environment. In particular there are three areas of novelty: (i) we suppress the background surrounding the face, enabling the maximum area of the face to be retained for recognition rather than a subset; (ii) we include a pose refinement step to optimize the registration between the test image and face exemplar; and (iii) we use robust distance to a sub-space to allow for partial occlusion and expression change. The method is applied and evaluated on several feature length films. It is demonstrated that high recall rates (over 92%) can be achieved whilst maintaining good precision (over 93%).

and then we have reverse "Googling" for images..

Here is a google game which is reverse of google's image search:

One has to guess the search word which generated a given set of 20 images in google's image search [robinson.name]

When things are moving forward, we have soomthing to talk about "those good ole days" but frankly the game is interesting initially but later gets boring due to the frequent repetitions..

Is it just colour?

I just did a quick search based on this [designer.am] image of a Qantas logo (that's the main Australian airline, in case you're wondering...) It's red, with a white kangaroo in the middle. My theoretical aim was to find photos of Qantas planes.

What I got was an awful lot of red planes - some of which were actually Qantas planes, but I think more by coincidence (i.e., they're red) than design. Many images had nothing to do with Qantas, or even a red plane - they simply had a lot of red in the image.

This is impressive in some ways, but in others it seems like it's simply looking for similar patches of colour. I haven't done enough testing to see what happens if,say, I gave it a half red half green image.

Interesting, but not ready for public consumption just yet. A bit like A.L.I.C.E. the artifial intelligence system actually - neat, but not practical. Yet!

Great!

Now I can find all the other naked pictures of Bea Arthur on the web!

eVision?

I guess eVision [evisionglobal.com] were just too early to market with their visual search engine. Here's a demo or three [evisionglobal.com] of eVe in action.

It sure was cool, just too far ahead of its time...

Several image viewers do this already

Programs like GQview (unix/linux) offer functions to search for similar images, mainly used to find duplicates.

It's not quite "put in an image and find me all the similar ones" but the underlying technology is the same, usually creating some kind of "signature" of each image and then comparing the signatures to find others visually similar.

Great for de-duplicating your por^M^M^MPhoto Collection.

IP Enforcement Nightmare

The Mona Lisa (famous and out of copyright) is often plagarized in whole or in part as part of commercial or satiric artistic works. These types of visual database engines have frequently been explained to me as being able to input the Mona Lisa and get a list of images that used the entirety of the image or just a part (such as the highly-praised subtle smile).

The big problem to me is specifying input. I know the "look" of the Mona Lisa's smile, but even with the best pen input methods I'd never be able to mimic DaVinci's subtle emotion of the smile; my hands just aren't capable of doing so. Using photos of the painting could simplify this, but this almost assumes that I'm only looking for the parody's and commercial exploiters of the image rather than the image itself (after all, I have the image to start with). And it raises the further issue that many photographic reproductions of the Mona Lisa that I can get my hands on are still under copyright and I'd be doing something legally questionable with an image long in the public domain.

Add to this the "infinite number of monkeys" issue where legally litigious companies will use technologies like this to scan the internet for litigation targets. Imagine Disney using a cell of Rafiki from the Lion King to find legally similar images that were created after the Lion King was released even if they were only superficially similar. Now do this for all movies back to Snow White or Steamboat WIlly and you could get to be a real visual mob boss with ownership (or at least threat of litigation) over huge libraries of works that weren't even created to intentionally violate Disney "Intellectual Property".

My need for this technology is small considering the input problems I'd have with my artistic abilities, while the litigation nightmare from large databases of "similar" visual data would seem to be more bothersome than helpful. I rather hope these visual search and categorizing methods don't catch on.

_Mind at Light Speed_ by David Nolte

I've not finished it, but I started a book called "Mind at Light Speed" by David Nolte a while back. He describes three stages of machines of light, and I can't do the book justice here.

However, he put forth the concept of replacing the bit as the common unit of data with actual images - best described as holographic images of light manipulated by light. A picture really _would_ be worth a thousand words in such a system!

He was my professor, it sucked.

Yeah, in my early undergrad days I took a 200 level course for a gen ed requirement of discrete mathmatics. Wang was the professor, and too this day I haven't had a course that was as difficult or completely freaking insane as the one he gave. Glad he is doing more research and less teaching.

This reminds me of Gibson's Pattern Recognition

Pattern Rcognition [williamgibsonbooks.com] is a novel by William Gibson, basically set in the present day or very near future. Image based search plays a central role in the plot. It's a very good read.

How does it do that?

I was looking at a picture of a plane on that web site and there was a link that said "Click for similar images". And what do you know? It brought up more pictures of planes. This is amazing stuff. How did it understand that I was looking at a picture of a plane?

more sophisticated than colour matching

Whatever algorithm they're using, it seems to be sensitve to the horizon line, colour, shading, orientation of the aircraft, etc. It seems to be operating at the level of a pigeon (who have been shown to discriminate photos depicting trees, water, and particular people - as well as art by Picasso and Monet. See http://www.pigeon.psy.tufts.edu/avc/huber [tufts.edu] for other examples. It will be some time before algorithms can match on the basis of model numbers and such. It took humans quite a while to evolve a cortex to enable such fine discriminations.

There is a GNU project related to this GIFT

From gnu.org:

The GIFT (the GNU Image-Finding Tool) is a Content Based Image Retrieval System (CBIRS). It enables you to do Query By Example on images, giving you the opportunity to improve query results by relevance feedback. For processing your queries the program relies entirely on the content of the images, freeing you from the need to annotate all images before querying the collection.

GIFT [gnu.org] It worked pretty well for me in the demos they linked too. I have been waiting for this type of application to gain momentum.

Sorry, someone else is currently using the system.

Got this error when testing out the system, they certainly need to fix this, especially if they want it to become popular.

Would it work for animated .gifs?

'Coz I'm looking for more information on this image. [pacific.net.au]

It says "multi lock on" and a date, but all Google reports is other forum posts looking for the creator of the image. Apparently, there's a high-res version of it too.

Apps Already do this.

iView Media pro already does this. You just tell it to find dupes and set the tolerence to loose. http://www.iview-multimedia.com/ [iview-multimedia.com]

It's db of airplane pictures, they are all similar

I clicked on the search for similar pictures link for a 727 and one of the results was a helicopter.

I guess anyone can get a research grant these days.

Pretty controllable test

A.net is very stringent on the photos they accept. You can submit hundreds of photos, and get rejected for such things as 'badmotive' (a runway sign blocking a single tire), very mildly soft focus, and lots of other pretty anal things (IMHO). So while the image count they are dealing with is high, the obvious resulting similarity among images will result in a high number of matches.

Now, do this for something like Google Images or PBase or collections spanning infinite numbers of subjects and image sizes, then I'll get excited.

No, I've never had a rejection from A.net, I've never submitted there. Two minutes in their forum will tell you how anal their 'screeners' are, for whatever reason. It's just freakin' pictures of airplanes, for chrissakes.

You can tell pretty quickly how they match

They are doing it based upon the shades of color in the image. So if you query for a image of an aircraft in flight with a lot of white clouds behind it, you get more of the same, but you also get aircraft parked on snow-covered ground.

Music

So when can we search for music? I'm trying to find this song that goes like "dah daahh, dumpiti, dum dum"? Any answers?

Spotlight

Imagine if this was incorporated into Spotlight, not having to rename your entire photo library.

Is this a joke?

I've tried two different images of airplanes; one of a bright red flying car on bright green grass and one of SpaceShip One against a deep blue sky. Both times, the results looked surprisingly like my query images in color composition only. Red planes on grass and white planes against a blue sky. Inauspicious start.

Next experiment: I took a picture of a highly distinctive plane, a harrier, climbing at a steep angle and viewed in profile. I got, in return, a list of passenger jets, and even a helicopter. Hardly surprisingly, all of the result pictures had the same bluish white sky as my original image. That was literally the only similarity.

According to the introduction on the search page the heuristics used compares colors, contrast and shapes in the images themselves. I saw no correlation whatsoever between shapes, and any correlation in contrast seems to be to be the result of the search engine simply looking for images that contain the same colors in a similar ratio to the original. In short, nothing to see here, move along.

On the other hand, one of the projects listed under the Penn State University link looks fairly fascinating. The Riemann a-LIP project [psu.edu] (automatic linguistic indexing of pictures) doesn't allow user input of images, unfortunately, but it does show some fairly fascinating attempts at verbally qualifying image data. For example, it describes a blue and orange mandelbrot as pattern agate shimer abstract scene, and a sunset over a lake as Berlin Devon Namibia landscape lake scene. Okay, it may still need some work, but it sure beats the hell out of the "find the same color airplane engine".

Oh, you mean like imgseek?

Oh, you mean like imgseek [python-hosting.com]?

Requirements

If image search is going to be the next big thing bots are going to need to crawl deeper and more often... Most engines probably don't have the resources currently to support the extra space/load.

But what I don't understand is...

...why does it show me pictures of donkeys when I use my school portrait for the search?

Oracle interMEDIA does this for a while now?

see http://www.oracle.com/technology/products/intermed ia/index.html [oracle.com] (oracle)
This document was written in 2002 (and that version is as old too)
I can remember some sales guy saying "You can look for a couch that's like this one. But blue."
So that means it should do a bit more than just color patches..
and then there is Google images..

This is similar

This Photo ID has not been indexed in the similarity database yet.

ummm...

It just keeps coming up with photos of airplanes.

Purdue University's 3D Shape Search

Purdue also has a 3D shape search. More can be found at Here [slashdot.org].

altavista

I'm pretty sure altavista had this feature several years ago, but removed it. I remember that it worked fairly well. Does anyone know what happened to it?

Professor Wang

I took Professor Wang's networking class this Semester and have him for databases in the fall. He is a very interesting professor and was in the same class as the people who invented google.

Uh-oh.

Given that it apparently has trouble telling the difference between an SR-71, a Chinook helicopter and JFK airport's formerly-TWA terminal [airliners.net], among other things... I think it could use some tweaking.

Of course, given the usual course of things, it will instead be deployed at JFK's formerly-TWA terminal, assigned facial recognition tasks, and immediately declare everyone to be among the 10-most-wanted terrorists. I can't wait.

How about just searching for an identical image?

This is great and all, but I would love for a way to upload a graphic file to a search engine and have it report back any identical (or nearly identical.. perhaps a "threshold" setting) copies on the web.

This would be useful to me as a photographer to see if anyone out there is using my photos.

Worked on something similar

About a year or so ago, I and three other Masters students worked on a similar project at the University of Southampton.

I've not RTFA (not had the time), but our approach was to split the images into segments (based on colour and texture) which were assumed to be objects. The segments would then be analyzed for various feature vectors, such as shape, texture, colour etc. These vectors would then be added into a database of numbers, and finally the segments grouped, giving a collection of classified sections which (hopefully) represent similar objects.

From related metadata such as keywords, you could then hope to build up an idea of what keyword matches which section. You could also come up with a relevance between two images, and thus search for similar images.

We didn't have enough time to make it bulletproof by any means, but our limited results were very promising.

Sorry I can't find the paper, but we've got some screenshots of the application here [soton.ac.uk] and here [soton.ac.uk] (you can see false colouring applied to the original image to display the segments)

Reee diii cuuuu looooooooous !

Ridiculous.

Image search will kinda work for airplanes in this database,as there are a very limited set of airplane model numbers, which are going to be attached to each photo.

But if the database didnt have these text clues the image search is going to be unlikely to see the similarity between an 747 in the air, as seen from the ground, with a head-on view of a 747, or one at the gate, or one in a hangar, or one in twilight, or one of a different color.

Maybe it could be done by outsourcing the task to India?

Too Literal

I was trying this out a bit, and have to admit that it's cool that something like this exists at all.

However, I think it would be better if it were able to realize what the 'background' was and filter it out. (Though I couldn't begin to guess how you'd do this.)

For example, I searched for this image [airliners.net]. Many of the results [airliners.net] are of something completely different, such as a white jet. Which is nothing like a camo helicopter. But the sky and the ground are pretty similar, and I think that's how it's matching.

It's incredible that we got this far, but I think there's still a long ways to go before it's at the stage where you put in an image, and are awed at how quickly it works.

Also note that I'd tend to think this would exponentially more difficult than searching HTML files, so it might be much more expensive to implement large-scale.

CBIR is the same problem as AI

In general image understanding is equivalent to general AI. We won't get a CBIR system that works well before we get an AI that works and vice versa, because people expect to be able to match the *content* of the image they submit as template and not the general appearance of the image. The problem is then too unspecified.

Even in the restricted context of aeroplanes this is not a trivial problem. Someone in the list of replies submitted an image of a warthog (A-10) and got nonsensical results. Somehow the CBIR system would need to be able to infer a model of the A-10 from a given random 2D projection, and match it to the other 2D projections of the same A-10 model that they do have in the DB. This doesn't sound impossible but it is hard, and I suspect the Penn State people didn't do that. Instead they are probably matching on colour, texture, general appearance, etc.

This is not to say that CBIR is not a nice problem to apply new image processing/image analysis algorithms to, which are developed all the time.

Video loops etc

When this becomes a little more refined, it will be nice to apply to frames of video to assist in creating video loops. For example, you mount a camera near the coast, and get 3 hours of footage of the waves crashing against the rocks. You want to make a nice loop of that, which can play indefinitely, but you want to avoid that 'jump' that always seems to occur. Find small sections of the movie where say...10 frames in a row are similar to 10 other frames... It'll take out a lot of the usual pain.

Re:wtf?

Because it still has problems - you'll note that the pictures seem to be compared simply based on color similarity. That's the same thing imgSeek [python-hosting.com] does (a great program for sorting and searching your photos) on photo searches. It works wonderfully if you're searching a very limited picture subset (say, airplanes), but if you search a wide variety of pictures, the results can be quite amusing.

Re:wtf?

don't mod me as redundant you niggers

Re:wtf?

Google actually did take this technology and try it. The first version of their image search had a "find similar" link next to every image. These tended to work okay at first (they weren't great, but you usually got enough photos back that you could visually scan them and find something of interest that was related to the original image). After a few months, for some reason, the "find similar" links started returning increasingly nonsensical results. After it degenerated to the point of near uselessness, they took the "find similar" link away from the image search results. I expected it to turn up again once they got the kinks worked out, but apparently they just decided to stop working on it.