Automatic Image Tagging

bignickel writes "Researchers at Penn State have applied for a patent on software that automatically recognizes objects in photos and tags them accordingly. The 'Automatic Linguistic Indexing of Pictures Real-Time' software (catchy name) trained a database using tens of thousands of images, and new images have 15 tags suggested based on comparisons with objects or concepts in the database. Not sure how you identify a 'concept,' and they're only talking about having one correct tag in the top 15, but still cool."

Not shockingly...

The vast majority of the images on the internets including The Google include "Pornography" in it's top 15 tags suggested. The accuracy rate is surprisingly high.

That Sucks

Researchers at a publicly funded institution are using their research results for personal (financial gain). Pennsylvania's tax dollars at work? How is this legal?

The other 50% is the problem

I've seen lots of systems like this. The problem is in the 50% of the images that don't work, so basically you have to manually tag 50% of your images.

I saw an interesting one about 10 years ago. It took an X-Ray image, did an edge detection, converted all the edges to a slope vs distance 2D plot, and conerted edge curves to a radius and distance plot, then used a kind of statistical correlation algorithm to pick which part of the body the image was from. I could imagine that you could apply something similar to the luminance of an image to pick out objects, and then maybe do some color transforms and stuff to improve results. The article says they do it in 1.4 seconds per image though, which is impressive.

Prior art?

Not RTFA to be honest, but can I claim prior art?

http://www.relle.co.uk/papers/2003-Content_Based_I mage_Retrieval__A_Words_and_Pictures_Approach.pdf [relle.co.uk]

We didn't have enough time to train the system properly, but itstarted off well...

LIPS

You've got to hand it to those cunning linguists at Penn State.

Google, others doing similar research?

I'm sure a lot of research is being done in this area, in fact there is lots of interest implement this sort of thing in DSP for robot vision. How much of what this patent covers overlaps with what the others are working on? Is this something completely out from left field or does it fit the trend of where this area research was headed anyway?

A video an the subject

Luis Van Ahn did something almost the same, his idea though is to use humans aswell.

View the video on Human Computation [google.com]

retrievr

For those who missed retrievr...http://labs.systemone.at/retrievr/ [systemone.at] While this is good for hours of entertainment, i hope what theyre promoting is better.

Nobody is impressed by this

FTFA:

The analysis takes about 1.4 seconds per image and in 98 per cent of tests suggests at least one correct tag in the top 15.

I suspect you could generate a list of 15 sufficiently vague words that would cover 98% of all images. Here's a start: people, sport, animal, trees...

I swear I saw a butterfly

at least that's what I told the psychologist. Then on the second look, it looked like splotched ink on a paper that was then folded in half... I hope this software doesn't think like me cause at the end of it all, I saw a segfaulted X server on fvwm

API

I currently work for the group doing this - a very cool new feature will be launched in the next week that I am writing (stay tuned). Yes - this project has been done many times before by many people (to lesser degrees of success than this), but the thing to keep in mind is that this is realtime. It takes less than a second for the tags to be generated. All previous systems required a much larger amount of processing time. Check out www.alipr.com to try it yourself!

1 out of 15 ? impressive

How do they get less than a 50% average that you'd get by just guessing?

(yes, assuming a normal distribution of 'concepts' in the pictures, etc)

w00t!!!

Now almost 7% of my pr0n will get tagged correctly!
That's cool, the rest of it will be like opening xmas presents!

*file: 123456.jpeg>open>Aghh! Goatse!*

Hmmm...This may be neat when it gets a LITTLE more accurate, but a cool start none the less.
Kudus to the gang for getting a grip on a hard problem...erm..nevermind.

In future news...

Image recognition software is making it even easier for your kids to find porn! More at 6...

IBM was working on this years ago...

Not sure how far they got, but remember reading that IBM was working on this and had some reasonable success at object recognition in images. I'd love to be able to classify the 10k digital images I've got around. Especially if it can recognize individuals (not that it would know their names initially, but would be trainable).

Reportedly

Reportedly the researchers showed the system a picture of a Death Star, and it correctly tagged the image with 'thatsnomoon'.

The system has clearly been let crawl the web for far too long.

Google will buy it...

Unless Jupiter Media [jupitermedia.com] gets to it first.

Someone like myself would understand the hours of data-entry and database development that goes into indexing imagery. I research photo copyrights for a living.

The fact that there is a feasible, automated system that can do the work will certainly cut down the man-hours for that sort of work; at least by half.

Pity, though. I heard that Google and others had a telecommuting thing that paid people to recognize what's in a photo. Sorry to hear they'll be out of a job soon.

Workarounds.......

.........SexSurfer logs in to begin his daily search of the web to find more images to rip in an effort to increase his database of porn images, utilizing this technology, only to find that most of the images consist of naked women with political statements printed on their asses......

Seriously now, I am sure their are people out there that have already got ideas rolling around in their heads about how they can use this technology to hijack images to their advantage. Once somebody understands how the technology works it is only a matter of time before it is used for nefarious purposes, by means of "tricking" the technology. And in the process, invalidating any possible means by which the developers can realize a return on their investments.

Personally, I'd love to use such a technology(if it actually works) to sift through the plethora of "crap" images I have to search through on the web. It can be really frustrating to do a search only to find that a vast amount of the results are TOTALLY out of context simply because of the title tag attached.

One tag in 15 correct

well that really had to happen, i've just tried it and you can't really go wrong if one of the top 15 tags is 'photo' and another of the tags is 'thing'.

image spam filtering

Now all they need to do is come up with a way to recognize spam words in image text without the overhead of OCR and they can make a fortune on that alone.

Uni assignment

This reminds me of a uni assignment that i did where we matched images based on colour.

I'm not...

... usually a pedant... but you don't train a database. It was likely a neural net, but TFA is rather thin on details. Anyone got a link to their paper?

Lots of implications for Surveillance, however.

Makes it easier to process all that data generated by all those security cams.

Is there a "Big Brother" category on Slashdot, yet?

Wrong approach?

This would be cool and all, but why not focus more on letting humans do the hard work-- like if I could take a picture of a tree and then press a button and say aloud; "redwood tree", and have that tag the file.

Well I sure hope they invented it before early 80s

When I was studying textbooks on how to do this in undergrad comp sci.

So they say...

That's all fine and great that they can tell us, but why the heck couldn't they make a web-interface for it so I could try it out?

Bullshit Patents

This just your standard data mining classification system, simply applied to image data as input, with the tags being possible classifications. This is an obvious application to ANYONE in the field. Software patents suck.

confucius says...

Considering the adage that "a picture is worth a thousand words", they're going to have a lot more words to index--where the words may not follow a specific taxonomy.

And that's one of the problems: does an image define the taxonomy or taxonomy defines the image [type]?

When it's going to be on Flickr?

I quickly run out of fantasy when it comes to assign tags to my pictures: an automated mass tag finder will save hours of my precious time while uploading photos to Flickr.

Neural Nets

How can you take a neural network and train it, then patent that?
That's like patenting training a dog to fetch a stick, it's completely rediculous.

You take software capable of generalizing a neural network algorithm by feeding it pictures and associating each picture with certain tags. It then creates a generalized algorithm model based on what you fed it initially. So that when you give new input it is capable of outputting tags most similar to what you initially trained it.

So yes this software can recognize boxes, shapes, other objects, maybe scenes etc and associate them with tags... but ask them how the algorithm works under the hood =) They have no idea... a neural network is like a black box after it has been trained. You feed it input and it gives you output based on it's initial training. The inner workings are chaotic spaghetti values set on each neuron weighting and can't be deciphered.

How can you patent software that is a black box inside?

"Yes hello patent office? I have this box that manufactures microprocessors. I feed it all the materials and it outputs a shiny new processor. I am not sure of the manufacturing process internally but the output works great. I would like to patent this manufacturing process.

"Okay your patent number is 247286-"BLACK BOX"-9

The whole point of a neural network is it generalizes what you train it and can future predict any input based on that.

It's like having the invention of the first mirror and everytime someone put something different infront of it, that person called up the art gallery because they had a new painting that they wanted in their name (because depending what was in front of it you get a different reflection).

Publications

Instead of speculating, why not just read all about the algorithms?

Main publications:
http://infolab.stanford.edu/~wangz/project/imsearc h/ALIP/ACMMM06/ [stanford.edu]
http://www-db.stanford.edu/~wangz/project/imsearch /ALIP/PAMI03/ [stanford.edu]
http://www-db.stanford.edu/~wangz/project/imsearch /SIMPLIcity/TPAMI/ [stanford.edu]

unimpressing

If it worked, it'd be very useful. However, getting the top 1 tag correct 50% of the time (which is the only quantifiable claim in the article) is pretty straigthforward. For most peoples photo-albums that can be done by the following AI-program: "print 'people'"

There's a few subjects that are so common that it's more or less a given they'll be in a large fraction of the photos. Outputting "people, buildings, nature, animals, plants, city" would probably give atleast 1-2 "correct" tags for 90% of whats in peoples photoalbums.

I had a class on neural networks and their (weak) sort of "ai", one task was to build a program to separate male from female names. The best programs could manage 80% or so, which is sorta decent. Until you realize that checking against static lists of the top 100 male/female names, if it's not in the list guess female if it ends in 'a', otherwise guess randomly will get you aproximately 95%. Furthermore, the latter program runs an order of magnitude faster, is more easily debuggable, can be understood by anyone, and can trivially be "extended" to reach 99% or more, simply by extending the lists of known male/female names.

link?

So where's the download link? How can software matter if I can't get it? ;-)

Pictionary!

I now have someone to play pictionary with.

Big deal

So what were they doing, throwing a dart at a damn board? That success rate is no better then randomly applying vague words.
 
Move along to real research.

wonder how this compares

I wonder how this compares with Riya [riya.com]. At some point, there were plenty of rumours [google.com] of a possible Google purchase of Riya. Then again they were rumours.

I haven't RTFA and I don't have any experience with Riya either, so consider the above posting a waste of time (if you must).

Very important...

Does it detect breast size?

Wouldn't it be easier...

Wouldn't it be easier to pay starving children in [???] $0.01 / hour to tag images?

Re:Tag for /.

"itsatrap" is the best... but use sparingly

Re:balls

Please see my other post in this thread. THX.

Re:Just how broad is a concept?

I think I could categorise most things using less than 15 (admittedly very broad) tags. Animal, person, plant, machine, sports, vehicle, furniture, book, etc.

So which of these categories does a mushroom fit into?