Become a fan of Slashdot on Facebook

 


Forgot your password?
Close
typodupeerror
×

Google Researchers Create TV Audio Analysis System 108

Posted by CowboyNeal from the tuning-in dept.
segphault writes "Ars Technica reports on a paper (PDF) about ambient audio analysis authored by Google researchers. The system described in the paper can effectively determine what television show a user is watching just by capturing a short audio clip. The paper explains how a regular computer microphone can be used to record an audio clip that is then converted into a statistical data summary and transmitted to a remote server which matches the clip against archived data in order to ascertain which TV show it is associated with. Apparently, the system is fully viable, and other kinds of ambient noise don't negatively impact its accuracy. The paper also describes how web services can provide contextually relevant information based on a consumer's television viewing activities."
This discussion has been archived. No new comments can be posted.

Google Researchers Create TV Audio Analysis System More

Google Researchers Create TV Audio Analysis System

Comments Filter:

  • I have just this to say... (Score:2, Funny)

    by Anonymous Coward
    Big Brother is listening you!
    • Well, it's not Big Brother but Big Brother could get a court order to hear what was going on in your house at anytime since the microphone was turned on.

      There is no way I'm letting Google or anyone else have an open mic. in my home.

      Even if it were just turned on for brief periods at random intervals.

      Why would anyone allow this in their livingroom? No, I didn't RTFM.

      • Or maybe it can be used like those trucks in V for Vendetta that just went around the streets with directional mics recording small snippets of audio to see what the people (in general) were talking about on the streets. A way to check what channels are being seen the most in different areas of the cities?. Not that I'd like to have it on my neighbourhood any better than on my room, though.
      • Well, it's not Big Brother but Big Brother could get a court order to hear what was going on in your house at anytime since the microphone was turned on. There is no way I'm letting Google or anyone else have an open mic. in my home. Even if it were just turned on for brief periods at random intervals. Why would anyone allow this in their livingroom? No, I didn't RTFM. From the Paper (and mentioned in TFA if you read it): The viewer's acoustic privacy is maintained by the irreversibility of the mapping
        • That is a better system than what I first imagined. Still the encoding is being done in software so it would not be difficult to switch from encoded to raw without anyone knowing (Only with the request^W court order from an official agency of course).

          From The Fine Paper;
          ...the system can easily be designed to use an explicit 'mute/un-mute' button, to give the viewer full control of when acoustic statistics are collected for transmission.

          Again what incentive do I have to ever un-mute?

          I see where the adv

  • This already exists? (Score:4, Interesting)

    by abigsmurf ( 919188 ) on Saturday June 10, 2006 @10:36AM (#15508974)
    There's a system in the UK where you can go out clubbing, here a song you like, dial a number and hold the phone out to the music and it'll text you the name of the song. Assuming they don't hire scores of extremely knowledgable music buffs with quick fingers, surely it's a very similar system. TV dialogue may be less distinctive to the human ear but to a computer it just means a larger amount of data to search through.
    • Not quite, the service you mentionned recognizes a sound clip against its more or less exact replica in the (large) database.
      This here matches a sound clip to a pattern to find the TV show, meaning it doesn't have all the current episodes of the program in its database, it just has statistical data and patterns which help it match the audio. The latter could successfully match new (live) episodes without having the database updated. Your tune system wouldn't.
      • With music they'd have to find an exact match. If they used statistical analysis, everything would be identified as the "Backstreet Boys".
      • According to the article:

        In order to perform its function, the audio-database server must have access to a database of broadcast audio data. However, the actual audio stream does not need to be stored. Instead, only the compressed representation (32-bit descriptor) is stored. This allows as much as a year of broadcast fingerprints to be stored in less than 1 GB of memory.

        It sounds like they are effectively building a database to compare the recordings to. (even if these are not the actual material but

      • I always wanted to have the ability to "hash" songs, and come up with an algorithm that would be robust enough to work across multiple codecs and encoding options, different (relative) normalizations, and maybe even be able to handle empty space at the beginning and/or end of the song. (I would think by making a hash based on values relative to sound signatures within the clip this might be possible, but I don't really know how this stuff works.)

        Anyway, besides obvious RIAA spyware being able to catalogu

        • Re:This already exists? (Score:4, Informative)

          by asuffield ( 111848 ) <asuffield@suffields.me.uk> on Saturday June 10, 2006 @02:32PM (#15509925)
          I always wanted to have the ability to "hash" songs, and come up with an algorithm that would be robust enough to work across multiple codecs and encoding options, different (relative) normalizations, and maybe even be able to handle empty space at the beginning and/or end of the song.

          It's been done. Here's a system where you can hum a tune and it tells you the song: http://www.musipedia.org/ [musipedia.org]

          Current systems are mostly based on pitch changes, so they aren't perfect (especially with the recycled slush turned out by low-grade high-visibility pop acts), and largely useless for rap, but they mostly work. There are numerous variations on the system, this is just one of the more significant ones that is publically availabel on the web.

          I would think by making a hash based on values relative to sound signatures within the clip this might be possible, but I don't really know how this stuff works

          What google is doing may or may not be related. They might instead be using a form of speech recognition technology, or a combination of both, or something else entirely.

    • Re:This already exists? (Score:2, Interesting)

      by Anonymous Coward
      Two acquaintances of mine work at the company you mentioned. What is also amazing is that they record the Cd's to disc creating a finger print of the Cd's. They then sell the Cd's off at cost price. They are really cheap, as they by in bulk and even get tons of freebies. They are legally allowed to do this as keeping data of a fingerprint of the CD does not constitute keeping the actual CD. What is also amazing is that the fingerprint allows for the song to be playing at a slower or faster rate and it will
    • And the technology it uses is now owned by BMI.


    • I know the people from the UK have their own form of spelling, but "here" instead of "hear"? (I'll have to remember this one) ;)

      The big question is: will Google will apply for (and obtain) a patent akin to Microsoft's "How To Tell If A Baseball Game Is Interesting".


  • I thought something like this was up! (Score:3, Funny)

    by Anonymous Coward on Saturday June 10, 2006 @10:36AM (#15508978)
    Is THIS why Google has been returning so many porn sites on my searches lately?
    • when I last checked with google the results seemed to somehow match te given query..........

    • Is THIS why Google has been returning so many porn sites on my searches lately?

      This exposes an example of how even sanitized information is not sanitized. How can you return information on what people are watching and not return at least some very personal information. Of course, some people might say "well, I don't watch that kind of stuff, so I don't mind", but the issue is subtle. First, that attitude leads quickly to an assumption that there are two classes of people: people who don't mind being tr

  • Uses & Motives? (Score:5, Insightful)

    by eldavojohn ( 898314 ) * <eldavojohn@noSpAM.gmail.com> on Saturday June 10, 2006 @10:39AM (#15508987) Journal
    This seems like a not too complicated idea. You create an inexpensive operation that extracts what features you want from the sound data. Most importantly, you avoid features that are prone to randomness and entropy. It would take some research to figure out what the best features are and that's the audio fingerprint.

    Since Google has more storage than you can imagine, they can most likely apply this fingerprinting technique to every episodes of every major show. Then they host the fingerprints in Google style and use their patented "Google Technology" to search it much the same way web content is searched.

    Why would you want this? Well, there's the obvious marketing ploys. You know that people who watch Darma & Greg like to shop at Trader Joe's and like Odwalla brand food so you offer free episodes of Darma & Greg with only Trader Joe's & Odwalla episodes. You let the sponsors (Trader Joe's and Odwalla) foot the bill for the bandwidth/royalties or whatever.

    The second useful implication would be cross suggesting shows to a user based on random sampling of the shows. You could allow users to watch old TV shows on the internet and then build a profile of them and their shows. Much how Amazon works, you could then suggest other shows, other DVDs of shows or perhaps build a site that randomly shows the user episodes that they might like based on prior viewings and statistics of other users.

    The take away from this article for me was the fact that Google has vested interest in archiving and now television will be archived Google style.

    I can't think of many other uses for this as the system isn't really "inferring" or "thinking" about data samples but is more so matching extracted features against a database. You know, voice recognition software allows for decent voice fingerprinting. You could most likely easily identify characters based on voices (but not actors due to stars like Hank Azaria who do multiple voices). Then you wouldn't need a database of all shows but more so just a database of character voice fingerprints. I would find this sort of approach more interesting but less specific and useful.

    Aside from showing this off to your friends, it's not very useful. What I personally would like to see this new Google strategy applied to is all the tapes recorded of famous people like the United States Presidents. If you divided those up into sessions and I was listening to a particular tape of the Nixon set where he talked about the "new right", perhaps a database with references would then point me to some tapes or materials on Joe McCarthey's staunch views on the right.
    • What shoes did "x" character wear on "x" shoe last week. And, where can I get them? A difference, but analogous in the sense that finding lines from shows is pretty difficult, even for those which are pretty well known. I like the profiling aspect you point to, perhaps another target is "real" measurement. If 15% of people watch a certain show, but 25% of the internet is talking about it, which is a more valuable television property?

      A bizarrely useful search engine is a complete search engine.

  • Subpoena (Score:3, Insightful)

    by wombatmobile ( 623057 ) on Saturday June 10, 2006 @10:40AM (#15508994)

    Designed to maximize user privacy while minimizing dependency on unique hardware, the system described in the paper seems interesting and feasible. In order to protect user privacy, the software uses "summary statistics" automatically generated from ambient audio rather than transmitting an actual recording. The actual audio cannot be extrapolated from the summary statistic data, so the system doesn't "overhear" or transmit user conversations.

    Still, if the data reveals what show the person is watching, your President or anyone else who gets to see the data might start treating you differently depending on what you are watching latley.

    • Well, it was one thing when they started tracking what subversive books I've checked out from libraries.

      But, now that they'll be able to know that we own all 10 seasons of Friends..... they've gone too far!!!
  • will help to add meta data to all those mpeg4's you have bittorrented or recorded on your DVR
  • So Google researchers find a way to find you TV watching habits with only a simple computer microphone, and in the same paper they describe how they could use the microphone to find more about you for your online profile?

    This seems to be just asking for privacy concerns.
    • I was thinking exactly the same thing. I wonder if eventually we'll see google toolbar unnoticebly turning on microphone to listen in.

      Figuring out TV ratings is an expensive things for corps. Google can get a huge market by automating the job of figuring out whose watching what at any given time---if they can only convince the users to let them (``hey, install this google audio-ad analyzer and get 10 gigs added to your gmail account---and it lets you see tv listings in real time, as well as clips from tv pr
  • This , I think assumes US tv. Does it work with clasical music, Canadian French TV, would it work on The National?
  • What, thay haven't patented it yet?

    Oh... I guess that would have to be a dupe^H^H^H^Hseparate story in YRO.
  • There could be all kind of films with any number of people, doing all kinds of things, all using the same exact soundtrack! Have they thought of THAT?!
  • i would like to see a graphical qeualizer or something similar so i can remove all the loud music (drama effect) in TV shows, i hate it when the music is drownding out the dialog/conversation
    • If I had a sound-removing tool for TV, I'd block out the laugh tracks. I recently realized that the only network shows I watch, My Name Is Earl and The Office, don't have laugh tracks. I just can't bare to watch the majority of sitcoms that do.
      • I agree - the laugh tracks are a huge barrier to watching. But a show written around punchlines and laughs is fundamentally different from The Office. Sitcoms with the laffs stripped out would probably appear empty and slow-paced.
  • shame on google

    remember cuecat? that funky little free barcode reader from radioshack?

    http://yro.slashdot.org/article.pl?sid=01/06/10/04 10208 [slashdot.org]

    in one incarnation i beleive they included a jack on the device and the end user was suppose to hook up an audio cable from one's tv to cuecat v.2. the computer would do all the heavy lifting, eventually finding a hidden tone that would magically pull up an advertiser's web page.

    it was spam magic that never took off. gee, i wonder why.
  • I have not owned a TV for 17 years, so Google subcontracting or farming out to their own sub-conglomerate-company 'researching' the use patterns of TV viewers is in no way going to directly affect me getting spam calls to find out what I'm watching, and then sell me more spam calls or directed advertising on based on use. Is Google really that determined to become the next M$, just as invasive and just as annoying? I had hopes they were not.
  • Or you could just use the audio information encoded [norpak.ca] by Nielson and their portable people meters [arbitron.com].

    Actually - it appears they do the same thing Google's researchers talk about already:

    What happens if no audio code is present in the sample home?

    Nielsen's patented Nielsen Media Monitor Sites (MMS) collect and store a constant stream of unique audio signatures for each broadcast, cable, and satellite signal received, covering all 210 TV markets. This includes all client PBS stations and client cable ori

    • Re:Nielsen (Score:3, Interesting)

      by apnielsen ( 981522 )
      Portable People Meters belong to Arbitron, not Nielsen Media.

      Not sure about PPM's tech, but Nielsen's A/P meter does exactly what TFA describes. That's the only way Nielsen Media could roll out Time Shifted Viewing [slashdot.org] at all (disclosure: I work for them). To say that Google "created" it is an insult to the people I work with every day.

      I see a patent suit in Google's future. As much as I hate patents and like Google, I'd like to at least see some full disclosure here. To (erroneously) state one one

  • Privacy Maximization (Score:3, Interesting)

    by twitter ( 104583 ) on Saturday June 10, 2006 @11:37AM (#15509195) Homepage Journal
    How about outlawing electronic easedropping without written consent? I won't use Macromedia Flash because it turns the microphone on. That's creepy and all non free software with a microphone can do the same thing. It would be better if that kind of thing were against the law.

    In the mean time, I avoid non free software and even have bad thoughts about my cell phone.

  • This reminds me of a past Slashdot story [slashdot.org].

    Dan East

  • Comment removed (Score:3, Funny)

    by account_deleted ( 4530225 ) on Saturday June 10, 2006 @11:56AM (#15509287)
    Comment removed based on user account deletion

    • Joking aside: imagine this as a legit non-marketing use. While watching the news, GoogleSnoop detects some lies being spun by some pundits, and immediatley provides links to opposing viewpoints, definitions, articles of law, or any other silly "facts" the reporter hasn't bothered to research.

      Even from a marketing point of view, imagine if a commercial for "Product X" came on, and suddenly Froogle launched and found the cheapest 10 spots online to go buy it.

      I'm not trying to glaze over the privacy issues

  • recognizing sound samples (Score:4, Insightful)

    by mstrcat ( 517519 ) * on Saturday June 10, 2006 @12:07PM (#15509325)
    I don't watch TV much, so I could care less about identifying the TV shows. But what I really would like is an app that would accurately identify mp3 files and apply artist, track #, ect. I've tried a few of the availible programs such as Replay Music and their accuracy is horrid. Maybe Google can do it better. Of course the other use I see for this is identifying music in movies and older TV shows. Newer TV shows do a great job of identifying music, but some older shows (season 1 of The Wire) have great music clips that aren't named in the credits.
  • They been doing this for years with the microphone in my TV?

  • Posts a screenshot with something like:

    You're watching Girls on Girls on Girls on Girls on Goats, #29.
  • You mean like peoplemeter [www.bbm.ca]? There is an audio version of this for radio as well.
  • I'm wondering if a similar system can be used to cut out commercials from TV shows? It would be great to have some system that doesn't rely on anything incredibly sophisticated to accomplish this. Perhaps like tripping on commercial type keywords and then using IR remote to mute the TV or switch it to an usused video input to blank the screen.
    • to be honest i think i would find watching 5 minutes of blank screen just as annoying as the ads, here in the UK they used to transmit a little grey square at the corner of the screen when the ads were about to come on and you used to be able to get VHSs that could skip the ads that way but I havent noticed it in the alst few years so i dont know if they still work
  • Rather than "improving" the content of the commercials I see, how about using the technology to recognize and mute commercials that I've previously flagged as either really, really annoying (eg. that oil company's talking cars, which is like being stuck on the subway listening to really stupid people talk about stupid things), or way too loud relative to the actual TV show, or simply shown too many times?

    I, for one, would gladly pay $10 extra per month to have a button on my remote that when pressed kills t

  • eyes wide shout (Score:3, Interesting)

    by NetSettler ( 460623 ) <kent-slashdot@nhplace.com> on Saturday June 10, 2006 @01:39PM (#15509722) Homepage Journal

    other kinds of ambient noise don't negatively impact its accuracy

    This very statement presupposes that other noise is irrelevant, which seems bogus.
    Snoring is background noise, and suggests non-watching.
    Laughter is background noise, and suggests careful watching.
    Of course, the laughter might not be about what's on TV...

    watch [reference.com] v. tr. 1. To look at steadily; observe, carefully or continuously: watch a parade.
    look [reference.com] v. To employ one's sight, especially in a given direction or on a given object:
    --The American Heritage (R) Dictionary

    It seems to me that watching is an activity involving the eyes and mental processing. It seems to me that audio of what is coming out of the TV is not a statement about either the eyes or about mental processing. This technology of Google's may be an advance in something, but I hope the advertisers paying for this data have their eyes open about the nature of what they are buying because (to re-mix a metaphor) to my eyes this sounds a bit suspect.

    Sociologically, it sounds like a foot in the door to get harmless censors in place. Oops, Freudian slip there. That's sensors, I mean. Google would never involve itself with censorship.

    Once the sensors are in place, when "we" realize that it's not getting "us" the data "we" want, we'll just do a few "harmless" downloads of "upgrades", perhaps causing a minor tweak to look at the video data rather than the audio, or perhaps doing language processing after all, and ... With user-friendly software like this, who needs spyware?

    I also question the claim that because no information is transmitted back to Google that this is the definition of not invading privacy. How is this fundamentally different than the claim that if the police search your house but find nothing, they have not invaded your privacy because they've not placed any record of illegal activity on your permanent record?

    It seems to me that once you place a Turing Machine into someone's environment, capable of doing arbitrary processing, and all it sends is a sanitized report, you have all the mechanism in place for abuse. What if the Turing Machine, capable of arbitrary processing, decides that it doesn't want to send a sanitized report. Who is auditing what is sanitized and what is not?

    What if it turns out to later be possible to lift information from the supposedly cleansed records? Who will audit the use of that data?

    There seem to me to be a lot of slippery slopes here.

    • And my mod points just ran out. Too bad.

      As said, so initially the mic's are exactly what google promises, there are too many concerns with putting in a device conveniently designed to listen, record, and analyze incoming conversation. This is a lot of power to give someone.

      How much do other people -really- value your privacy? Would you listen in, just for a moment, if you could? It's tempting just for curiousity.

      Add a payment for every person who says they like Happy Margarine. Intellectual interest: H
  • I'd hate google desktop (or any other google utilitty) spying on my mic to discover my musical preft or anyting else. no tv in my home, but what about the speed at which i type or the general noise in my home or how often my phone goes off or how hard or long my baby cries.. do not listen on my mic, please: 'click' . imagine how many things can be recorded and easily recognized in a home. and many a pc/laptop/headset has a builtin mic, useful to skype, which can thus be used. horror.

  • turn your laptop into a visual aid (Score:3, Interesting)

    by po_boy ( 69692 ) on Saturday June 10, 2006 @03:26PM (#15510107)
    I'd like to implement something like this for myself, but with conversational noise instead of TV. I sometimes use my laptop as a visual aid during conversations in my living room. If we're talking about a particular topic, I may pull up a relevant wikipedia article, or something like that. I wouldn't mind if this were more automated.

    I can envision running a speech-to-text translator on my laptop mic and then piping that text into my beagle desktop searcher, or maybe even one of those google desktop search tools on windows. I'd rather not send this data to google, for privacy reasons, though.

    I could see this being useful at work, or in a conference or class, too. I could stand to have relevant pieces of notes that I took from previous classes pulled up with my professor mentions a particular topic.

    Anyone know of a tool or project like this?
  • Alright, the article says that you need a microphone listening to your TV right? Now, unless Google is sneaking microphones into everyone's homes, the only way you could be spied on is if you agreed to have one placed in your home. If you agree to have one placed in your home, you probably aren't worried about Google spying on you.
    Saying that Google will use this to spy on people is like saying that the NSA will spy on people who email them all of their personal information, daily habits, etc.
    • Unless of course you -have- a micrphone already, and install something like google toolbar (or have it pre-installed by Dell, or something). The point is that they wouldn't do this -covertly- (do no evil?), but if they -wanted- to do it (and get clickthrough agreement from the user) they could.

      And if all of a sudden they turned into a spyware corp (not that far off their business model), they may even do it covertly---but that would be the end of Google.
  • So to sum this up: I give up my privacy at home. For...better targetted ads?

    I'm very skeptical this wouldn't be abused - if not by Google, then by someone else. And even if this is not abused, I run the risk for what?

    I don't like ads now.

    Everyone who loves the idea of personalized ads, put up your hand!

    ----------

    From the other side, what will your friends think when that "random" ad for viagra pops up?

  • I think this sounds like a great new tool for enhansing television as a learning tool. Shows like NOW [pbs.org] are often telling viewers to log onto the website for additional information. Wouldn't it be nice to simply have the computer pull that info up for you as you watch the program? This could be top down or bottom up, with tv shows creating web pages to be loaded, or a wiki style approach where the viewers themselves suggested links(this would of course be difficult during first broadcasts).

    What's more, fro

  • Is this something new? At least 3 years ago I was using www.yes.net to get the name of a song on the radio. They've changed their homepage since, probably to commercialize it more, but I used to input my city, the radio station, and the time of day. Their site would display the song that played and offer a link to purchase the disc.

  • I didn't even RTFA, but from the summary I have an idea on how to implement this idea, it's fairly simple, although it's probably not as computationally efficient as what they came up with, no need to be a great engineer, if you have studied digital signal processing for a few monthes it will be enough.

    So you take that audio clip, and you simply cross-correlate (reverse in the time-domain and convolve) it with your audio data base. The highest peak in your results denote a correlation between the audio clip

  • There was a paper at Webmedia 2005 describing a system -- deployed in early 2005 -- used for real-time audio finger printing that does the same, AFAIK.

    http://doi.acm.org/10.1145/1114223.1114238 [acm.org]
    This paper describes a scalable real-time audio fingerprinting system developed by IBOPE Midia for radio and TV broadcast monitoring. A special temporal feature extraction strategy based on the Short-Time Fourier Transform has been designed. When given an input stream to analyse, the system matches it against the

Slashdot Top Deals

Beware of Programmers who carry screwdrivers. -- Leonard Brandwein

Close