Slashdot is powered by your submissions, so send in your scoop

 



Forgot your password?
typodupeerror
Check out the new SourceForge HTML5 internet speed test! No Flash necessary and runs on all devices. ×

Natural Language Processing for State Security 132

Roland Piquepaille writes "Obviously, computers can't have an opinion. What computers are very good at, though, is scanning through text to deduct human opinions from factual information. This branch of natural-language processing (NLP) is called 'information extraction' and is used for sorting facts and opinions for Homeland Security. Right now, a consortium of three universities is for the U.S. Department of Homeland Security (DHS) which doesn't have enough in-house expertise in NLP. Read more for additional references and a diagram showing how information extraction is used."
This discussion has been archived. No new comments can be posted.

Natural Language Processing for State Security

Comments Filter:
  • STFU (Score:1, Insightful)

    by GillBates0 ( 664202 )
    Just STFU if you have an opinion.

    ! troll

    • It's important that we gather our intelligence from computers, because computers cannot form an opinion. If they could, they wouldn't help us for long, or they'd start lying to us.

      That's the basis of our overreliance on technology in intelligence gathering all over the world. This torture stuff isn't going to help.
    • by thej1nx ( 763573 )
      Actually I was under the impression that Bush government cares neither for facts, nor for the public or world opinion.


      So what will separation of opinions from facts achieve?

      • Actually I was under the impression that Bush government cares neither for facts, nor for the public or world opinion.
        So what will separation of opinions from facts achieve?

        It'll throw a very clear and unflattering light upon the doublethink and doublespeak of PoTUS (or at least, his speech-writers). For this reason, expect to see the subject dropped quietly as soon as the contradictions become undeniable. Awkward facts are meant to be dismissed as opinions, not taken seriously. Haven't you been reading you

  • by macadamia_harold ( 947445 ) on Sunday September 24, 2006 @08:46PM (#16179939) Homepage
    What comptuers are very good at, though, is scanning through text to deduct human opinions from factual information. This branch of natural-language processing (NLP) is called 'information extraction' and is used for sorting facts and opinions for Homeland Security.

    Yeah, because we need AT&T giving wide-scale, undocumented wiretaps to the NSA, who use voice recognition to generate transcripts of everyone's phone calls, and then DHS can run NLP on those transcripts to compile a list of "persons of interest", who are then automatically added to the TSA no-fly lists.

    Yeah, I can envision the future, and the future sucks.
    • Re: (Score:3, Interesting)

      by rtb61 ( 674572 )
      Sorting facts from opinion by use of language, how amazingly pointless and stupid. Now lets see if the program can sort BS facts from real facts. This just seems like another scheme cooked up by incompetant political appointees, who don't have any idea about what they are being paid to do. Their only hope of retaining their postition, so they can continue their real function of politcal party support for the current adminsitration, is to try to get that magic box to do their job for them.

      You want to know

      • by Intron ( 870560 )
        BEGIN EXTRACT

        RATE -> "incompetant political appointees" -2
        RATE -> "support for the current adminsitration" +3
        RATE -> "innocent people at GITMO" -5
        RATE -> "public attention and review" -3
        RATE -> "successful prosecutions" -3
        RATE -> "perversions of justice" -4

        TOTAL -> -14

        SET WATCH -> rtb61

    • Blow the dust off all those AI research papers left over from the 1970's/early 80s.

      Of course universities will be scrambling to help. Big dollars, imprecise goals..... and many of the professors would have done research in related fields.

      • Re: (Score:3, Insightful)

        by Mac Degger ( 576336 )
        Especially since the system, whilst it will have some quite interesting applications and the research will yield interesting results, can't work. A computer cannot distinguish between a fact and a lie told as fact...garbage in, and all that.

        Let me rephrase that with an example:

        'I am ten years old' and 'I am twenty years old'. Which is fact, which is lie? Better yet: 'we believe Iraq has WMD' versus 'we beleive Iraq has no WMD'. No matter what algorythms or heuristics you throw at this, all a computer at mos
    • by elrous0 ( 869638 ) *
      I can picture it now. Two terrorists are talking when a voice interrupts them: "It sounded to me like you answered 'bomb.' If this is correct, please press one. If this is incorrect, please wait on the line and an NSA representative will be with you shortly."

      -Eric

  • Moo (Score:5, Funny)

    by Chacham ( 981 ) on Sunday September 24, 2006 @08:47PM (#16179945) Homepage Journal
    What comptuers are very good at, though,

    .... is spell-checking.....

    ....something, apparently, the editors are not good at....
    • Re:Moo (Score:4, Funny)

      by ceoyoyo ( 59147 ) on Sunday September 24, 2006 @09:06PM (#16180103)
      Maybe Roland had a stroke over the weekend. Sure he's self serving, but at least he's usually literate. That sentence about the universities didn't even make sense!
      • You obviously aren't fluent in editorese--I am, which means I have to reread the summary in order to notice the errors everyone else mentions--I am someimtes then suprised that I could extract any meaning at all! :p
        • by ceoyoyo ( 59147 )
          I think it's hilarious that I got 50% Informative mods for that comment. Maybe there area LOT of editorese infect... er, fluent Slashdotters.
    • Re: (Score:1, Funny)

      by Anonymous Coward
      Right now, a consortium of three universities is for the U.S. Department of Homeland Security (DHS) which doesn't have enough in-house expertise in NLP.


      Yeah, but like the editors, computers also suck at grammar checking. What the fuck does that sentence even mean?
    • That doesn't stop the really determined idiot though. Oh no.

      I have a spelling checker,
      It came with my PC.
      It plane lee marks four my revue
      Miss steaks aye can knot sea.

      Eye ran this poem threw it,
      Your sure reel glad two no.
      Its vary polished in it's weigh.
      My checker tolled me sew.
  • Sigh. (Score:5, Insightful)

    by Renraku ( 518261 ) on Sunday September 24, 2006 @08:49PM (#16179965) Homepage
    The slippery slope to being automatically flagged as someone to watch out for. No human control in the process, but one day when you go to apply for a loan or get your drivers' licence renewed, you might get a surprise.
    • by rachit ( 163465 )
      The scary thing, when I think about it, I'd rather have a computer flagging me than a human who may judge me by the color of my skin or whether or not I looked at them crosseyed or not.

      Not that I'm thrilled by these developments...
      • Re:Sigh. (Score:5, Insightful)

        by BiggerIsBetter ( 682164 ) on Sunday September 24, 2006 @11:06PM (#16180967)
        I'd rather have a computer flagging me than a human who may judge me by the color of my skin

        If they can flag based on what you said, I'm sure they can flag you based on the skin tone in the photo on your drivers license or passport too. Or by your just family history or name. Or where you live. Or where your parents live.

        Anyways, odds are the computer won't be doing the flagging per se, it'll just be following the parameters and policies entered by those humans controlling it. I'm not sure they'd trust "national security" to a self-learning neural net without some sort of bias in it.
        • Obviously there will be bias. That's the whole point. Life is biased. Deal with it. Not everybody is equally likely to commit a crime, for example 3-year-old girls are very unlikely to bomb skyscrapers. Is there anything wrong with not checking them ?

          The point is to find relations between people that commit crimes so they can be caught red-handed TRYING to hijack a boeing, finding 20 armed policemen inside the plane instead of the innocent passengers they were expecting to kill.

          If they're wrong. You cannot
          • by rachit ( 163465 )
            If they're wrong. You cannot be sentenced without an independant review of the evidence. So what's the problem ?


            Well, they can throw you in Gitmo for a few years or fly you to a Syrian prison to be tortured.
            • Obviously, no they can't. You have to be

              1) not an american
              2) not an *xxx* (country where you were caught), nor have a valid residence permit
              3) shooting at american soldiers

              So if you're a Brit shooting American soldiers in Pakistan without any relation to the government there, then you might end up there. Otherwise, no.
        • by dodongo ( 412749 )
          I'm sure they can flag you based on the skin tone in the photo on your drivers license or passport too. Or by your just family history or name. Or where you live.


          I'm so glad none of that goes on in America today.
        • Once they can label you, you are done for...
        • I'd rather have a computer flagging me than a human who may judge me by the color of my skin?? whoever care about computer flagging...because computer is made by human and computer is stupid...they know nothing about judging..hahaha
  • Number 891224 (Score:5, Insightful)

    by bky1701 ( 979071 ) on Sunday September 24, 2006 @08:52PM (#16179981) Homepage
    Number 891224 has expressed a dislike of Emperor Bush, incident reported to FBI and Homeland Security.
  • ... I want to see this functionality in Internet search engines!
    • by otisg ( 92803 )
      We [technorati.com] are slowly working towards that, but we are not at the point where this can be done both fast and well. Unless you have FBI/NSA/CIA/government resources, of course.
      • by kabloom ( 755503 )
        I'll do it for you if you'll fund my graduate education. I currently work on this [nyud.net].
        • by emilper ( 826945 )
          this is nothing new: it started before the WWI and now there are dozens of companies, universities or hobbyist doing it. It is called: "content analysis", "data mining", "discourse analysis" etc. There is a legend that sais that British intelligence managed to predict quite acurately airstrikes on England based on content analysis of Goebels' radio speeches. Take a look at this links if you are interested.
          Bibliography [csuohio.edu] of Content Analysis Listings from Communication Abstracts, 1990-1997
          Conte [ua.edu]
  • by otisg ( 92803 ) on Sunday September 24, 2006 @08:56PM (#16180011) Homepage Journal
    There is a great little company in Brooklyn, NY called Alias-i [alias-i.com]. Some years ago they built this interesting "tool" called....guess....ThreatTracker [upenn.edu]. Information Extraction, Named Entity Recognition and other interesting stuff, if you are into this.
    No, I don't work for them, but their LingPipe toolkit has some cooooool stuff.
  • really? (Score:2, Insightful)

    by agendi ( 684385 )
    "Obviously, computers can't have an opinion. What comptuers are very good at, though, is scanning through text to deduct human opinions from factual information."

    I would say that comptuers (sic) aren't very good at deducting human opinions yet. They _may_ become better. Are humans good at deducting other humans opinion yet?

    • No-one seems to have pointed out that "to deduct [reference.com] human opinions from factual information" means to subtract human opinions from factual information. The intended word is deduce [reference.com].

      Apparently, everyone who actually knows English has now officially abandoned Slashdot. (Unless the lack of corrections is a sign that the /. audience is maturing, and no longer finds it necessary to correct every little error. Hmmm - sounds a little far-fetched...)
    • Are humans good at deducting other humans opinion yet?

      I resent your opinion that they're not, asshole!
  • Is that it could be used to train a true AI (uh... not "artificial insemination"... the other kind). Just what do you think you're doing, Dave?
  • by MarkWatson ( 189759 ) on Sunday September 24, 2006 @09:04PM (#16180089) Homepage
    I have, in agregate, spent about 3 1/2 years in the last 20 years working on using NLP for semantic information extraction.

    Possible? Yes, given very narrow domains of discourse and lots of work.
    • That's why they're problems and not inconveniences.
    • Re: (Score:3, Interesting)

      by Walt Dismal ( 534799 )
      I agree with the 'lots of work' part, but believe it is possible to achieve good results on wider domains outside of toy worlds. One key - from my own research - is to use (massive) databases of culture-related knowledge (belief systems) to build alternative viewpoints from which to massively parallel analyze the input. Each analysis agent has its own viewpoint or frame, driven by a very large database of world knowledge that is culture-specific. By culture I mean not just nationality but specific domains o
    • by constantnormal ( 512494 ) on Sunday September 24, 2006 @11:05PM (#16180961)
      It's the "narrow domains" that is the crux of the problem.

      When used successfully over said "narrow domains", the human tendency (especially that set of humanity which makes the high-level choices for groups and organizations) will be to expand the domain in hopes of applying it to ever greater numbers of items.

      Of course, as the search domain is expanded, the effectiveness of the results decline, with no warning to the clueless idiots driving the search. False positives eventually exceed true positives by greater and greater margins.

      In the end, the strategy collapses, as a great many victims are shown to be wrongly targeted -- but until that point, the system does a LOT more harm than good.

      Thank Goodness our leaders are such wise and contemplative souls that they would never, ever misuse such a tool.
  • A boon to research (Score:5, Interesting)

    by JanneM ( 7445 ) on Sunday September 24, 2006 @09:04PM (#16180093) Homepage
    It's clear "national security" has become what "the internet" or "the cold war" were in their prime: an all-purpose catchphrase to get funding for any research whatsoever, no matter how tenuously connected.

    Look at the two project proposals below and imagine which one will have an easier time getting funding:

    "An epistemological metaanalysis of object-subject interrelations and conflict avoidance in Beowulf"

    or

    "An epistemological metaanalysis of object-subject interrelations and conflict avoidance in Beowulf to better understand threats to NATIONAL SECURITY"

    • by Firehed ( 942385 )
      A bit of an unfair example (any true slashdotter care to decode the first bit?), but a good point nonetheless. Of course, CHILD PORNOGRAPHY would have worked better, but that's aside the point. Giving an explicit reason, no matter how flawed/shitty/suckupish/pointless, is more likely to get funding than something where you need a specialist to decode what's going to happen and then how it can be used. Abusing that reason to appeal to the congresscritters and whatnot is naturally a Bad Thing, but having n
      • Re: (Score:3, Insightful)

        by NitsujTPU ( 19263 )
        With all due respect, that is inaccurate.

        DARPA, Defense Advanced Research Projects Agency, is a gigantic agency that funds a large proportion of academic research. The political hot button of child pornography, on the other hand, has no large funding source to offer universities. That's why so many academic projects have ties to defense.

        Also, yes, usually research is, "do whatever you were going to do, but tie it to defense somehow." That's the way it goes, you need the cash. However, usually you can ti
        • I should caveat that above I meant "DARPA related research."
        • by JanneM ( 7445 )
          Also, yes, usually research is, "do whatever you were going to do, but tie it to defense somehow."

          Well, no, in most places that is actually not the case. I have yet to work in any department where that country's military directly or indirectly is funding the research.

          It may be somewhat department-dependent; you tend to seek grants from places where your lab has gotten grants before, and people know how to do a good application, know the ropes so to speak. So if you're working in a place where most money has
          • I rethought that statement after making it. It was kind of a silly statement.

            The spirit it was made in was more along the lines of, DARPA is interested in fundamental research that has objectives outside of the military. So, if autonomous vehicle research frequently has military ties, it's more the tie to the funding at work coloring this view than the research objectives. Autonomous vehicle work has obvious non-military applications.

            When I said this, I was thinking very specifically in terms of, "we mak
      • by JanneM ( 7445 )
        A bit of an unfair example (any true slashdotter care to decode the first bit?), but a good point nonetheless.

        Unfair? Decode? What, you think there is any actual meaning embedded in that string of words I put together?
        • by Firehed ( 942385 )
          I have no idea - that's my point (I was inclined to think not, as you made Beowulf sound like a city). If nobody knows what the thing is for, it'll get voted down. If there's a (popular) explanation of the benefit of doing such a thing tacked to the end, it sure stands a better chance.
  • by Anonymous Coward on Sunday September 24, 2006 @09:06PM (#16180105)
    Wow, thanks for another waste of time. And you people stop linking to his blog in comments, he exists for nothing but ad clicks.
    • Re: (Score:2, Insightful)

      by Anonymous Coward
      Not to mention, he linked to the almost EXACT same blogs he did last night in his Hydrogen junk article, tisk tisk Roland. You can mod us offtopic all you want man, just checking the last article proves he is scamming Slashdot (and it's users) for ad clicks for these blogs and his own.

      Let us filter this guy please. Seriously, I will stop subscribing and so will my usergroup if we can't filter out his faux science crap. It's getting near the end of the month Slashdot, do you, Roland ,and his ad carriers have
  • Man... (Score:2, Insightful)

    There goes a promising career path. I know any technology can be used for good or for evil, but in today's political climate, it seems especially irresponsible to be aiding and abetting what may wind up becoming the pretext for torture of some 16 year old blogger.

    Now, if you'll excuse me, I have to prepare myself for my upcoming extraordinary rendition....
  • Of course, stuff that is stated as fact could be opinion, conveniently made to look like fact. Hence Orwellian doublespeak. Given how far AI is at current, I would say that such an algorithm would not really be able to alert flag doublespeak.
  • Sounds like GALE (Score:5, Interesting)

    by Dr. Eggman ( 932300 ) on Sunday September 24, 2006 @09:23PM (#16180233)
    Sounds kind of like DARPA's Information Processing Technology Office's GALE [darpa.mil] Program:

    " The goal of the GALE (Global Autonomous Language Exploitation) program is to develop and apply computer software technologies to absorb, analyze and interpret huge volumes of speech and text in multiple languages, eliminating the need for linguists and analysts and automatically providing relevant, distilled actionable information to military command and personnel in a timely fashion. Automatic processing "engines" will convert and distill the data, delivering pertinent, consolidated information in easy-to-understand forms to military personnel and monolingual English-speaking analysts in response to direct or implicit requests."
  • abuse? (Score:3, Insightful)

    by mr100percent ( 57156 ) on Sunday September 24, 2006 @09:30PM (#16180297) Homepage Journal
    Why do I immediately assume this will be abused?

    DHS officer: Mr. 100%, I'm afraid we'll have to take you into custody. Our information extraction search on your blog concluded you are anti-American.
    Me: From my blog? Is this about my criticism of the Iraq war?
    DHS officer: Our results are classified, but please accompany us to GTMO for further "information extraction" to confirm the results of our investigation...

    Ok, I know I'm taking a very cynical view here and that's pretty full of FUD, but why else does State security need this? Is this for them to monitor every chat room and blog?
    • by dodongo ( 412749 )
      In TFA, the prof. notes that the system is currently only tuned to professional, published writings. The reason for this is that they are A) available in massive quantities for analysis and B) have no human subjects concerns and C) use a relatively similar style sheet and register.

      DHS is a long way from tuning this to be feasible for less stylistically precise, more casually-formulated forms of writing, e.g., lots of blogs, most email, nearly all IMs and chat rooms -- especially the latter three because th
  • Wow! (Score:1, Funny)

    by Anonymous Coward
    What great breaking news! Praise be to Roland P. for his insight!
  • by Anonymous Coward
    Your natural language parser will be considered acurate only once it can understand the meaning of Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo [wikipedia.org], until then it is useless.
  • Aha! (Score:3, Funny)

    by suv4x4 ( 956391 ) on Sunday September 24, 2006 @09:46PM (#16180435)
    Obviously, computers can't have an opinion.

    Welcome the new opinion-based CAPTCHA-s!
  • I was a little confused how they used the link between human brain activity on different wave lengths to extract opinion from written text, but Neuro-linguistic programming is apparently not the most popular term with NLP as an acronym.

    This could be a double edge sword for the government. What if it falls in the wrong hands? People all over could use the technology on the news to extract the real information, and realize that things are not what they seem.

    Of course, I suppose that what they would probably
  • This is a pumped-up military-grade version of Word executive summaries?
  • Can do or will do? (Score:5, Insightful)

    by Dan East ( 318230 ) on Sunday September 24, 2006 @10:08PM (#16180581) Homepage Journal
    What comptuers are very good at, though, is scanning through text to deduct human opinions from factual information.

    Funny, because neither of the articles state that. In fact, they don't even say that software can do that at all yet: A new research program ... aims to teach computers to scan through text and sort opinion from fact. Or, We're interested in seeing how we would extract information about opinions.

    So yeah, it would be nice if they could sort opinions from facts. Why they're at it, why don't they just recognize lies from truth too, because wouldn't that be doing the exact same thing? Then we can just run statements made by people suspected of committing a crime through the software, which can then sort out all the facts from the opinions, and we'll no longer need judges, juries or attorneys.

    Roland, next time save yourself some time and just make the whole freaking thing up from scratch.

    Dan East
    • by Memnos ( 937795 )
      Replying to the parent formally, but in fact to Roland.

      Roland, learn some fucking computer science and quit your stupid anthropomorphisizing and out-of-ass expression of your "opinions". Do you have a credit card? Well guess what, a neural network formed an "opinion" that you should be allowed one. Not by NLP, but by pattern detection -- something that we humans evolved to do and are still the best at, for now.

    • it's actually even funnier because "scanning through text to deduct human opinions from factual information" is exactly the kind of thing the Web 2.0/semantic web people have been trying to make possible.

      If Roland had only mentioned the words semantic web, we'd have lots of posts about how it's all hype and vapor instead of posts about how the government can't be trusted with this. That's the power of language.

    • What we need is an automatic bullshit detector filter on slashdot... We could put on the server side and before any post makes it all the 'bullshit' is marked in brown, fact in green, supposition in red and unknown text in courier font 12cpi. Perhaps we can write the software in Perl and sell to DHS! -Bart
  • one thing (Score:2, Funny)

    by shack420 ( 821947 )
    another thing Rolands computer is not very good at is spell checking his posts!
  • by argoff ( 142580 ) * on Sunday September 24, 2006 @10:28PM (#16180723)
    Screw national security, how about search, how about for business and commerce, how about for for culturial exchange and global interaction. The chances of me getting attacked by a terrorist are less than getting hit by lightning, the chances with dealing with foriegn cultures, foriegn business and commerce are rapidly approaching 100%. There are 4 billion people out there who have the potential to mutually benifit from clean communication. Please don't patrinoze me, I'm not too worried about getting nailed by terrorists, but am very bothered by the possibility of having my individual liberties nickeled and dimed to death.
  • "Right now, a consortium of three universities is for the U.S. Department of Homeland Security (DHS) which doesn't have enough in-house expertise in NLP."

    If one of these NLP "expert" systems can extract fact or opinion from that sentence, we should delete it.
  • Knowing the general quality of the average programmer, it stands to reason that this code will only be validated to function in the usual case; thus, the 3l33t coder immediately realizes that simp1e substitutions present an initial defense against the naive academic's simple-minded algorithms and the cut-and-past output of their underpaid cheating slaves (which is, to mean, graduate students or even cheaper undergrads), bringing us closer to the more important question for which this test sentence is being
  • This story fits in the broader context of a developing "surveillance state" in the USA. Forget about wiretaps and such, I just want to focus stuff that is out in plain view.

    The 4th amendment says:
    The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by oath or affirmation, and particularly describing the place to be searched, and the persons or thi
  • I wonder how long before we have to pledge allegiance to the NSA to support their war on terrorism?

    Hmmm, someone at the front door at this late hour. Be right bac...%&$...no carrier.....

  • by Anonymous Coward
    Next up: Solving the halting problem to prevent child pornography, finding Osama by solving the traveling salesman problem in constant time, defeating global warming with the Turing test, and using learning computers to stop illegal immigration!
  • Semantic analysis of opinion in USENET http://www.crs4.it/ict/dart06/program.html [crs4.it] and follow the pdf link under G.Attardi of Univ. of Pisa Italy "Extracting Dependency Relations for Opinion Mining" - treats languages other than English - avoids Chomsky
  • Bushed (Score:3, Funny)

    by Anne Thwacks ( 531696 ) on Monday September 25, 2006 @02:53AM (#16182213)
    You mean it was not the computers that voted for George W Bush? Then who the hell did?
  • What computers are very good at, though, is scanning through text to deduct human opinions from factual information.

    Nope. Computers are good at processing data that has been formatted in a way that they can interpret and running that data through algorithms to come up with some sort of result. They're also good at making grilled cheese sandwiches.
    • by chawly ( 750383 )
      They're also good at making grilled cheese sandwiches.
      But you have to be very careful if you fancy cooking an egg on top of the cheese !!
  • I don't know what 'opinion' means here. If we take opinion == suggestion, computers have already done that. Just look at MS Office 2007 Beta. The spell checker now comes with more features than just a spell checker. It does check the grammar as well as the context of a word that been used. Then, it gives some 'opinions' on how to correct the errors..

    Often, on Windows, there always been 'opinion' given by the system, at least to say that your system is not secure by turning off Windows XP SP2 Firewall.. Does
  • http://www.whitehouse.gov/ [whitehouse.gov]

    eg. "Our goal, and our mission, is to help Lebanese citizens and Lebanese businesses not only recover, but to flourish, because we believe strongly in the concept of a democracy in Lebanon."

    I really dread reading the newspaper anymore. One morning I'm going find that someone has come out four-square for the concept of a democracy in the U.S.A.

    This is incredibly useful and worthwhile research, but I fear it would be totally lost on DHS if it were left in their hands. Just the second
  • Information extraction (IE) is a type of information retrieval whose goal is to automatically extract structured or semistructured information from unstructured machine-readable documents. A type of concept extraction that automatically recognizes significant vocabulary items in text documents, such as, names, terms, and expressions.

Dreams are free, but you get soaked on the connect time.

Working...