
Northern Light Technology Makes Deal WIth C.I.A. 125
Llywelyn writes: "The C.I.A. has evidently written up a contract with the group Northern Light Technology to develop a search engine that can sort through the C.I.A's increasing mound of unprocessed data. Unfortunately, one of the consequences of this is that Northern Light's public search engine is fated for destruction later this month. " It's inevitable, IMHO, that some of this happen - the search engine world is overpopulated right now, and with the economic downturn, more and more companies will move to where they can survive.
Top Ten (Score:2, Funny)
"pr0n"
"Where is Osama?"
"missing equipment"
"JFK"
"Castro"
...et multiple cetera...
Re:Top Ten (Score:1, Offtopic)
Probably the biggest online version of "Where is Waldo?" known to man
Mixup (Score:1)
it aint the CIA.... (Score:1, Redundant)
Re:it aint the CIA.... (Score:2, Informative)
Re:it aint the CIA.... (Score:2)
Re:it aint the CIA.... (Score:2)
How about this definition: A contractor is a private entity with a contract.
Duh.
The CIA gives In-Q-Tel orders, they don't sign deals.
Re:it aint the CIA.... (Score:2)
Re:it aint the CIA.... (Score:2)
Contractors get paid, and they get to decide how they are to fulfill their requirements. In-Q-Tel has no autonomy: anytime the CIA wants to change its marching orders, it just says aye.
We have a similar concept here in Canada: We call them Crown Corporations. The CBC (and its bigger cousin the BBC) is not a contractor, providing television services for the government of Canada. It's a part of the government. It's just not organized like a typical government agency, but rather organized like a corporation.
I wonder: Have you ever worked for a government contractor, or in government? The difference is pretty huge. (I've done both.)
Re:it aint the CIA.... (Score:1)
Contractors get paid, and they get to decide how they are to fulfill their requirements
IQT is being paid by the CIA for data processing. They chose to hire Northern Lights to supply a search engine to help handle the data given to them. i rest my case, rebuttals shall be ignored.
and btw. yes, i too, have worked both as a civilian and contractor for the government. as a matter of fact, i'm still a govie contractor.
Re:it aint the CIA.... (Score:2)
Northern Lights is a contractor. (We're not arguing about that. We're technically arguing about whether they're a sub-contractor.)
You seem to believe that everything that receives government money is a contractor. This would make, for example, the BBC a contractor. This is my final post.
Shoulkd use google for dealing with criminal (Score:1)
Hmm (Score:2)
I could use the hits..
Re:Hmm (Score:1)
Thank God it's not Google (Score:1)
Re:Thank God it's not Google (Score:1)
Search engine lost (Score:1)
Re:Search engine lost (Score:1)
Re:Search engine lost (Score:1)
Searching on my typical 'handle', 'webwench', for example, I usually get about 100 hits out of Google off one site -- atTheFaire.com, mostly. That webwench is apparently pretty busy. I can click through page after page of results and not find one about me.
In Northern Lights, I get one search result for AtTheFaire, rather than tons, and one result for each of several sites, and I'm the subject of #7.
I have to say, I like it. Just in time for it to go away.
Still Available ... Why Not Google? (Score:5, Informative)
One of the neater features of Northern Lights was the folders. I liked how they organized the info and let it flow out of your continuous clicking.
What is funny is that Northern Light is actually a better search engine than Google for specific info. Northern Light drills down on the subject only and doesn't take into consideration links to the info as Google does.
Re:Because Google does not work very well (Score:1)
Paste the following into Google:
"communiques would bring aid and succor"
You will be taken to my "free speech links" page. Any other random phrase cut and pasted from that page will work as well, though some are not as unique.
Re:Because Google does not work very well (Score:2)
BTW, You might notice that some pages don't actually have the phrase you were looking for. That generally means that there's are links pointing to that page with that phrase in them. For example, if you search for "html 4 spec" [google.com] you get this [w3.org] page. That page doesn't contain the phrase "html 4 spec", but there are lots of links pointing at it that contain that phrase, so it's probably the page you want if that's your query.
Interesting/amusing query: "click here" [google.com]
Re:Still Available ... Why Not Google? (Score:2, Interesting)
Now, that's unlikely to be what's driving the NSA decision but it sure makes me wonder how "deep" Google really is.
Re:Still Available ... Why Not Google? (Score:2, Informative)
I don't mean to be Google's official defender/apologist, but searching Google for The Hobyahs results in 323 hits [google.com]. It displays 171, with an option to also show the rest which are filtered out as duplicates. Must've been a server problem when you tried. (Itself not a good sign, but I've never seen a failure like that myself.)
Re:Still Available ... Why Not Google? (Score:2, Informative)
words & phrases within the search results, and then uses a built-in general understanding of what makes for good cluster labels (e.g.,
phrases) to do its thing.
Besides web search, there's clustering of lots of other internet content at http://vivisimo.com/html/demos/index.html
such as corporate websites (Microsoft, Sun, Cisco, HP), government, universities, science, news, ebay, etc. etc.
Re:Still Available ... Why Not Google? (Score:1)
Bull.
How's this for an obscure query: How many nanoseconds long is a shake?
Try it in Northern Light [northernlight.com] and be fuddled for a while. Now try Google [google.com] and WiseNut [wisenut.com]. You won't even have to follow the hits. The answer is in the summary of the second or third hit. Not bad, IHMO.
Could be worse... (Score:1)
Huh? (Score:2, Interesting)
I too feel there are too many search engines. It is extremely rare that I don't find what I'm looking for on Google, but it's not good to put all your eggs in one basket. While Google may be the best today, there needs to be competition in every market (well, almost every market).
Re:Huh? (Score:3, Informative)
Public free search engine didn't pan out for them.
Re:Huh? (Score:1)
Northern Light does a great deal of enterprise work (from my understanding...) that predates the CIA work for a long time. I think it is totally coincidental.
I mean, it's the beginning of the new year, they probably made the decision sometime last year to close the public site as it just wasn't brining in any cash - who makes money from web advertising? nobody. you can't sustain an army of engineers on a free public search site. at least you can't when the competition is tight, and though the Northern Light search engine is very powerful, more precise and relevant than Google, it's just not as popular. Less eyeballs = less $$$ per click from advertisers. And Northernlight kept their integrity and never sold out like AltaVista (The advertisement search engine), and kept it "all about search", and relevancy, for so long.
You can't give away something for free if it is costing you alot of money to do it forever.
On the other side, enterprise search, classification and what they call "content integration" services (see their press releases on northernlight.com) probably make a lot more money, and since their site has shown how GOOD they are at search and classification, and selling premium documents, probably puts them in a really good position in this market.
I think it is a wise move for them as a business, though I will miss the search engine.
Re:Huh? (Score:1)
Northern Light is a good search engine, but they don't know how to design for the masses. There are too many choices on the home page, Joe Average would never know where's the best place to search, and searching a couple of times in the wrong place would send him packing to one of the many other choices with just one serach box. And the design is cold/institutional, as if you were visiting a bank.
All this is fine if they want to appeal to an elite niche market, but there's no way to survive on that.
Cult of inteligence (Score:2, Offtopic)
Anyone notice the blank line at the top of this article?
What's going on here? I smell a censor!
The cia has [deleted] civil liberties
[deleted] Former Director [delted]
without any thought
[deleted]
won't stand for this!
(please don't mod this down if you haven't read the book)
not necessarily (Score:2, Interesting)
according to this [slashdot.org] article posted back in August.. there should be even more search engines popping up in the future.. most of the ones on that list are still in beta mode.. personally, i like Teoma a lot
Re:not necessarily (Score:1)
Re:not necessarily (Score:1)
An even greater advantage to users would be a search engine that "knows" your subtle preferences; i.e., if you're with H&R Block, you want TCP to mean "Taxpayer Compliance Program." Also might be a great medium for targeted advertising, without spreading your private info all over hell and back.
What? (Score:1)
Re:New job for the 21st century? (Score:1)
Think about some of the NEA projects which are doing their work by computer analysis of old photographic plates. The "data" has been there all along. What was missing was the information.
From the viewpoint of the various world (not just US) intelligence agencies this is a tremendously important activity. IIRC, over 50% of the US CIA's data comes from open sources. It's the ability to put all the pieces together into a coherent picture that is the key.
Well someone has to do something with that data (Score:2, Interesting)
It makes you wonder about the QUALITY of the data they are collecting,
Are they going o index their HOWTO's aka
HOW TO WIRE A CAT WITH A MIC (Seen previoulsy on slashdot)
As they teach you DAY ONE in Naval Intelligence
Everything is simply a piece of a puzzle, expect nothing monumental, no matter how small it is a detail of a bigger picture.
Now without addressing information in this manner and looking for ONE big hit , the process of intelligence gathering is broken.
YOU NEED SOMEONE to al least try to review this data before its all clumped into a selective search engine and forgotten. The CIA has horrible record retention policies in place. My bet 10:1 this is the worst possible fate of this data.
Probably just be better to auction it off to some willing buyer and hire more spooks to gather more data to auction off again, then maybe then the CIA could be considered usefull at least as a govt profit center
Re:Well someone has to do something with that data (Score:1)
I don't know about that, it seems to me that data in a search engine becomes only more powerful. Each document is still available to be read in toto, to a larger potential readership (anyone with access). And patterns undiscernible to any single reader (like the same names appearing in 50 out of a million documents) can be found and assembled from their diverse sources.
Great idea (Score:1)
Plug: related technology (Score:2)
Lots of companies have vast amounts of data, both text and otherwise to which they need access. The company I work for, Maxim-IT, Inc [maxim-it.com] focuses on search technologies for non-structured data (like CAD files and the like).
This lead to the CIA's very own "Ten Most Wanted"? (Score:1)
Of course, this would be a bit different than that of the FBI's list, with a ranking of search strings rather than people. I would predict the top three being:
(1) where hell Osama bin Laden Cave Country
(2) bad-guys terrorists attack United States not nice people
(3) Natalie Portman hot grits petrified
Re:This lead to the CIA's very own "Ten Most Wante (Score:2)
Another one bites the dust.... (Score:2)
Intriguingly, according to some the growth of the web is slowing. The last search engine to index the web before it reaches quiescence is the likeliest candidate for survival.
Re:Another one bites the dust.... (Score:2)
Except that this was exactly how a lot of the search engines said that they would make money. They would advertise themselves by setting up the free web-search services, and then make deals with private organizations to index their files. The public search becomes an ad for the moneymaking indexing venture.
That was how OpenText and that crew wanted to make their first billion, anyway, IIRC.
It's just that they thought that lots of companies would want their services, and not sign them to exclusive contracts as it appears the CIA has done here. So they'd keep the public sites up to attract more clients. The exclusivity of this contract makes the public site redundant.
On Intelligence (Score:5, Interesting)
As much as the focus tends to land on it, information gathering is not by any means the weakest link in the intelligence system. Probably we hear most about it because
a) it is glamourous (think James Bond), and
b) it often affects our civil liberties.
But the real problem with intelligence is the processing of retrieved raw information. They gather so much of the stuff it's extremely difficult to sort through it to figure out what's relevant and what's not.
That is why whenever something bad happens (like Sept 11) the intelligence community looks sloppy. In retrospect they can dig out wads of unprocessed information that would have given advance warning of the disaster. Then they take a lot of heat for missing it, even though they may not really be at fault. Sometimes it's a matter of finding a needle in a haystack.
It's a little more interesting to geeks because it's an issue of pure computer science. Processing raw data into meaningful information is computing at its best.
But developing better algorithms as a response to a national disaster is never going to be a solution that catches the public's imagination.
Re:On Intelligence (Score:2)
Kinda like that scarecrow that scares off the giraffes. What? You don't see any giraffes? Must be working then
One thing I will say is that I give these terrorist orgs more credit than to think that they have truly useful data somewhere online. I don't know if this is really going to help the CIA, or if it's just a public showing of action, but I can't imagine
Re:On Intelligence (Score:2, Interesting)
Er.... I thought this article meant that it would be searching the CIA's own databases for the mounds of information that they can't quite link together to find the master plan.
Like the way the KGB could never find Bond, despite the fact that he always introduced himself as "Bond, James Bond".
If they can link all their data together (e.g searching transcripts for Osama (not Bin!) might have an interesting result).
Maybe that'd just make too much sense.
Re:On Intelligence (Score:2)
Re:On Intelligence (Score:2)
google has some cached already [google.com], so why not?
No, they aren't going to prowl the internet for intel on terrorists, they need technology to analyse the mountain of info they have already collected, but don't have the manpower to analyze.
Re:On Intelligence (Score:1)
It would be useless.
Chet
Illumination (Score:2, Interesting)
Not discontinuing search engine business... (Score:3, Redundant)
According to fuckedcompany.com, they will be converting their search-engine service to a paid-only model.
Re:Not discontinuing search engine business... (Score:1)
Too bad fuckedcompany didn't read the press release [northernlight.com].
Goodbye Northern Light :( (Score:1)
Too bad. Northern Light was my primary search engine for a long while, and was my secondary recently (no need to say what #1 was.)
The really nice thing was Northern Light's categorization of hits. It was often far more useful than trusting Google's "sort by relevance"
Damn this is sad.
So (Score:1)
Your bias is showing (Score:5, Insightful)
You make it sound like working for the CIA is some odious move of last resort. Perhaps the management and staff of Northern Light is excited about working with the intelligence agency. Perhaps they see it as a way to help their country. Perhaps the processing of terabytes of data is a thrilling prospect from a purely intellectual point of view.
The standard /. dislike of all things governmental is not necessarily mirrored through all geeks.
Re:Your bias is showing (Score:2)
I'll say. I like the Government! The Government (through DARPA) paid for my Masters' degree. That's Your Tax Dollars At Work, circa 1998. Thank you, taxpaying Geeks!
Re:Your bias is showing (Score:1)
Now, the people currently in the mechanism... Them you can and should look at. Closely. Find out who's doing what. So you can go right for the cancer and leave people who are doing a good job, actually serving the people.
And no, it's not easy, it's an eternal struggle. But it's better to take thoughtful action than to just go "The Government did it!" and forget that the also paid for imadork's education. Better to find the good people in the CIA who wanna shift through the pile of data to stop another 11/09 from happening and encourage them them group them with the idiots in the organization selling drugs.
I'll start before I REALLY ramble. Gah.
This is a venture capital deal, not a contract (Score:5, Informative)
And the article misspelled Gilman Louie's name...
Nick
Re:This is a venture capital deal, not a contract (Score:1, Informative)
(a former Spectrum Holobyte/Microprose employee)
There are less search engines than you think (Score:5, Informative)
This is just not true. Over the last year, more and more search engine companies are effectively consolidating - by licencing "search engine technology" from another company. The real down side of this is that the more popular (popular by the licencing...not by users) are a "pay" engine, whereby companies can move their listings higher in the rankings by paying a fee.
This has two main side effects.
One, there are a lot of search engines out there that are really the same search engine. Same query, same results.
Two, when you search with them, you're not really getting what you asked for, but what someone payed for.
I understand why companies are doing this - there isn't a really strong revenue model for search engines right now; banners don't cut it.
I suspect that soon, good search engines will just be a (hopefully) inexpensive pay site, where you pay $30 a year and can use that search engine.
Pay search engines... (Score:4, Insightful)
But can you imagine all the bad possiblities if they were able to actually tie all your searches together and see WHAT YOU searched for? Sure they can do it by i.p. or cookie, but an actual account, probably verified by credit card?
On the otherhand, a search engine is a basic need to use the internet. And I'd be quite surprised if some of them didn't start heading this way, REALLY CHEAP though. Incidently I don't know exactly when google.com became my ONLY search engine, replacing altavista.com, but it happened. Probably because of the excellent results (not perfect though) and the light interface.
On another note, I get the BEST referrals from google.com to my site. I get the MOST referrals from msn.com to my site. I say BEST from google.com, because the people that find my site through them, most likely want to see my site, and end up staying. MSN's referrals are usually pretty broad topics.
Re:Pay search engines... (Score:2)
Signs of convergence (Score:2, Interesting)
Microsoft is making a stab at this using SmartTags. Of course, the intent there is not to make the web more useful, but more Microsoft.
Then there's the w3 symantic net (http://www.w3.org/2001/sw/), and finally there's the grand daddy of them all: Xanadu (http://www.xanadu.net/).
No real thoughts here, just an observation.
End result... (Score:2)
Pretty stupid if you ask me. I doubt they are then intelligent enough to change the spider agent string and route the spider through various innocent proxies to disquise it is the FBI spider...
I mean, what does the FBI do? Enter in "warez" into a search engine and go out and bust heads depending on the results returned?
You know what I really thin@~.~.~.~.~.~~~..~~#~~
NO CARRIER
Re:End result... (Score:1, Insightful)
"Ahmed, have you posted all our secret plots to the website yet?"
"Yes Jimal, just as you have instructed. And I have modified the robots.txt to keep the great satan's search engine crawler from discovering our plans!"
"Allah be praised!"
Sheesh, oh and it isn't the FBI either.
CIA's data has nothing to do with this (Score:1)
What about GeoSearch? (Score:1)
The feds are watching and reading (Score:2)
Re:The feds are watching and reading (Score:2)
More info (the Lee's own desc of what happened to him) can be found here:
http://www.kuro5hin.org/story/2001/12/20/211923/8
Neal Stephenson's "Library" anyone? (Score:2)
Grab.
Library of Congress and the CIA (Score:1, Funny)
Re:Library of Congress and the CIA (Score:1, Insightful)
The CIA doesn't overthrow third-world despots, they put them into power so that politically connected companies keep getting richer (under the guise of "stopping the commies"). Hmm, I guess that needs updating now... I know! Replace commies with terrorists.
The Truth about the CIA (Score:1, Insightful)
Re:The Truth about the CIA (Score:1)
Sure... and Augusto Pinochet saved us from the terrible threat of Argentina invading the United States.
http://www.lakota.clara.net/
Wait... I've got a better one... Manuel Noriega
protected us from fierce banana-weilding attackers.
http://www.cnn.com/resources/newsmakers/world/n
How about... Saddam Hussain?
Or... Peru's President Alberto Fujimori?
http://www.csrp.org/rw/rw1081.htm
I can go on and on.
Just search Google for CIA dictator.
All petty tyrants served up by the CIA to do a few basic deeds: Provide growth & processing of narcotics for the George H.W. Bush drug-smuggling empire and provide a cover for the Nazi-controlled CIA to bankrupt nations so they may buy up mineral rights, oil land, media control, political elections, and crush the thinking majority under the thumb of the insane Nazi minority. The CIA has no interest in protecting America and is in fact the biggest danger to freedom and sanity in the world.
Peddle your fraud elsewhere shill.
Yawn.... (Score:2, Insightful)
Too bad (Score:2, Interesting)
Anyone know of another decent engine with a good boolean implementation?
60 Minutes Investigation (Score:4, Funny)
Reporter: You mean, the procurement officer paid $6,000,000 for grep?
Guy with face hidden: Yes.
Reporter: What happened to him when they found out?
Guy with face hidden: Well, they were going to hit him with a hammer, but they decided that would be hypocritical...
Flooz (Score:1, Funny)
Northern Light Top Stories (Score:2)
I hope it will still be available. I've tried others but not found a better one. Anyone else?
I remember the last time someone made a deal... (Score:1)
I think his name was Fidel..
haha..
gallix
Related buyout?!? (Score:1)
Too Many search engines (Score:2)
NL is 2nd best (Score:1)
Not as good as Google; better than Altavista.
Altavista used to be second best but NL overtook it.
The problem (for NL) is that second best is not good enough: I only use Google.
Of interest (???) to slashdot readers: isn't NL one of the last VMS diehards?
Re:IMHO (Score:2, Funny)
Non-English speakers are probably going to have more trouble with the rest of the words in that sentence...not to mention your explanation.
Re:IMHO (Score:1)
Re:IMHO - see the Jargon file! (Score:2)
IMHO
[from SF fandom via Usenet; abbreviation for `In
My Humble Opinion'] "IMHO, mixed-case C names should be avoided, as mistyping something in the wrong case can cause hard-to-detect errors -- and they look too Pascalish anyhow." Also seen in variant forms such as IMNSHO (In My Not-So-Humble Opinion) and IMAO (In My Arrogant Opinion).
Anomaly
PS - God loves you and longs for relationship with you. If you'd like to know more, please email me directly.