Super-Fast RDF Search Engine Developed

The Register is reporting that Irish researchers have developed a new high-speed RDF search engine capable of answering search queries with more than seven billion RDF statements in mere fractions of a second. "'The importance of this breakthrough cannot be overestimated,' said Professor Stefan Decker, director of DERI. 'These results enable us to create web search engines that really deliver answers instead of links. The technology also allows us to combine information from the web, for example the engine can list all partnerships of a company even if there is no single web page that lists all of them.'"

Official DERI Website

Here's the link to the official NUIG: DERI (omgwtfbbq) website in Ireland:

DERI [www.deri.ie]

Re:TMA: Too Many Acronyms

Why assume everyone knows your acronyms.

OMG: Oh my God!
WTF: What the fuck?
BBQ: Barbecue.

HTH

Re:Official DERI Website

I tried to access that site, and I got a good look at their DERI Error.

This could be huge

Except for the minor little problem of getting everyone to agree on the ontologies. Being able to search quickly is important, but until somebody comes up with the Dewey Decimal System for all knowledge, it won't mean much.

Next up: Ontology spam

Yes, creating a consistent ontology is challenge. But the bigger challenge is the lack of incentive for ontology truthfulness. If this type of search becomes popular, ontology spam and OSEO (Ontology Search Engine Optimization) will become a booming industry.

Re:This could be huge

Ah, but the Dewey Decimal system only works because responsible people are involved in categorizing everything. They let just anyone publish information on the internet these days.

Why would I want to search...

for a Radio Direction Finder?

Links!

These results enable us to create web search engines that really deliver answers instead of links.

I need both: answers *and* links! Many times when I search the web, I don't know for sure what am I searching for, let alone being able to ask specific question...

Search solved. World hunger next.

Having solved the problem of search, and providing a breakthrough product that has consciousness to what was previously mere series of tubes, now the National University of Ireland announced that it is going to solve world hunger next, may be in three months. Other projects in the pipeline includes cure for cancer and solving full Navier Stokes equation.

Hype

users should get more relevant results

Yet another /. article parroting an uncritical popular press account of a press release.

RDF?

I didn't realize Steve Jobs' Reality Distortion Field was able to be harnessed and bottled in a search engine, or any software for that matter. His abilities are boundless!

I'll prove him wrong

"'The importance of this breakthrough cannot be overestimated,' said Professor Stefan Decker, director of DERI."

This is without a doubt the greatest invention in the history of time!

There, I just proved the professor wrong. Muahaha.

contradictory

The importance of this breakthrough cannot be overestimated

I think he just did...

Cannot be overestimated

- "The importance of this breakthrough cannot be overestimated"

The importance of any event can be overestimated and quite often is overestimated. It is called hype.
When speaking of XML, XHTML and semantic WEB then the word "overestimated" fits just nice.
If this was not the case then HTML should long have been dead and the whole WEB should have been based on pure XML with meaningful tags.

-- Do not read me, I am a stupid tag

Could be interesting, but missing details

What kind of data set did they use? The structure and contents of the graph that is the data in an RDF database has a huge impact on the performance of query execution, and different applications have different structures.

What kind of queries are they running? There are several different RDF query languages (think of SeRQL, RDQL, N3, SPARQL, etcetera) and some of them support quite complex queries. Quickly finding the answers to a simple query like

SELECT ?name WHERE ?name <http://xmlns.com/foaf/0.1/name> "John Smith"

is just a matter of an indexed lookup and not very special. But, like in SQL, much more complex expressions can be generated that require complex index operations on the query execution level. Having implemented an RDF database that supports SPARQL queries an order of magnitude faster than the software the W3C uses for their experiments (which, admitedly, doesn't have performance as a prime requirement), I know that it's possible to do simple things fast, but the interesting part is handling RDF queries that don't easily map to efficient database operations.

Which brings me to the most important point: where is their detailed report? Can I get the software somewhere and perform my own tests? The article is too vague to draw any conclusions about what their RDF database does, and how good it is. I'd love to read up on it, but I can't seem to find the information.

Here's the Tech Report

Hello, I am one of the main developers of SWSE. True, the press release is vague, but there is only so much you can say in a press release aimed for the general public.

We have a Technical Report available at http://www.deri.ie/fileadmin/documents/DERI-TR-200 7-04-20.pdf [www.deri.ie] that should answer most of the technical questions.

From the abstract:

"We present the architecture of an end-to-end search engine that uses a graph data model to enable interactive query answering over structured and interlinked data collected from many disparate sources on the Web.

In particular, we study distributed indexing methods for graph-structured data and parallel query evaluation methods on a cluster of computers.

We evaluate the system on a dataset with 430 million statements collected from the Web, and provide scale-up experiments on 7 billion synthetically generated statements."

Re:Here's the Tech Report

You are too modest. You're the lead author. Congratulations on a first-rate contribution to mankind. And such a young pup [harth.org], too.

SUPER Speed

Colonel Sandurz: Prepare ship for light speed. Dark Helmet: No, no, no. Light speed is too slow. Colonel Sandurz: Light speed is too slow? Dark Helmet: Yes. We're gonna have go right to... SUPER speed. [everybody gasps] Colonel Sandurz: SUPER speed? Sir, we've never gone that fast before. I do'nt know if this ship can take it. Dark Helmet: What's the matter Colonel Sandurz? Chicken? Colonel Sandurz: [Wimpering] Prepair ship! [Calms down] Colonel Sandurz: Prepare ship, for Ludicrous speed. Fasten all seat belts. [everybody fastens in their seat belts and locks all of the doors] Colonel Sandurz: Seal all entrances and exits. Lock all stores in the mall. Cancel the 3-ring circus. Secure all animals in the zoo... Dark Helmet: [Takes the intercom from Sandurz] Gimme that, you petty excuse for an officer! [speaks into the intercom as Sandurz puts on his seat belt] Dark Helmet: Now hear this, Ludicrous speed... Colonel Sandurz: [Interrupts] Sir, you better buckle up. Dark Helmet: [to Sandurz] Ah, buckle this. [Into the intercom] Dark Helmet: SUPER speed, go!

Two things...

First, giving the amount of time and the number of items searched means nothing. Are they doing it on a BlueGene or an Apple II?

Second, the problem with "the semantic web" if you're relying on people providing the metadata themselves, is the reliability (trustworthiness?) of the person creating the metadata. There's a reason the meta name="keywords" tags aren't a significant factor if at all in any of the major search engines' ranking systems.

sounds fishy

Of course a search based on meta data is going to be faster and more accurate, but only when the meta data is correct. We've had this since the beginning of the interweb; people would load up their pages with bogus meta data just to generate search traffic. Because of this dishonesty, search engines have had to resort to other methods of evaluating and indexing pages (for example, based on actual content).

I don't see any difference between this new RDF and that old stuff.

Save the hype

So now we have a search engine capeable of making a godzillion searches in a data domain that does not exist yet. That's all great and dandy, and we do indeed need new models and architectures for search engines once (if) the web goes all semantic. However, when (if) the semantic web ever becomes a reality, this search engine will long be retired. So, this result is great from a research point of view, but don't expect it to leave the lab.

Developer on this project

As one of the developers on the project (along with user aharth), feel free to ask any specific questions you may have here. The article is quite vague and so I refer you to a technical report at http://www.deri.ie/fileadmin/documents/DERI-TR-200 7-04-20.pdf/ [www.deri.ie].

This is great and all

but why would I want to search several million statements from the Robotech Defense Force? I mean, sure I'm an Anime nerd, but there are limits...

RDF is a bad idea

I just read the basics [w3.org] of RDF and I can see that this could be a really really bad idea. If RDF is intended as an internal data representation for a search engine company to use then this is great. The search engine company or your own company's search engine staff can police and audit your RDF data. However, if I'm reading this right RDF is *supposed* to be populated by *volunteered* data. As such you're going to suffer not just the Wikipedia effect but all the problems seen in MetaData from an internet generation ago.

You'll see RDF associations linking the president to a crass picture of a donkey or a goat of some kind. You'll see companies set up to deliberately poison RDF data with false links designed to drive traffic to a site... you'll see sock-puppets and all kinds of other attacks.

This whole effort reminds me of the "this is spam" bit that was proposed to stop spam. You can't expect spammers to say to themselves, "wait, I better flip the this-is-spam but to true before I send this" you also can't expect people to not abuse the RDF system in similar ways.

Don't expect that if you RDF search for Stephen King that everything that comes up was actually posted by him. Imagine the pages that would get attributed to the president or Mr. T as a prank... the information would only be useful if you could verify the document as legitimate first.

The "is part of" feature is the most likely target of abuse I think. I could say that everything I wrote is part of the New York Times or as part of some official document that gets searched for often. The result would be erroneous hits in RDF search and artificial authority for my crack pot theories.

What the hell is RDF?

Hard to know if this article is worth reading or not when the summary doesn't even tell me WHAT RDF IS!! Criminy.

wonderful

just what the man needs, a good search engine for his mesmerizing skills, im leaving california now.

Overestimation

"'The importance of this breakthrough cannot be overestimated,' said Professor Stefan Decker, director of DERI."

I agree. In my estimation, this could well foretell the cure to AIDS, cancer, world hunger, war, and genital warts.

42

NO.

The first answer will be 42.

Re:Great!!

Now all we need to do is get everyone to start using RDF.... wait.. you dont even know what that is??

It's the Resource Description Framework [w3.org], which RSS is a subset of.

Re:"'...this breakthrough cannot be overestimated"

I asked the RDF search engine, and that's what it told me. Maybe if we ask it the right question it could come up with an answer to do that. Now if only we could devise a machine powerful enough to tell us what that question would be...

Re:Boo @ Slashdotters

...and all they get is the usual, automated Slashdot moaning.

At least you've realized the truth, this being ./, everyone owns a spambot that searches for keywords in stories. You get this particular crapflood when the bots realize that something with the words "revolutionary" and "internet" (and possibly also "semantic")has appeared. Everything here is just markov chain output. Welcome to the future.

Also, there's an intelligent thread further up where the lead researcher posted that no one seems to be responding too :(