Slashdot Log In
World's Largest Databases Ranked
Posted by
CowboyNeal
on Fri Dec 12, 2003 08:05 AM
from the cylinder-sizing dept.
from the cylinder-sizing dept.
prostoalex writes "Winter Corp. has summarized its findings of the annual TopTen competition, where the world's largest and most hard-working (in terms of load) databases are ranked. The results are in, and this year the contestants were ranked on size, data volume, number of rows and peak workload. I wrote up a brief summary of the top three winners in each category for those too lazy to browse the interactive WinterCorp chart."
This discussion has been archived.
No new comments can be posted.
World's Largest Databases Ranked
|
Log In/Create an Account
| Top
| 356 comments
(Spill at 50!) | Index Only
| Search Discussion
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Google (Score:5, Interesting)
Re:Google (Score:5, Informative)
see http://www.cs.rochester.edu/sosp2003/papers/p125-
Re:Google (Score:5, Informative)
(http://www.wastl.net/)
One might see a database as merely a "big file" with mechanisms to access and modify it consistently (and surely, Google has some means to ensure consistency). A big file does not disqualify for the term "database" just because it is not produced by one of {Oracle, MS-SQL, ...} or cannot be queried by the language SQL.
It is also possible to consider the Web to be a database (of Web sites). Or an XML, BibTeX, dbm, whatsoever file.
Sebastian
Re:Google (Score:5, Informative)
(Last Journal: Monday February 27 2006, @09:54PM)
Since Google's stuff seems to be developed in-house, they don't have a major database vendor to nominate them.
Re:Google (Score:5, Informative)
(http://bonoki.com/)
Doesn't have to be relational (Score:4, Interesting)
MasterCard (Score:4, Interesting)
(http://seenonslash.com/ | Last Journal: Friday May 11 2007, @04:02PM)
I agree that there are many companies who would not want to be in that list. There's a small competitive advantage if you keep what technology you use secret.
My porn database (Score:3, Funny)
(http://evilempire.ath.cx/)
Re:My porn database (Score:5, Funny)
What are the pros and cons to databasing (sp.?) your porn? - except perhaps, reduced chance of getting a girlfriend, and chance of ridicule, obviously...
Hey, this is the right place to ask ;)
SQL Server? (Score:5, Interesting)
(http://o2kewl.net/)
I would have liked to see SQL vs non-SQL ranking too.
Re:SQL Server? (Score:5, Informative)
(Last Journal: Friday August 20 2004, @04:37PM)
Typical Microsoft calling their product something generic that should apply to any SQL server. Almost like calling a product .. Windows.
It was originally called Sybase SQL Server but was later picked up by MS who adapted the name. Typical /. objectivity.
Re:SQL Server? (Score:4, Insightful)
(Last Journal: Friday August 20 2004, @04:37PM)
Well, "SQL server" is a stupid way to refer to a RDBS. That's like calling Apache "perl-server". I'm not surprised the only people chosing to name their RDBS products as SQL-something-or-other are the open source developers and Microsoft. Also I've never heard of MS sueing MySQL or PostgreSQL for use of the term SQL in relation to a RDBS.
Besides, the product is officially called Microsoft SQL Server and has always been, just like Microsoft Windows, but everybody refers to it as SQL Server or, if there is possibility of confusion, MS SQL Server or MSSQL for short. Is it malevolence on the part of Microsoft if people can't be bothered to use the full name of each and every one of their products?
Re:SQL Server? (Score:5, Insightful)
sPh
Re:SQL Server? (Score:4, Insightful)
(http://forums.fanhome.com/)
Re:No, it's 30,000GB (Score:4, Informative)
No IMS? (Score:5, Interesting)
(http://www.angelfire...irak/tutorial/day10/ | Last Journal: Sunday November 04, @05:00PM)
Re:No IMS? (Score:5, Informative)
(http://www.angelfire...irak/tutorial/day10/ | Last Journal: Sunday November 04, @05:00PM)
IMS is the database that was used to keep track of things for the moonshot. It is an IBM product. It is hierarchical as opposed to relational. Because of this it can do certain things very quickly, though in general it isn't as flexible as say DB2. Because it has been around so long, applications where having a DB was really important tend to have bought IMS a long time ago and developed systems around it. If your system is old enough, large enough and still works well for you there is no need to migrate to relational. Most of the world's financial transactions pass through an IMS system at some point. It is very stable and has uptimes that measure in years if not decades by now.
Because of this I am surprised that it is not on the list. There are really big IMS databases out there that run a lot of transactions. Because it isn't relational there is some bigotry against it and it is ignored in the popular press.
What surprised me... (Score:5, Interesting)
Re:What surprised me... (Score:5, Informative)
S'okay, I have plenty
But it surprised me that the peak workloads were measured in 100s of concurrent queries. If I had to make a wild guess, I would have guessed 10s of thousands. My blessed ignorance destroyed.
You would typically see tens of thousands (or more) of concurrent connections to a middleware layer - like Tuxedo - which would then multiplex them down to hundreds of connections to the database. This is because there is a lot of latency in establishing a connection, in fact logging in often takes an order of magnitude longer than running an actual query, yet few users submit transactions nonstop. So there is no sense in maintaining tens of thousands of expensive user contexts on the DB server, and there is no sense in requiring intermittent (relatively speaking) users to log out after a short idle period. Middleware does nothing but manage concurrent user contexts, and it can do so very efficiently. A database can't, because it tries to preallocate as much context as it can, and that doesn't match real-world usage patterns, and anyway, database vendors concentrate on their SQL engines and leave middleware vendors to manage the rest.
Of course, if you are a big database vendor, you probably also sell middleware, but there's no-one who tries to bundle the two into one, any more than you'd want a web server to have its own filesystem.
Re:What surprised me... (Score:5, Funny)
(Last Journal: Monday January 17 2005, @09:36AM)
Here I find a knowledgable person on Slashdot,
Who has given a well-written response,
Answered the question without flaming the askee,
Didn't use numbers/symbols for letters,
Never slammed MS or SCO,
And was modded up?
29 TB is the biggest? (Score:4, Interesting)
I recognize Oracle and DB2, but could someone give a brief synopsis of what the other database systems are? And what is an MPP archetype?
Re:29 TB is the biggest? (Score:5, Informative)
You can find the link to the article yourself but
Re:29 TB is the biggest? (Score:5, Informative)
(http://www.menwithrecords.com/)
I support very large Oracle databases for a living (very large meaning > 1TB), databases that must be up 24/7. Backups are done in a number of different ways:
1) Disk syncs, block by block, between disk subsystems at disparate locations, to retain multiple copies of a database in different locations. They can be synced to more than one location too, so you can have as many copies of the database as you want. Your main database is the only "hot" database, the others can be brought up and recovered if needed. We mainly use EMC disk subsystems to do this, the process is called BCV (can't remember what that stands for right now)
2) Real-time replication. One-to-one or one-to-many. All databases are "hot" at all times. This can be great for load balancing too since you can have multiple system onine at the same time. Very difficult to maintain and monitor.
Large databases just can't be put to tape anymore. Even if you did, it would take days or weeks to recover them if they failed. Disk to disk is about the only way to provide backups for really large databases.
Re:29 TB is the biggest? (Score:5, Interesting)
Personally, it has it's drawbacks, but if the indexing is right, you can join hundred million row tables at amazing speed. Based on my experience in data warehousing, it's performance Oracle can't touch (no, I'm not paid by NCR...just a user).
http://www.teradata.com
Overview:
http://www.teradata.com/t/go.aspx/?i
Switches (Score:3, Funny)
(http://www.carotids.com/)
I wonder how much of this database is everytime users have switched to and from AT&T to get those cash bonuses!
94.3TB!?!?! (Score:5, Interesting)
It takes a truely amazing staff to maintain (backup, adminisister, maintence, sit and stare at screens) the servers and maintain the integrity of the data but, good lord...
A 94.3TB database? My upmost, and highest kudo's to those DBMA's and admins there. That is one gigantic task to operate. Being it's AT&T and assuming a great deal is billing and maintence functions these have to be up I'm sure a good 3 nines if not greater.
Regardless of the result of the study, which without actually reading the entire study the end results are simply a short-read of a geek pissing contest, I find it truely amazing how much work, man-hours, and midnight pager calls go into maintaining these databases. I know I don't want our DBMA's jobs and certainly wouldn't want to be a DBMA on a 94.3TB farm but, I know those that do and love doing it. It's a speciality skill and apparently these guys do it right...
Kudos...
Re:94.3TB!?!?! (Score:4, Insightful)
What else do you expect from the company that kinda sorta wrote Unix?
Archive.org not on the list? (Score:4, Interesting)
Quote:
"The Internet Archive Wayback Machine contains over 300 terabytes of data and is currently growing at a rate of 12 terabytes per month." Taken from here [archive.org]
Re:Archive.org not on the list? (Score:4, Interesting)
(http://symbiont.mn.sabren.com/ | Last Journal: Monday November 27 2006, @11:07PM)
Anonymous (Score:5, Funny)
(http://suso.suso.org/ | Last Journal: Tuesday March 09 2004, @12:03AM)
Other factors? (Score:3)
SMP? (Score:5, Informative)
(http://equalarea.com/paul)
Frightening (Score:3, Interesting)
(http://www.therandymon.com/)
Daytona? (Score:3, Insightful)
France Telecom? They must be doing something wrong (Score:4, Funny)
As of 2001-01-01 [ambafrance-zm.org], France had a population of about 59 Million. As it turns out, however, France Telecom (FTE) provides services to a dozen countries, not just France. Checking Yahoo! Finance, I see that
FTE had 2002 revenues of 49B [yahoo.com], with 240,000 employees.
ATT had 2002 revenues of 40B [yahoo.com], with 71,000 employees.
Finally, SBC had 2002 revenues of 43B [yahoo.com], with 175,000 employees.
So nothing terribly unusual about the size of their database. But it's obvious that the French employees are a bunch of unproductive slackers...
bah, meaningless (Score:4, Interesting)
Without system descriptions (like in tcp) it merely shows that such a top-end is feasible.
What about total cost?
annual cost?
time to build?
software versions?
hardware?
staffing composition?
I mean really, a 500 gbyte database on a modest single CPU server is far more challenging than a 2 TB database on a 64-CPU E10k.
We are larger: 500TB (Score:3, Interesting)
(http://modesto.sf.net/piave/index.html)
Press release:
http://www.slac.stanford.edu/slac/media-info/20
Cheers
Only on Windows platform! (Score:5, Informative)
Lastly, in the Windows OTLP category HP servers were used by 7 of 10 organizations, and Microsoft SQL Server was the DBMS choice for seven respondents.
Neither WindowsNT, nor MS SQL are generally a choice for the top databases. In fact, to make the entry in this list, a Windows-Database was required to be only half as big as databases on other platforms:
In order to qualify for the TopTen program consideration, any commercial production database implementation was required to feature a minimum of 500 GB of data for Microsoft Corp.'s Windows and NT platforms and 1 TB of data for all other platforms
ms