Postgres Beats MySql, Interbase, And Proprietary DBs

Mike writes: "An independent organization tested Postgres 7 vs. MySql, Interbase, and two leading commercial databases using the ANSI SQL Standard Scalable And Portable benchmark and found that Postgres was the clear winner. In fact, Postgres was the only open source database to offer similar performance to the two commercial applications. The results are detailed here."

Re:Let's hear it for lie, damn lies, and benchmark

They compared the bleeding edge postgres (7.0) with the old-as-heck mysql (3.22) - they're now up to revision *22* of the development series for mysql - that's a pretty huge amount of changes. I would have been much more impressed with this if they had ran the comparison between 3.23.22 and 7.0.

Postgres 7.0 is the current stable release. According to http://www.mysql.com/downloads, MySQL v3.22 is the current stable release and v3.23 is the current beta release of MySQL. Given that, I can't blame them for not testing a stable product versus a beta product.

Even if there have been "a pretty huge amount of changes" as you state, it's still marked as a beta product. As for myself, I wouldn't use a beta product in a production environment until it's been marked as stable by the developers, no matter how stable other people might say it is. That's why I use MySQL v3.22 right now. If something is marked beta, there is more than likely a reason for it. It would be irresponsible of me to risk using it and risk losing customer's data. I'm sure other businesses and individuals can't afford to take those risks with their data either.

Yes MySQL now does transactions

It was added a while back. And still, mysql (when using MyISAM) is a lot faster than competing databases. Sorry I've done my independant benchmarks in the "real-world".

Don't believe me, just test them out for yourself. MySQL opens a can of whoopass, but people just don't realize that I guess...

I really strongly encourage everyone to benchmark their database servers independantly, instead of trusting these "independant" companies like.... well we all remember MindCraft, can we trust these organizations?

Re:I'm shocked!

The tests were performed using ODBC drivers. ODBC drivers vary widely in quality but are always inferior to the native API of any given database. For Interbase they used the ODBC driver from Interbase 5. This is a notoriously poor performing driver. I've found Interbase to be as fast as SQL Server 7 and nearly as fast as Oracle 8i when accessed using native methods.

consider the source

This 'article' is nothing more than a press release from Great Bridge [greatbridge.com].

There may be some additional information learned by reading the results of the benchmark from
http://www.tpc.org/New_Result/TPCC_ Results.html [tpc.org]

Although I am having a hard time finding any reference to Postgres on that page. Can anyone find any better references?

-k

Re:Not Surprising

I am quite surprised that the Postgres results are almost 3 times that of proprietary databases. That seems kind of fishy - surely the big proprietary databases aren't THAT slow.

Someone who was very familiar with Postgres and not very familiar with Oracle, Informix or whatever else might easily obtain that sort of result. A misconfigured database can creep along at a snail's pace.

Re:MySQL

This is very very true.

The native interface to Oracle is *much* quicker. Having used Native drivers to Oracle on ColdFusion machines I can definately say it works quicker than ODBC.

At my work we write scalable applications and in the near future I would like to sit down with a copy of Oracle, tune the hell out of it and spend a week optimizing the atabases, PostgreSQL and Oracle, and MsSQL 7.0 because Benchmarking just one system like a Database is not really telling, Stick it on one of my Cluster of ColdFusion machines and lets see if the DB can keep up with the rest of my Application.. THEN I will be impressed not until then.

I read the TPC Benchmarks I even know how they work, but they still seem useless to me because We cant afford some 70K / Year Oracle DB Admin to come and constnatly Tune our Databases for us, if I cant learn enough to get it resonably well configured in a certain time frame I will go with something that *CAN* handle the load, and if only Oracle can ( which I dont believe ) Then we would be forced to find a solution that performs, But I have a copy of DB2 as well and you guys would be very suprised at the little places support for Databases come from.

I have read and installed ColdFusion on Linux and in the documentation and release notes Allaire has instructions on how to set up Both MySQL *AND* PostgreSQL via ODBC, so its not like a native driver, but it is support and I find it rather impressive that a big company that is more or less leading the way for commercial web application development platforms reaches out to support both MySQL and PostgreSQL.

My opinon is if you cant do much with the database what good is it?

Jeremy


If you think education is expensive, try ignornace

Free Speech Issues in Licenses?

I found this the most interesting quote:

The two industry leaders cannot be mentioned by name because their restrictive licensing agreements prohibit anyone who buys their closed source products from publishing their company names in benchmark testing results without the companies' prior approval.

Apparently, if you succumb to the MS, or Oracle virus (or whomever it was they tested), you're not allowed to talk about your experiences comparing them to other products. I wonder exactly how legal that clause is in the license....

D

A book is coming out soon

I don't know if it's a good book, but there's a book called PostgreSQL [fatbrain.com] by Jeff Perkins coming out in October. Fatbrain didn't have a description, but Amazon did:

PostgreSQL is the perfect book for you if you use PostgreSQL at work and on your Web sites wherever you expose data on the Web using Linux and Apache. It covers the new features of PostgreSQL as well as the PostgreSQL processor, which defines all necessary objects in a database, to get acquainted with SQL and to test ideas and verify joins and queries. Database developers for corporate and Web applications will find this book useful. It is geared toward intermediate to advanced developers who have designed and administered databases, but not PostgreSQL. The accompanying CD includes PostgreSQL, plus sample databases and modules.

If you just want to use it (and not admin it), O'Reilly's Programming the Perl DBI [oreilly.com] has some info on accessing a PostgreSQL DB (hint: it's not that different from any other DB when seen through DBI). Oh yeah, MySQL & mSQL [oreilly.com], also from O'Reilly has a little bit about it (but not very much at all). I guess readmes, man pages and HOW-TOs [linuxdoc.org] are your friends for the next couple months.

If you're really curious, throw it on a test machine and (if possible) "port" some apps to use Postgres instead of MySQL or whatever. You probably won't reach any real conclusion (or do nearly enough work to justify moving to another DB for a production environment), but the effort will very likely get you very familiar with how it works, how to set it up, how to admin it, its performance, its quirks, etc. That's both a good and a bad thing, BTW... :-)

-B

no docs?

Sadly, the docs are not free. Now, I'm not really whining about shelling out the $40 or whatever for the 'interbase handbook' or whatnot; my complaints are these:

Timeliness. Programs CHANGE! Open projects change QUICKLY! You must have open docs, or at least docs not committed to paper, to revise them.

Sheer quantity of docs! Ever used Oracle? Oracle8: The Complete Handbook is not all you need! You inevitably end up buying the whole goddamn bookshelf! RBDMS's are complex beasts! With that in mind, closed docs means that such a project can't be called 'free' after all.

I guess it's 'Raymond Open' not 'Stallman Free'.

Re:details?

However, that sort of licensing term is unlikely to stand up in court. It'd be nice if someone with at least half a testicle would stand up and tell the world that the terms are bullshit and most likely illegal.

I benchmarked Oracle and Microsoft-SQL against one another for box weight (that is, how heavy the software, packaging, and associated manuals are) and found that the differences are scale-dependent. Overall I found that Oracle was heavier.

Go ahead, guys, sue me. Good luck; you'll need it.

There's no such thing.

There's no such thing as free speech. Haven't you noticed that the First Amendment to the Constitution reads:

Congress shall make no law [...] abridging the freedom of speech,

except in those cases where it is deemed to harm corporate profits, [...]

Libel, slander, copyright trademark or patent violations, licensing agreements, saying posting or printing anything that someone with more money and more lawyers than you doesn't like...Free speech is a dream. --jb

details?

i didn't see who the leading proprietary databases were that they competed against....anyone else find some details?

also, i'm all for postgres, but doesn't it seem funny that their business is based on postgres solutions, and now they come out with this "independant benchmark" claiming that postgres is the best?

This could be another attempt at "benchmarketing" ;)

-Doviende

"The value of a man resides in what he gives,
and not in what he is capable of receiving."

Interbase

Interbase fits right into a Linux environment.
Borland has just recently released its source code [borland.com] and so what we have now is an open-source, royalty-free, Borland-quality database to use and abuse.

Download links are:
Client and server source code [conxion.com]
Server Linux binaries [conxion.com]

From personal experience, Interbase is perfect for a tight budget situation where you need to server a medium-size userbase.

--
Kiro

Re:Let's hear it for lie, damn lies, and benchmark

Ahh, it hasn't taken long for the MySQL wwenies to crawl out of the woodwork.>[?

They compared the bleeding edge postgres (7.0)

PostgreSQL 7.0 is the current production release of PostgreSQL. Try getting your facts right.

--
My name is Sue,
How do you do?
Now you gonna die!

MySQL

The reason MySQL was slower was because they used the ODBC drivers. The MySQL ODBC drivers are known to be significately <sp> slower then the native drivers. Although I do agree it is probably the best way to test a large number of databases using ODBC drivers, I would like to see the results of all the databases tested using native drivers.

-Tripp

why aren't these on TPC.org?

Was it not www.tpc.org [tpc.org] that ran these tests? In that case, I don't put much stock in them. Take a look at the top ten results by price/performance [tpc.org] or even the complete results by database vendor [tpc.org] shows no mention of Postgres. I'll believe it when I see these results ratified by the TPC.
---

Drivers are important too

Something I think many people, benchmarkers included, forget is that the drivers are important too. What I mean is, if I am going to use a piece of hardware/RDBMS/etc... then what drivers I can use with it are going to be an integral part of my overall experience with the product.

If one DBMS/3D Card has better drivers, even though it is "slower in theory", then that means that my overall experience will be a better one than the "theoretically faster, but with crappier drivers" product.

What does this mean? Trying to equalize products on drivers is often an exercise in finding which product has the most tuned driver. IN the case of Postgres, it appears their ODBC driver is tuned much better than the others. However, very few people I know use ODBC drivers for MySQL, and not many use them for Oracle either. They all use the native drivers. Thus, this benchmark doesn't mean anything to them, because a non-real-world situation was benchmarked.

I wish people would perform real-world benchmarks: i.e., run what people would actually run. That's one thing that I really like about the gamer-oriented hardware review sites. They post a bunch of meaningless BusinessMark2000 and AppMark2k scores, but they also go in and show you how fast the actual games will play on the hardware. That is CRUCIAL to my purchasing decisions. RDBMS vendors should benchmark one database with its best-performing driver vs. another database with its bets performing driver. Then we could really get an inkling of an idea as to how the thing will really perform in the field.

Testing with the same drivers only looks fair; in reality, as in this Postgres benchmark, it was likely the deciding factor to making Postgres "trounce" the competition.

Let's hear it for lie, damn lies, and benchmarks.

A few comments... The most noticible and glaring issue is the "independant study" was (surprise, surprise) commissioned by Great Bridge, and great bridge's reason for existance is to sell support and services for postgres. Not the strongest indicator of impartiality. The entire press release is designed to sell postgres, not to provide a fair comparison.

Issue two. They compared the bleeding edge postgres (7.0) with the old-as-heck mysql (3.22) - they're now up to revision *22* of the development series for mysql - that's a pretty huge amount of changes. I would have been much more impressed with this if they had ran the comparison between 3.23.22 and 7.0. As with everything, folks, don't believe benchmarks, especially ones in press releases. Believe real-world tests. I've used both, and I'm using MySQL 3.23.22 for my site.

Re:if Postgres is so FAST...

If you take a look here [postgresql.org] you'll see there's one in the works. However, the author has been kind enough to post the book here [postgresql.org] :) -- Mando

Re:2 leading commercial databases

Why do you suppose the "2 leading commercial databases" were never named? It would be interesting to know what they were

The article points out that the companies prohibit publishing benchmark results when you buy their product.

those unmentioned proprietary dbms...

everybody wants to know what they are. one has to be MS SQL server... why else would they even have mentioned NT server? every other database will run on linux. the other is probably oracle, just given ned lilly's discussion of them on the postgres mailing list, although i wouldn't be surprised if it was something else. here's a reference to a message from ned:

http://www.postgre sql.org/mhonarc/pgsql-general/2000-06/msg00390.htm l [postgresql.org]

Re:Yes MySQL now does transactions

t was added a while back. And still, mysql (when using MyISAM) is a lot faster than competing databases. Sorry I've done my independant benchmarks in the "real-world".

Too bad the transactions only work with Berkeley DB tables (not MyISAM).

I tried MySQL again last week to see if I could certify it for use on a failover cluster (logging and transactions are VERY important for that!). I didn't even get to the part about simulating a node failure before it flunked the test (which postgreSQL passed). I found that sometimes it would silently ignore BEGIN and ROLLBACK. That may be a Beta issue, so I'll look again when it's released. If the version in use doesn't have the support for Berkeley tables compiled in, it will silently create a MyISAM table instead. Then, it will silently ignore any transaction commands (BEGIN, COMMIT, ROLLBACK).

It's be one thing if it simply failed with errors returned, but silent failure to use transactions is the stuff nightmares are made of! I'd actually rather have it core dump than do that.

Re:Good book on postgres?

There is a PDF for the upcoming Addison-Wesley book, by none other than PostgreSQL's Bruce Momjian, available here [postgresql.org].

Re:those unmentioned proprietary dbms...

Er... I really can't comment, other than to say that it's not IBM, Informix, or Sybase ...

Ned Lilly
VP Hacker Relations
Great Bridge