MySQL to Adopt Solid Storage Engine

hmart writes "As seen on ZDNet's Open Source Blog MySQL is taking another step to defend from Oracle's recent acquisitions of InnoBase and Sleepycat. From the article: 'MySQL responds by getting Solid Information Technology, a proprietary database vendor, to take its solidDB Storage Engine for MySQL open source, under the GPL, starting in June.'"

None

[mrnobo1024]

No matter which storage engine you use in MySQL, the SQL syntax is still the same. All you have to change is the ENGINE= line in your schema definitions.

Solid has good stuff!

[sgifford]

We used to use a Solid SQL database a few years back, before I really knew anything about databases or SQL. It was extremely reliable, fast enough for everything we did, and very easy to administer (even for people who didn't know anything about databases or SQL). I haven't used it recently, but if their technology is still top-notch, this will be very good news.

well, not entirely

[Reality Master 201]

You might adopt different strategies in code to deal with storage layers that supported different levels of transactionality or locking atomicity.

Hooray

[brennz]

As Mysql continues to catch come towards the Mid-tier in the market, Postgresql is charging at Oracle.

The entire Database market is heading towards OSS.

Re:More Info?

[jadavis]

can anyone point me somewhere which describes the deliniation of responsibilities between generic MySQL code and the storage engines?

Interesting question. However, MySQL does not completely abstract the storage engine. Different storage engines have different semantics at the higher levels. For instance, MyISAM does not support transactions, while InnoDB does.

Something I blindly stumbled over...

[Frosty Piss]

Within two clicks of the Slashdot story page, I found THIS [solidtech.com].

Re:well, not entirely

[jadavis]

There are actually many differences. Full-text indexes (FTI) only work with MyISAM, for example.

If you change storage engines, you certainly need to examine all non-trivial MySQL code.

With MySQL, it's often hard to get all the features you want at one time. In the case of full text indexes, you can't also have transactions because only InnoDB has transactions.

Re:Full text support

[mthreat]

I'd suggest checking out some of the fulltext search libraries if you want flexible, fast fulltext search. It may not be as easy as using the database's built-in support, but the extra effort may pay off in performance and extensibility. For starters, check out open source Apache Lucene [apache.org], which has ports to several languages, including Python, C, and .NET (C# I believe), and the original in Java. It's easy to get up and running quickly, and designed well enough to be customizable later.

Re:This is good news

[Antony T Curtis]

Oh, I doubt that there is a single MySQL developer who would be offended. We are not so aloof that we cannot take criticism - actually we welcome constructive criticism, it lets us know what needs to be improved.

Perhaps we have been a bit lazy: Assuming that the partnership with Innobase would continue forever and that MySQL would eventually acquire Innobase.

MySQL has a few advantages - it is a small company which can react quickly to unexpected changes. We learn. Jim Starkey has joined us - he is a veteran of relational database designs and a delight to work with - and we do plan to have our own storage engines with transactional goodness, we already have Cluster but we need something more lightweight usuable by everyone.

In the meanwhile, it is good to take advantage of the MySQL architecture and make partnerships which can make use of one of our strengths. Having 3rd party storage engines is something which has been quietly encouraged for many years and is now something I think will see more publicity.

(disclaimer: opinions are my own and not of my employer nor anyone else)

Re:This is good news

[Anonymous Coward]

I was very surprised when I was forced to switch from Microsoft SQL 2000 - MySQL had a much smaller memory footprint, and the performance just blew me away. To this day I still cannot believe the performance difference. It really speaks to the power of Open Source.

No, it speaks to the power of ACID database integrity. If you don't worry about real database stability, then you can get away with a lot of performance enhancements. ACID compliance is achieved by writing extensive logs before, during and after database operations. MySql has far less overhead, that's why it's faster, not because it's open source.

That said, there are some applications where the lack of stability isn't an issue...

Re:Here's an idea.. . Develop your own!

[Antony T Curtis]

Hmm.. I'll bite.

(1) "MySQL has many applications written for it"
* Not if you enable strict mode, or if you use storage engines that don't support the features you need.

If your application is designed for MySQL specificially without strict mode, then strict mode is not for you. It is more useful to people porting their applications from some other product to MySQL than to the existing MySQL user base.

(2) "MySQL is optionally SQL compliant with strict mode"
* Only if you want to forego 99% of the existing MySQL applications, and start fresh.

Ditto above. Perhaps the existing MySQL user wants to go to strict mode in order to support databases *other* than MySQL. Have you concidered that?

(3) "MySQL has transactions"
* Not in MyISAM, which means no full text indexing.

MyISAM was never designed with transactions in mind. It performs its intended function excellently. Not all data is useful to keep in some kind of transaction context - take for example a table mapping UPC codes to product names and descriptions: it will never need to change in a transaction so having transactional overhead would be wasteful.

(4) "MySQL is free"
* Only if your application which links against the client library is also GPL.

Myth used to scare people away from opensource GPL code.

(5) "MySQL is fast"
* Only if you use MyISAM, which means no transactions or many other features that aren't available in MyISAM.

Some people would disagree with that statement. Quite a lot of people have found that for their needs, MySQL is lighter and faster (or at least, as fast as the big name products) even when never using MyISAM tables.

There will be people whose needs are best served with products from Oracle, IBM, Microsoft, Sybase... I am sure that no one in MySQL claims that it is the only solution to all problems: MySQL is a young database product which is actively being developed so it will improve.

Re:well, not entirely

[Antony T Curtis]

With MySQL, it's often hard to get all the features you want at one time. In the case of full text indexes, you can't also have transactions because only InnoDB has transactions.

You know, it really depends.... If you are creative, you can have your cake and eat it too.

The most straightforward way would be to create a MyISAM table with the same primary key as the InnoDB table. If the commit succeeds to the InnoDB table, you can safely add the row to the MyISAM table and have your fulltext indicies there. Do something similar for deleting rows and for updates if neccessary.

It is not rocket science.

Re:Why did they do this?

[ZeekWatson]

Why? InnoDB was already GPL'd. Why not continue to use and develop it?

Because mysql sells a private version of their DB server. They can't do this with GPL'd code in it. They had a separate (non-GPL) license aggrement from Innobase Oy to use the InnoDB engine.

Re:This is good news

[ocelotbob]

SCO's still somewhat big in the medical profession. Being one of the first "cheap" unixes, a lot of small medical offices went to SCO in the early 90s during the first big rush to digitize patient charts, etc. I've got a buddy who deals with a lot of SCO installs on a daily basis. It's still used because though SCO is a pain in the ass, migrating away from these recordkeeping apps would be even more painful.

Re:I hope they've got a perpetual licensing agreem

[Antony T Curtis]

There will be no need to pressure existing users off from InnoDB .... InnoDB is GPL code and as such will always be available: This is the strength of open source software, it is very hard for players to pick up their toys and leave the sandbox. If they leave the sandbox, they have to leave their toys there. Besides, Oracle has stated that they do not intend to change Innobase's business. Lets hope that they keep that promise.

Of course, MySQL can encourage people to use a particular product choice based upon future roadmaps but I don't imagine that InnoDB support will ever be axed.

(opinions are my own, not of my employer nor anyone else)

Re:and how many....

[kestasjk]

Perhaps if your SQL is trivial and/or you're writing with portability in mind then abstraction is a good idea. If you're doing anything with transactions, locking, indexing, anything which requires the use of SQL which isn't available or identical on all databases then you can't (fully) abstract. If you're not writing with portability in mind abstraction is a waste of cycles and, more importantly, development time.

Re:This is good news

[swillden]

The problem being once these important OSS coders see oracle source, they are tainted. Any further OSS contributions will have to be heavily scrutinized for IP violations.

Yes and no.

Strictly speaking, there is no such thing as a "tainted" developer. There are three different forms of IP law that Oracle might try to use to control what the programmers do after they leave the company:

1. Copyrights. Copyright only covers the specific code, not the ideas. A developer could learn how to do something while working at Oracle, and could then leave the company and implement the same thing and it wouldn't constitute copyright infringement as long as the second implementation was done from scratch. Technically. In practice, if the resulting code was extremely similar, the developer might have a hard time *proving* that he didn't take a copy of Oracle's code with him when he left.
2. Trade Secrets. Oracle could tell the developers that things they're working on are trade secrets, and make them agree not to disclose them. That only applies to the specific ideas and techniques that are designated as trade secrets, though, and only applies as long as they're secrets that haven't been revealed elsewhere.
3. Contract. Oracle could make the developers sign non-compete agreements. Those can't last very long, though, and tend to be hard to enforce.

So, if the developer has access to a good lawyer, it seems like Oracle really can't do much to prevent future work. But, of course, a wealthy corporation can afford legal battles much better than an indiviudal can, and any or all of the above would lend enough credibility to a lawsuit by Oracle that the coder might end up having to fight it in court... which can be really expensive even if you win. Then again, this is exactly the sort of case where the EFF and others would probably be willing to step in and help out.

Finally, if Oracle failed to terrorize the devs into meekly going along, Oracle probably wouldn't want to go to court very much. Not only is it bad PR, but it does cost a lot of money, even if you're a corporation with deep pockets. There has to be a reasonable expectation that the cost is worth the result, which seems pretty unlikely. Threatening a lawsuit you probably can't win doesn't cost much, but actually pursuing one does.

So oracle does not need to keep them. They just need to expose them to even bits of their db source, and they have tainted the coder.

No, I really don't think that's enough. What they really need to do is to terrify the coder with weighty, frightening non-disclosure agreements and non-compete agreements. If they do too much of that, though, they won't get the hire in the first place.

Of course, IANAL. But I have read the relevant law, and have some familiarity with the applicable precedents, thanks to other cases.

I really hope that the open source developers who've gone to work for Oracle have run their employments contracts past attorneys of their own.

Adopt? No!

[Khanom]

There is a world of difference between "MySQL to Adopt Solid Storage Engine" and getting "Solid Support", as in, from TFA: "...getting Solid Information Technology, a proprietary database vendor, to take its solidDB Storage Engine for MySQL open source, under the GPL, starting in June".

Solid is just offering YA storage engine that works with MySQL, and they happen to be GPLing it. This does -not- mean MySQL is adopting anything from Solid.

Slashdot editors: please read TFA once in a while and please stop writing misleading headlines.

Good

[Godji]

So MySQL is keeping up to Oracle's increasing power - that's good! What about PostgreSQL though? What are they doing to keep up? Anyone know?

it is solid!

[sad_]

we're using HP data protector backup/restore software, which uses solid as its database for storing all data information.
HPDP itself is as stable as an italian taxi driver stuck behind two priests in a skoda, but thank GOD, the database is rock solid and doesn't corrupt when HPDP goes belly up again.

so all in all, a good pick of mysql!

Re:Good

[adnonsense]

Adding new stuff all the time? Have a look at the website, it's very informative: What's New in 8.1 [postgresql.org].

Re:Article is wrong about location

[Anonymous Coward]

They aren't lying. The founder and lead developer (Monty Widenius) is from Finland, as is the CEO (Marten Mickos) is also Finnish, as well as a bunch of other sales and development staff.

It just so happens that Monty was working for a Swedish company (TcX DataKonsult) when he came up with MySQL, thus the Swedish company that exists nowadays: It's the old TcX, renamed MySQL AB.

One might also note that Innobase Oy (creators of InnoDB) are also from Finland.

Re:Here's an idea.. . Develop your own!

[kpharmer]

> What would be the list of priorities for you for features/changes to be made to MySQL in order for
> you to seriously concider it. Bear in mind that MySQL does not have an infinite amount of developers
> so delivering on all of them overnight would not happen.

Now that MySQL has implemented most of the standard database features, the next step is to harden them and the rest of the codebase. At this point I'm not interested so much in features as I am high-quality & robust implementations.
    - this means that if I add a column to a table I won't have to spend hours copying the data back & forth under the covers
    - it means no more gotchas - with broken date handling, exception handling, type conversions, value truncations, etc.
    - it means that individuals clients shouldn't be able to turn off strict mode.
    - it means that non-strict mode should be deprecated - and gone in a few years
    - it means that the optimizer should be able to handle very complex joins and almost always come back with the very best possible query

Now, *after* the above have been addressed there are some additional features to add. These are mostly oriented towards reporting functionality, where I think mysql is pretty weak right now. My list would include:
      - oracle-style range partitioning: this means that you can using mysql for large reporting & warehousing applications. We're not talking union-all-views here, but the ability to attach & detach partitions, and disregard partitions of data from queries. I often see 10x performance gains from partitioning over indexing in this kind of app.
      - query parallelism: need to be able to split reporting query work across CPUs. These large queries, often scanning a 100,000+ rows can typically get linear performance improvements in my experience from query parallelism.
      - automatic query rewrite: need to be able to automatically rewrite queries in order to hit summary tables if they exist. This is critical for some types of powerful reporting tools, or reporting performance boosts for commercial apps.
      - materialized views: simple method to manage summary tables. Not critical since this can be developed in other ways, but this has become the preferred way to manage summary tables.
      - federation: the ability to redirect a query against a table to another database to be resolved. This allows you to define reference tables on one database, then share them across many.
      - query failover: the ability for the mysql client to automatically reroute a query to a failover server. This allows you to provide failover without any code changes at the application layer.

that's all i've got time to write, but should cover most of what I'd look for first.

Re:This is good news

[Saanvik]

You mean projects like these (see Oracle's development tools page [oracle.com]))

• Apache ADF Faces
• Eclipse EJB3 Tooling Project
• Elipse JavaServer Faces Tooling Project
• Eclipse BPEL Designer Editor Project

Take a look at Oracle's OSS page [oracle.com] for more projects. One of the biggest is OCFS, Oracle's clustering file system which they released with a GPL license.

Sure most of it is to make it easier for them to sell software for Linux users, but it's out there, with various OSS licenses.

Re:Here's an idea.. . Develop your own!

[Antony T Curtis]

this means that if I add a column to a table I won't have to spend hours copying the data back & forth under the covers

Support for this is being implemented but it does rely on the storage engine to know what to do. Otherwise, the fallback is to do it the old slow way.

it means no more gotchas - with broken date handling, exception handling, type conversions, value truncations, etc.
it means that individuals clients shouldn't be able to turn off strict mode.

Good suggestions. I'll try to make sure that they're not forgotten.

it means that non-strict mode should be deprecated - and gone in a few years

This would mean breaking backward compatibility - something that I think will be undesirable. There are some embedded applications which do not benefit from strict mode for example.

it means that the optimizer should be able to handle very complex joins and almost always come back with the very best possible query

This is the "holy grail" of database systems... What may seem trivial to us to decide what query plan is best may be non-trivial to calculate. There may be a large number of optimal paths but the optimum one may be very hard to deduce due to local minimas. Many databases have such large and complex query optimizers that the query plans have to be precalculated and cached. MySQL has a skilled query optimizer team and I am sure they will do great stuff.

oracle-style range partitioning: this means that you can using mysql for large reporting & warehousing applications.

Already actively in development and may be previewed in the 5.1 tree.

query parallelism: need to be able to split reporting query work across CPUs. These large queries, often scanning a 100,000+ rows can typically get linear performance improvements in my experience from query parallelism.

Only available in MySQL Cluster to a limited degree. Tricky to make a general solution but it is something we are aware of.

federation: the ability to redirect a query against a table to another database to be resolved. This allows you to define reference tables on one database, then share them across many.

Already available and in 5.0 tree, although improvements are possible.

query failover: the ability for the mysql client to automatically reroute a query to a failover server. This allows you to provide failover without any code changes at the application layer.

AFAIK, JDBC connector already provides this and I think the .Net connector as well. The C client I think has not implemented it yet but it is something we know about.