Slashdot Log In
How To Implement A Database Oriented File System
Posted by
CmdrTaco
on Sat Mar 30, 2002 01:44 PM
from the something-to-think-about dept.
from the something-to-think-about dept.
ALundi writes "A really great read from Andrew Orlowski over at The Register on how Benoit Schillings and Dominic Giampaolo created the 64-bit journaled and attribute based Be File System. Schillings and Giampaolo discuss a variety of design and implementation issues, including data integrity and file system performance. " Interesting in the context of MSFTs plans to
implement a DB filesystem
in future versions of MS Windows.
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
BeOS file system rocks. (Score:5, Interesting)
If you want some real insights into the OS as a whole, check out the BEOS Bible [amazon.com]. not so good if you want an in depth discussion, but for non-kernel hackers it's a fun read and very informative.
Read the section explaining how their address book works. it's really cool.
Reminds me of the BeOS 5 Bible reading on this. (Score:4, Informative)
If I remember correctly, Be originally used a "true" database backend as the filesystem, but ran into performance issues compared to the R5 fs implementation. I can't help but wonder how many of these issues were largely due to the speed of the technology used at the time.
I suppose it depends on your application, but I know a lot of web-based platforms already use true db backends (Oracle, PostgreSQL) to handle all data storage, representing a filesystem as a hierarchial set of rows in numerous tables. I've written several applications this way, and am currently working on a content management platform which also uses this model. Need to make a change to the filesystem structure (adding attributes, changing the security model)? Just modify the DB structure and you're done, especially with databases like PostgreSQL where you can use the database engine itself for a *lot* of functions (via triggers, stored procedures, security settings, etc).
As more and more functionality is brought into web-based application environments, I can see the importance of "old style" filesystems starting to fade somewhat for a lot of apps. Yes, they'll still be necessary (the database itself has to reside somewhere, obvisouly), but not in the same way they used to be. Just a few thoughts
Re:Reminds me of the BeOS 5 Bible reading on this. (Score:3, Informative)
It's a true (NFS, SMB) mountable filesystem, with all the flexibility that provides, but also the replication and other Enterprise features you want from a database.
Re:Reminds me of the BeOS 5 Bible reading on this. (Score:2, Interesting)
I checked over the specs for this over at the Oracle site... looks impressive! Now, if only you didn't have to use Oracle to take advantage of this sort of thing; PostgreSQL is standing in first place as my database platform of choice for most tasks.
Is there any kind of equivalent work being done in the OSS community, perhaps based on PGSQL? The platform I'm working on for web-based content management is actually headed toward being usable (and will be licensed under the GPL as soon as that happens), but doesn't use the "real" system FS for storage at all. BLOBs are used for file storage instead, and performance seems to be fairly good
I suppose the best way to go is to structure the system such that it can be used for intranet-style doc/file management, in addition to being able to dish up web-based content (whether for internal or external access/use). We'll see how it shapes up, I guess
Hardware has come a long way since the days of BeOS 4 (I believe that's the last version that used a "true" database backend for the filesystem). With the added power we have these days, it seems the old performance issues might be largely eliminated. Now, I'm not trying to portray myself as an advocate of "throwing hardware at a software problem"; to me, it's really more a case of "we can now do things we couldn't do before." This is somewhat akin to being able to do better 3D modelling on PCs due to new capabilities in 3D hardware acceleration.
New FS Engineer at Apple! (Score:3, Interesting)
Does this mean that Apple is finally going to put some kind of reasonably modern filesystem under OS X?
Have they finally seen the true genius behind their own iTunes interface?
Have they finally realized that they will shortly be THE ONLY operating system that still relies on file extensions as the primary way of identifying files?
I truly hope that this snippet is as wonderful as it sounds, as it may finally restore my faith in Apple, as well as cure me of my unhealthy Debian and XFS addiction.Re:New FS Engineer at Apple! (Score:3, Insightful)
You gotta love how every Unix/Linux/Windows user now talks about how file extensions are so bad. A few years ago these same people used to say how crap Apple's idea of meta data was, and that file extensions were better (cross platform, etc, etc). The arguments that I've had with Unix people at different places I've worked. It seems to take the rest of the world at least 10 years to catch on to ideas...
Re:New FS Engineer at Apple! (Score:2)
Not in a modern interface, it isn't. And it's also far too easy to screw things up; a file's association shouldn't change just because its name does. Storing immutable data like association in a mutable location like the filename is a hack that should never have been used. It's a great shame to the industry that it ever was.
But then, that's Microsoft for you. Pushing inferior standards onto everyone else.
Re:New FS Engineer at Apple! (Score:3, Insightful)
Ugh, don't get me started... Here's a short list:
- Only a single piece of metadata can be stored (filetype)
- Combining data (filename) and metadata (filetype) kind of belies the definition of metadata in the first place
- It can get very confusing to see things like file.txt.rtf
- The tendency of systems to want to hide the extension, and the resulting confusion when your FTP client says the file is named stuff.zip but your interface just says "stuff"
isn't it easier and faster to look at the filename than metadata and file contents??Only if the metadata is stored in some other inode.
In the case of BFS, the inodes are always 1024 bytes, but the inode information is only about 300 bytes; that meant that BFS had 700 free bytes in the inode--free in the sense that they don't take up any more space, but also free in the sense that you don't need to do an extra seek or an extra read to get them. Now that's free! You can go about 700 bytes, but then it needs to allocate more blocks (I don't know if that requires another inode, though).
Also, in this case, even if it is just slightly faster to look at the filename, you lose every other possible metadata feature, just to get an essentially unmeasurable increase in performance. I'd take BFS and all of its features at half the speed if I could use it on any OS I wanted.
Dominic's Book (Score:5, Informative)
AD doesn't yet have wide acceptance, DBFS doomed (Score:2, Informative)
Re:AD doesn't yet have wide acceptance, DBFS doome (Score:2, Insightful)
Damn right they are! For good reason too: it's cranky and fussy and likes to corrupt itself. When the school I went to threw the Microsoft Official Courseware labs out the window because they were impossible to implement, I got my first taste of why AD as it stands is pretty much useless.
If Microsoft had stayed standards-compliant with open standards like LDAP and Kerberos 5 and so forth AD would be much less of a nightmare than it is now. But no, typical MS, they had to "embrace and extend" it. As a consequence, they have shot themselves in the foot.
This is the reason why most MS shops hold desperately on to their NT4 PDCs even though 2K has NT4 beat nine ways to Sunday. 2K cannot do the old-fashioned SAM-based domain even if you cajole it, beat it about the head and shoulders, or ask it nicely. And for most shops, that kind of domain is all they need.
Of course if they went with Samba they could decommission their old fugly NT4 PDC, heh heh...
Re:AD doesn't yet have wide acceptance, DBFS doome (Score:3, Insightful)
Database-like (Score:5, Insightful)
XFS [sgi.com] is also "database-like". But BFS seems to be rather more ambititous an effort -- and very intriguing.
This is one of several BeOS features that the Open Source community should reall consider stealing. But let's consider these features individually, with one eye on whether they're likely to achieve acceptance outside the ranks of BeOS enthusiasts. Let's not waste time on wholesale BeOS clones and compatibility layers. Those are exercises in denial. BeOS was a nice piece of work, but it's as dead as CP/M. Deal with it.
Re:Database-like (Score:2)
Yes, BeOS is dead. Long live OpenBeOS. [openbeos.info]
Software Necrophilia (Score:2)
Look, there's a lot of good stuff in BeOS, and a lot of us would like to see it preserved. But it's the technology that's worth preserving. The platform is just a way of delivering the technology. If you insist on totally re-inventing the platform, either as an OS or as a compatibility layer, you force all your potential users to start over completely. And they just won't do it. You want proof? OS/2. DR DOS. NextStep. AmigaDOS. All of these were impressive products. I personally would prefer any of them (or BeOS for that matter) to Windows. I might even prefer some of them to Linux. But like most users, I have to concentrate on platforms that have a real user base.
wrong way round (Score:2, Interesting)
Simple joins, and most of them are can be replicated with links if necessary. Almost all the databases I've seen would lose little from moving out of the DB and into the filesystem.
It doesn't scale to complicated joins and huge datastores with complex triggering but for most stuff it simply isn't used.
Too many developers have the mindset of placing tree based data into RDBMS which adds complexity.
Re:wrong way round (Score:5, Interesting)
Simple joins, and most of them are can be replicated with links if necessary. Almost all the databases I've seen would lose little from moving out of the DB and into the filesystem.
But then, you lose all the advantages of database journaling (not just integrity, the ability to rollback to a previous state, if necessary), consistency (you have to make sure the files can't be accessed by other operations before the commits are done) and replication.
If I were building an application today with Oracle, I would be very tempted to use iFS for these reasons.
Parent
Re:wrong way round (Score:2)
Tis true, again for the majority of dbases this simply isn't a problem anyway. Week after week I am presented with projects that have "we store the data in a MySQL database" when there really is no operational need for such a system.
Plan9 and Hurd with synthetic file systems and filters also present an extension to the simple disk based file system. I'm not for a minute suggesting thrwoing all that indexing and query optimisation away where it's needed.
Rememebr the ask slashdot a while ago about "how can i keep track of my photographs". People were seriously suggesting Oracle and MySQL. All of his data would fit on a floppy and proably into the memory of my microwave oven or car stereo!
Re:wrong way round (Score:2)
Surely thou jesteth! open() has to be one of the most expensive unix system calls! What you're proposing would move slower than cold molasses.
C//
Possibilities (Score:3, Insightful)
My fear, and I think the reality is that Microsoft will not be so kind.
jrbd
Need support for versions and schema changes (Score:2)
A DB as a file system could still be a mixed blessing though - anyone tried to store code in a database, or other files with lots of different versions that may have different structures? Generally, DBs are weak when it comes to namespaces (like directories), versions (except in some special cases features for time series) and 'schema evolution' (changing the data structure).
Nothing impossible though - I think Oracle were getting there with the Internet File System, so would be nice to have a PostgreSQL FS in Linux to start playing with!
Problem is that file systems don't care about data structure and so don't care about changes to that structure, so you can have half a dozen different versions of your address book floating around
I prefer object-oriented graph filesystem (Score:3, Interesting)
- need much less overhead than database system
- can hold multiple ways to access one object, easing semantic link
representation
- can be secure by allowing only the workflow links.
It should be organised as a set of object that hold data within them,
with links between them.
Leveraging a DB FS (Score:5, Interesting)
One could provide a helper app that allows you to look at the stored files in ways other than your typical file listing. To do this would require various metadata attributes to be associated with the data. Helper apps would be provided for all of the standard applications that read and write to data to the file.
For example, one app could set attributes to categorize data. One could then search for data on your system, and potentially others, much in the same way as you would search for stuff on Yahoo.
Say you are in a rush to finish your taxes but you need to put together an itemized list of business expenses. You have information on them stored in various places including text files, e-mail, spreadsheets, etc. You could use find and grep to go poking around looking for them or you could use a helper app that does a quick search of attributes and presents you with a list of candidates and ca even call up other apps or services to look at the data. Once you have identified the data another attribute can be set that your tax software uses to record it and pull it into your tax forms.
Re:Leveraging a DB FS (Score:2)
mySQL filesystem (Score:2, Interesting)
Has anyone actually tried it?
I'll wait for SP2 (Score:2, Insightful)
Even more than normal Microsoft bashing, this sounds like a huge challenge for MS to get right. I can't imagine that they'll manage to retrofit a DB filesystem and make it peform adequately on the first try. It would be tough enough even if they started with a clean slate and a small, independent team of top talent. Unfortunately, although they do have some first-class developers, they also have tremendous legacy baggage and a group-think culture.
I'll wait for at least Service Pack 2 before I put any real data on an MS DB filesystem.
Unix lags (Score:2, Insightful)
Linux is a nice remake of a legacy os, but is hardly the future.
The open source community needs a good object storage to base a more futureproof os on. Badly. (And a way better UI than XWindows can give us.)
Re:Unix lags (Score:4, Informative)
No, you don't.
Be used the following method of determining which app would open a file:
Notice that nowhere in here does it take into account what app created a file.
Even better, you can always right click on any file (or any list of files of the same type) and get a list of:
- The preferred app for that file
- The preferred app for that filetype
- Any apps that can open that filetype
- Any apps that can open the supertype
- Any apps that can open any filetype
And the list shows up pretty much instantaneously. Do that in any other system, anywhere, no matter how many hacks you add. And yes, you'd use it every single day of your life if it were available to you.So no, I'd say that you don't understand the problem.
Parent
Re:Unix lags (Score:3, Insightful)
OpenBeOS BFS (Score:3, Informative)
They have a fantastic amount of the work done [sourceforge.net] already such as
o Read-only BFS
o Kernel Interface
o Full Attribute Support
o Indexing
o Symlink Traversal
o Queries, full UTF-8 Support, and support for non-indexed attributes.
I believe they are also fixing some problems that were in the original FS.
I am sure they would be glad for some more file system engineers. Come to think of it, the rest of this open source project is going really well too, but as always it needs more programmers....
Re:OpenBeOS BFS (Score:2, Informative)
Practical File System Design (Giampaolo) (Score:2, Informative)
I used to like this idea. (Score:3, Insightful)
Not much user, lots of system and iowait, that's what. We run into a whole new realm of needing accounting for these kinds of things.
my two cents (Score:3, Interesting)
If the data is shared, and you have libraries that are shared, then why not ask the data to display itself (object.display(x);) and have it call to a standard library (system library?) which queries a system properties database object as to what application to display it? Don't actually store the display code in the objects, but have the objects query the system as to what the user has specified to display that type of data with.
Dominic: That's what I mean. Some people are very anal about organizing things in rigid hierarchies and others are 'I know what I want to find'.
I think there is a place for hierarchies, but not as the base organizational method of the filesystem. I would like to see a hierarchy of attributes, or keys, or whatever you want to call them. When you save an object (off the internet, or out of your head), a title is only one possible attribute you want to give it. When I save a pr0n jpg, it doesn't need a damn title, I need to mark what it's a damn picture of (amateur AND cumshots AND redhead)! Perhaps start with people, places, things. Or later in the hierarchy, sound -> music -> various bands as well as various artists as well as various sound effects as well as dates and live or studio, all keyed (so to speak) and queryable. But the hierarchy is for browsing. Just for browsing, because browsing is important (when you want to look at cumshots, you want to look at cumshots, but when you query for cumshots, watersports and lesbians, well that's bloody well what you should get), and micro$oft's nice little explorer looks about right. Although instead of a stupid directory tree, we have a tree of object properties and types, and any object can be in any number of places in that tree, depending on it's attributes (categories?).
I know of course that I haven't really said anything new in this post, and I know that performace needs to be taken into account. This is, however, the way things are moving, and all we really need is a really good, really fast, solid state storage medium. When permanent storage is as fast as or faster than RAM is today, the database filesystem will finally become a reality, until then, we'll sure be gearing up.
Cheers, Joshua
brute force (Score:2)
There are brute force ways of doing it: you build some kind of ad hoc database system and dump it into kernel space. You may be able to engineer such a system reasonably well, but to me, it is in bad taste: indexing is such a complex and application dependent area that nobody can guess ahead of time what kind of indexing people will want a few years from now. The Be file system looks like it's too complicated to interoperate well, and too simplistic to be of much use for anything rather than fairly primitive indexing operations.
A better way of doing this is to figure out a protocol for notification and updates between a traditional file system and user-space database indexing services. Yes, that's harder, but that's what software engineers get paid to figure out. And, as far as I'm concerned, if you can't figure out how to do it right, it's better not to do it at all rather than doing something half-baked.
Threat to Open Source? (Score:2)
Re:Threat to Open Source? (Score:3, Interesting)
Functionally, database-based file systems are an old hat. If they were the magic bullet Microsoft and BeOS think they are, they would have caught on long ago. To me, it looks more like a bunch of college hackers getting mightily excited about a whizbang feature of little real value. (Database based file systems have worked well in some niche markets--IBM is selling some systems with such file systems.)
Something needs to be done about indexing and search, but putting a database into the kernel is not the right thing.
Microsoft is NOT putting SQL Server into Windows!! (Score:5, Interesting)
As the ExtremeTech article pointed out, they are not even considering putting the full-blown SQL Server into Windows. SQL Server is too resource-intensive (it really wants to use all of the available CPU, memory and disk space), too much overhead, and most importantly to MS, too profitable (sales of SQL Server / BackOffice make up about 10-15% of MS's revenue.) There's no reason to bundle it if people are willing to pay a ton for it separately!
As the article says, they're thinking (nothing decided yet) about including MSDE, which is exactly the same as SQL Server 2000, except it is tuned for 5 concurrent users (and hard-limited to 10), the database size is limited to 2GB per database (the same as the Jet DB, aka Access), and it doesn't have the nice GUI admin tools bundled.
Also, the OFS (Object File System) discussed previously probably won't get added either. There's a good reason why it was talked about way back in the Cairo (pre-Win95) days but never implemented - it's really, really hard to do, and it's hard to even convince anybody of its value. (Just look at Be.) Active Directory was originally supposed to be an object store, but I don't think anybody uses it for that (if anybody even uses it at all.)
What probably will be included is an improved version of Indexing Service, which is currently included in Windows 2000 and XP. For those of who are fortunate enough to be unfamiliar with Indexing Service (formerly Index Server), it's an NT service (think "daemon") that periodically scans the file system for new / updated files, and then adds whatever metadata it can extract into a database of sorts, which is then used to speed up searches in the built-in Search dialog on the Start menu.
There are a couple of problems with the current implementation:
So, in summary, MS's plans for the DB-in-the-filesystem look a lot more like Reiser4 [reiserfs.org] than like BeFS or SQL Server.
An Earlier Example of a DB Filesystem (Score:2)
PICK has an SQL-like language to create reports and search and select databases. The data structures are kept in a "dictonary" file for each file. Records are variable length and seperated by upper acsii characters. (254 for fields, 253 for values (sub-fields), and 252 for sub-values.)
PICK is pretty much considered a "legacy system" by most people any more. (If they have heard of it at all.) It had some features that were far ahead of its time. Unfortuantly, Dick Pick, the creator of the OS, was unwilling to improve on it. So PICK remained in the dumb terminal world, while everyone else moved on. The vendors of PICK made improvements over the years. It was the only thing that kept it even vaguely current.
Microsoft is claiming that you will not need datastructures for their new system. DB filesystems are very dependant on embeded filestructures. It has to be there. I have no idea how they will be able to take a structure as complex as Word or MPEG-4 and make it "transparent" and portable. More likely you will be held hostage to their OS. It will be portable to "upgrades" of the OS and no more. (At least until they make that data format no longer supported...)
Integrated database computers: IBM AS/400 (Score:5, Informative)
IBM's AS/400 (a midrange computer system targeted for commercial use/accounting/warehouse/etc...) is based on an object-oriented database filesystem which is implemented at the firmware level (SLIC) rather than at the OS-level - and this system has been around for about 20 years and IIRC it always had quite good performance.
-arch----
A few words about its architecture, if you're interested...
The operating system (OS/400) itself runs on top of this object-oriented low-level "OS" by calling its APIs - as a result, most parts of OS/400 are platform-independent. If you'd manage to get the SLIC running on another hardware platform, you could probably install a nearly unmodified version of OS/400, and it would do its work.
Actually, I'd call the SLIC code the 'real' operating system kernel rather than OS/400, because OS/400 itself would not work without an apropriate SLIC layer.
Everything on the system is an object, so you'll always have to use the object's methods to perform some operation.
For some applications that may be an advantage, because security is enforced on each object at the firmware level. For other applications it might also be a disadvantage, because you'll always have to use a limited set of APIs for modifying data. That blocks many methods commonly used for writing highly optimized code.
-end arch----
One of the benefits of having a database-filesystem is probably the fact that you do not need to run a database product on top of the OS.
Every object on the system can be backed up and restored in a very simple way. Logical files (multiple logical views of one physical file) can help to keep data management simple and consistent.
On the other hand, you will have to update the entire OS (including the kernel) when you need to install a new release of the database - which means, that you'll have to reboot the machine.
And - last but not least - the more code you have in the OS kernel, the higher is the probability of having dangerous bugs somewhere in the kernel.
It should not be necessary to mention, that bugs in the OS kernel may compromise all system security.
There are certainly many advantages and disadvantages regarding the database-filesystem issue, so I think it all depends on what you want to do with your computer.
-----
kind regards from Austria,
octogen
PS: i hope my english isn't too poor..
And - by the way - even Microsoft uses AS/400 boxes for running its business, so what do you think, where did they get their inspiration from...?
Re:Integrated database computers: IBM AS/400 (Score:3, Insightful)
This guy has it right. I have been working AS/400's for YEARS! The OS is built around the DB2 database. And IBM beat MS to the idea a long time ago. But, MS likes to make everyone think they actually "Innovate" things. When in fact they don't! They buy up companies, steal technoligies etc... The only thing MS has perfected is the marketing engine. And yes it is true MS uses AS/400's. In fact MS wanted me to come interview to work on their team that did the MS--> As/400 integration work. (I told them to blow me, I wouldn't work for a company I dispised!).
Dominic is at Apple! That rocks! (Score:2)
Re:Dominic is at Apple! That rocks! (Score:2)
Dominic, we are delighted to learn, has subsequently joined Apple as a file system engineer.
This is the best news that I've had in a while on the Mac OSX technical front.
Allright, Dominic! Get in there and kick some ass!
Re:We already have a "database" file system... (Score:3, Informative)
It uses B-Trees which if you RTFA is thought not to be a very good solution, but it was revolutionary for its day.
The idea of having a resource fork to all files, a little name badge for every file that tells file type and creator type, is tremendously helpful too. However, the resource fork on Classic MacOS files tends to corrupt at the drop of a hat, rendering this little innovation, also introduced with HFS, fairly useless.
Oddly enough, however, I find good ol' MacOS fairly robust...so much so that there is no need to reformat and reinstall on a regular basis. Its memory management sucks, but aside from that glaring problem it's a beautiful operating system. I have PCs running Windows and PCs running Linux here at Catseye Labs, but it's my good ol' G3 that I do most of my work on.
Re:We already have a "database" file system... (Score:2)
In my opinion unless there is a trivial method to reduce ALL the data of a file to a single stream of bytes that can be read/written with Unix-style calls, this is doomed to be a failure and a return to 1960's "pip" programs. MicroSoft would love this (it would quickly make all file transfers proprietery) but it would be a disaster for all software development.
Unfortunately I see no sign that anybody is going to address this. Even older Unix's have "attributes" like name/time/date/owner/permission and there is no way to communicate these without the program knowing about each of them and making up it's own encoding.
Re:If its so easy . . . ? (Score:2)
While it's great for performance, it changes how people understand filesystems. This isn't necessarily a bad thing, it's just a change.
Re:If its so easy . . . ? (Score:2)
SGI's XFS uses and agressive advanced cache system and pre-allocates sectors. What this means is that small files, like temp and working files, never actually make it to the physical disk. They are cached and used from the cache. If they are short lived, then no disk I/O occurs at all.
Pre-allocating sectors means the system can study where the best location for the file is before it actually gets written. This not only minimized fragmentation, but it also speeds up writes.
Re:Noooooo! (Score:3, Informative)
BeOS began development in 1991. the first iteration of a DB backed file system came out with BeOS 1.0 and BFS replaced in around 1994ish.\
soooooooo.......MS again is a day late and a dollor short.
same with MATURE Office productivity products, since workperfect was mature long befor word 1.0 was released, as was lotus spread sheet.
perhaps they did good work on the OLE crap, but that stuff sucks (imho) and Adobe has much better products to crate desktop publishing work that OLE was intended to help with....not to mention that word still can not do what TEX can do.
Re:Noooooo! (Score:2)
Re:it was another story a few weeks ago (Score:2)
Advice to the victims. Backup early. Backup often.