Catch up on stories from the past week (and beyond) at the Slashdot story archive

 



Forgot your password?
typodupeerror
×

Comment Re:Keep it simple... (Score 0) 209

Wow. Insults hurled from the peanut gallery are strong this evening. I can assure you that I definitely know the difference between a database and a file system. I spent years writing low level disk utilities (PartitionMagic, Drive Image, and Drive Copy) as well as lots of experience fine tuning Postgres databases. At no time in the article did I suggest using a database to replace file systems.

Comment Re:Yes it is (Score 0) 209

I started programming Didgets after Microsoft abandoned its attempt to get WinFS working properly. I liked the idea of having a single system that handled both unstructured and structured data and tied them together in meaningful ways (e.g. tagging). If I understood correctly what they attempted to do, they tried to take NTFS and SQL Server and join them together somehow instead of building an entirely new system from the ground up. They abandoned it either because they could not make it work technically, or they decided it would just be a tech support nightmare when problems arose.

Comment Re:Is he arguing against file systems in general? (Score 1) 4

Obviously, we need some kind of data storage manager that effectively manages large amounts of unstructured data. What I am arguing is that the conventions that almost every file system currently has, are outdated and need to be replaced with something better. Whether you call the replacement something like FileSystems 2.0, an object store, or some other similar name is not really important. What really is important is that the replacement system is not just an incremental change that must have 100% backward compatibility; but rather is fundamentally different in how it manages that data.

We need a system that can read a metadata table into memory and do fast searches against it for a volume with 200 million files (or more) without requiring massive amounts of expensive RAM. We need a system where you can easily attach dozens of meaningful metadata 'tags' to each of those files and then find anything/everything that has certain tags in just a few seconds. We need a system where indexing and security are not handled by an optional separate software layer that can be easily bypassed.

"A file system can be designed with variable-sized metadata". That may be true but all the major file systems that I have worked with have a fixed sized record (that I think is too big).

"In many if not most file systems, the file name is part of the metadata." Absolutely true, but file names are almost always stored separately from the file table and must be read separately. For almost all systems, you can't just read the inode (or similar metadata record) and determine if the file is a photo. You have to traverse the directory tree using its full path to find its parent directory; find the file name within that directory; and then compare its extension to known photo types. Even though that may be several disk reads but can still happen in a blink of an eye for just one file; if you want to know how many photos you have on your volume of 200 million files, this can take a really long time.

"For some file systems, the inode...serves as the anchor point for references." When I spoke of 'stored references' I was referring to references outside the file system. If an application stores the path to some file it needs in some config file, the file cannot be renamed or moved without affecting that application. An application cannot store the inode number in its config file since the file may not have the same inode number the next time it is accessed.

Again, I am not arguing that file systems have not served a very useful purpose. What I am arguing is that when the number of files being managed exceeds a few million, the existing systems are wholly inadequate to handle them effectively. This is to say nothing of when the 100TB drives promised by the drive manufacturers become part of consumer systems in about a decade from now.

Comment Re:Step back into the stone age (Score 1) 4

Who said anything about directories not existing? Didgets has a very robust set of 'Folder Didgets' that can form traditional hierarchies like those found in traditional file systems if you want to use them to group files together. In fact any file can exist within several different folders simultaneously (without all the hard and soft link complexities of other systems). Tags are just another way of classifying files instead of relying solely on appropriate folder names in the file's path. It does not use ZIP files or any other compression program to store or group files. It sounds like the idea is just 'half-baked' to you because you jumped to all kinds of false conclusions.

Submission + - It's Time to Replace File Systems (substack.com) 4

DidgetMaster writes: Hard drive costs now hover around $20 per TB. Drives bigger than 20TB are now available. Fast SSDs are more expensive, but the average user can now afford these in TB capacities as well. Yet we are still using antiquated file systems that were designed decades ago when the biggest drives were much less than a single GB. Their oversized file records and slow directory traversal search algorithms make finding files on volumes that can hold more than 100 million files, a nightmare. Rather than flexible tagging systems that could make searches quick and easy, they have things like 'extended attributes' that are painfully slow to search on. Indexing services can be built on top of them, but these are not an integral part of the file system so they can be bypassed and become out of sync with the FS itself.

It is time to replace file systems with something better. A local object store that can effectively manage hundreds of millions of files and find things in seconds based on file type and/or tags attached is possible. File systems are usually free and come with your operating system, so there seems to be little incentive for someone to build a new system from scratch, but just like we needed the Internet to come along and change everything we need a better data storage manager.

See Didgets for an example of what is possible.

Comment Hmmm... (Score 4, Insightful) 159

So the data 'isn't ready for primetime' (meaning ready for public consumption), yet somehow the data is good enough to make public policy and claim that you are 'following the science'??? They can't have it both ways! Either the data is highly suspect or the actions put forth by the CDC are. Hiding the data from data scientists or even hobbyist who will likely point out how a policy isn't supported by the data, just makes them look even more guilty.

Comment It's not just music (Score 4, Interesting) 256

The same formulas apply to movies, books, software, games, and other 'content'. Innovation is often frowned upon as all the money flows to 'tried and true' forms. This is why everything looks the same or is just a recycled version of something created decades ago.

I have created a revolutionary new kind of data management software that does a bunch of things many times faster than conventional databases and file systems. I am surprised at how many people refuse to even consider it simply because it is not a 'drop-in replacement' for their current solution.

Comment Re:So its time to ... (Score 1) 96

Is it really that easy to fork a huge project like Qt? I don't know how many developers it has, but wouldn't the forked project require just as many to fix bugs, write documentation, add features, and do all the other stuff the main project would do? All because someone didn't like a single new feature.

Comment I like this idea (Score 2) 96

I use Qt for the admin tool for a data management system I have built (www.Didgets.com). The GUI provides a great way to display information, display content, and create graphs and charts. I don't have any ads currently, but if I wanted to it would be nice if the GUI provided support for it.

Programmers, just like musicians, actors, writers, and other content producers; want to be paid for their creations (in this case software). Either the end user needs to pay for their direct use, or they need to get someone else to pay for them (i.e. advertisers). Through television, radio, websites, and magazines the other providers have a way to give the consumer free access to the content in exchange for viewing, listening, or reading ads. What really irks users is when both methods are used (e.g. I pay for cable but every channel is still flooded with ads). I can see software developers doing the same thing. I pay for a software license or buy a subscription, but the software continues to show me ads!!!

Still it would be nice if I could offer potential customers a choice .. buy a license for ad-free use or use my software occasionally for free and let the advertisers pay me instead.

Comment Define 'short amount of time' (Score 1) 128

"Over the past 450 million years, life on Earth has been devastated by at least five mass extinctions, which are typically defined as catastrophes that wipe out more than 75 percent of species in a short amount of time...",

Bear in mind that people who study things like mass extinctions usually think of time in geologic terms. To them 100,000 years is like a blink of the eye. Homo Sapiens have been walking on this Earth for only 'a short amount of time'.

Slashdot Top Deals

"Money is the root of all money." -- the moving finger

Working...