It probably has something to do with the difference between claims and description in a patent application. Claims are the part that matter. Often the claims are constructed so they *just barely* pass the obviousness test, e.g. by taking two ideas that are too obvious by themselves, but combining them in a way that's less obvious. The description can then be far more general, and is often shared between many patents, but that doesn't affect the validity of the claims *at all*. To determine the validity of a patent you have to look very carefully at what is being claimed, and only refer to the description as background to understand the claims.

Disclaimer: IANAL and I don't give legal advice. I've just been through this nearly a dozen times.

Disclaimer: I'm the project lead for HekaFS, which is based on GlusterFS.

If you're concerned about data protection, you'll want to worry about node as well as disk failures. Some distributed filesystems, including Lustre and PVFS*, take a rather old-school "use RAID and implement your own heartbeat/failover between server pairs" approach, and that just sucks. GlusterFS and Ceph don't have that wart; neither do MooseFS or XtreemFS, which I would consider the other alternatives. They all have their own forms of replication built into the filesystem, so you don't need to set up and maintain another layer for them. Unfortunately, neither MooseFS nor Ceph survived even simple tests - write a few files in parallel, flush caches, read them back in parallel - when I ran those tests on the same hardware as GlusterFS and XtreemFS which did fine. That was a while ago, though, so take that with a grain of salt. Ceph in particular has a lot of awesome technology and has a very bright future IMO, but it's taking a while for it to realize that potential.

Out of GlusterFS and XtreemFS, the choice has a lot to do with your exact use case. XtreemFS has a pretty strong focus on wide-area replication, so if that's part of your need now or likely to be in the future then it's probably a bit stronger. GlusterFS does have some wide-area replication, but I consider it rather weak. Within a single data center, I'd give GlusterFS the edge. It has better local performance than XtreemFS in my tests, and it has what I consider by far the best setup/management interface.

The one caveat I'd offer is that all of the filesystem I've mentioned excel for sequential access for large files. For random access, and especially for metadata-heavy workloads, they all suck to some degree. As others have mentioned, you might very well be better off with a simple NFS server pair with cheap shared storage and heartbeat/failover to ensure availability.

Look them up. They already allow you to attach arbitrary metadata to a file. Most modern filesystems and user-level utilities support them already. They're even used as the underpinnings for security mechanisms such as POSIX ACLs and SELinux. Sure, there are issues with performance when you have *lots* of xattrs on a file, and that's a fruitful area of research, but we sure don't need some brand-new Microsoft-invented thing to deal with metadata.

Doing something for 7857 files and doing it for 10 billion are very different situations. 7857 files, including metadata, can easily be sucked into memory in one big chunk and unpacked/examined from there. That simply doesn't work for datasets larger than memory. At the higher scale, modern filesystems do tend to fall apart, badly, so different approaches are needed. Comparing your paper airplane to an F-22 doesn't make it look like you know anything about writing software properly. Quite the opposite.

...when people in the community, instead of setting a good example, fetishize the act of trolling itself. When high technical contribution is combined with presentations full of pornographic images/metaphors and Twitter streams full of laughter at others' consternation, such childish behavior becomes the New Conformity. It's just as cliquish and pointless as the Old Conformity these rebels without a clue pretend to reject, but whenever aspiring programmers see that opinions presented in one set of clothes get a quicker/more friendly hearing than the same opinions presented in a different set of clothes it's totally predictable how they'll respond. They'll imitate all the off-color and trollish behavior that they see, and some of them will end up stepping over lines that actually matter. It's all good fun until promising projects and startups fail because would-be users and collaborators get turned off by the hipster posing. What kind of sociopath would make a decision where the only possible upside is a few laughs and the potential downside is colleagues losing their jobs? It doesn't matter if you feel your own job is secure, or if you feel that people shouldn't react as they do; anybody who pulls this kind of stunt doesn't deserve a job or funding or anything else but our contempt.

The Mitre list does include "Use of a Broken or Risky Cryptographic Algorithm" but in my experience that's far less common than improper use of a perfectly good algorithm. Many algorithms and modes have known weaknesses that require specific generation/handling of keys and initialization vectors to maintain good security. Most algorithms and modes that are secure against unauthorized *reading* of data still require an extra MAC step to prevent unauthorized *modification* of that data (including targeted bit-flips). Developers often take shortcuts in these areas because doing all of "the right things" adds a lot of extra complexity and can absolutely kill performance. Look at recent events involving Dropbox and Jungledisk for examples. I don't think the Mitre list adequately conveys that cryptographic security requires not just good low-level algorithms like AES or Blowfish but also good higher-level (usually domain-specific) algorithms governing how the low-level algorithms and their inputs are used.

Exactly, and thank you for saying that. I suspect that Facebook's actual customers - advertisers - are very happy indeed with them. Users? Screw 'em, just not hard enough that they leave.

I recently went through a related exercise with my daughter. I showed her ponycorns because it's exactly the kind of game she already likes. She got the idea that we could create a game too, since I'm also a programmer and she's a whole two years older than Sissy. At that age I'm not about to teach her actual programming, but I thought it might be a good way for her to see the creative process at a higher level - plus we'd end up with something that she could enjoy and show off to her friends. We had a lot of fun with her drawing the characters which I then turned into sprites, and recording audio, and brainstorming about what puzzles would be in the game. Now the effort has stalled, mostly for lack of a decent framework. I mean, all we need is basic point-and-click stuff, maybe even an inventory and stuff like that, but it would sure be nice to have the characters actually *move* smoothly from one place to another instead of just disappearing from one place and appearing in another. Oh, it would be extra nice to have something open source, or at least runnable on Linux. I looked at dozens of frameworks that I found on http://www.ambrosine.com/resource.html and elsewhere, and very few could meet those simple requirements without getting into full-out 2D suitable for side-scrollers and platformers - meaning that they're way more complicated than I need and generally don't "scale down" to the simpler stuff very well. I tried Adventure Maker but quickly ran into its limitations even with a project as simple as this. I might try GameMaker next, even though it's also Windows-centric (I can use my wife's machine if I have to), because the other offerings out there seem so incredibly thin.

The point is not that I personally need help finding a tool with which to make this particular game - though suggestions would certainly be welcome. The problem is that the "state of play" is just so incredibly piss-poor overall. Forget about finding something that even an older child could use *themselves* to create a game that doesn't totally suck. It's hard enough to find something that a *professional programmer* (albeit not a game programmer like Sissy's dad) can stand to work with long enough to get such a result. Something just good enough to let parents and kids put together a simple adventure/puzzle game on par with ponycorns, to give them something that's fun and that just barely hints at what you can do if you can program, would go a long way toward making them want to learn more. As far as I can tell, such a thing doesn't exist.

(NB: I'm the founder/lead for CloudFS)

Thanks for the mention. To be quite clear on this, the at-rest encryption that's currently in CloudFS is not as secure as we'd like it to be, or as secure as it will be when it's released. To put it another way, it's more secure than Dropbox or Jungledisk have proven to be, it's probably more secure than a couple of dozen other similar cloud-storage options (it's hard to tell since so many are not open source), but it does have flaws. To be more specific, it's secure against inspection by someone who only has the ciphertext - such as your cloud provider. However, it is not secure against transparent modification (flipping a bit in the ciphertext flips the corresponding bit in the plaintext). Also, since it's currently CTR-mode encryption, if someone has both ciphertext and plaintext for the same part of a file then that part of the file becomes readable from just ciphertext thereafter. These flaws are not acceptable; the current code is only a stopgap. This is exactly why I made the point on Twitter recently that even the strongest ciphers with long keys can still result in weak protection if used improperly. I'm sick of seeing cloud-storage providers crow about how strong their transport encryption is but say nothing about on-disk encryption, or mention using "military grade AES-256" on disk but say nothing about how. Worst of all are the ones -who require that you give them keys - which for all you know will be stored unprotected right next to the data.

The good news is that I've been consulting with some real crypto experts - I admit I'm not one myself - on this. We've worked out a block-based scheme that all involved believe will address the above flaws, while also handling concurrent writes correctly (something most "personal backup" alternatives fail to do). The performance cost is more than I'd like, but I think it's no more than necessary and the parallelism inherent in the underlying system should still yield more-than-adequate performance. I've already begun implementation, and will fully disclose all the details once I get a bit further along.

You've decided to permanently archive only your side of the record and he never got the privilege of adding his defense (or eventual outcome) to it.

Not true. Even aside from the fact that anyone can get their own blog, he has commented on my site and I have allowed him to. If he wants to add more in his own defense, he may and he knows that.

You've decided to permanently archive only your side of the record and he never got the privilege of adding his defense (or eventual outcome) to it.

"Sociopathically"? You certainly can think what you want, but I'm not the one posting like a sociopath.

I should take lessons in civility from someone who joins the conversation with "You're a dick"? I don't think so. It's hardly a surprise that someone who acts like that would take an extreme "forget what happened" attitude, either. Very self-serving of you. Maybe an argument for people to be kind and forgiving shouldn't be delivered with such jarring contrast to your own behavior.

For what it's worth, I don't want to punish this guy forever. That's why I made the post un-searchable, helping him "move past the idiot stage" as I hope someone will do for you someday. It does have a little to do with accurate historical documentation, though I don't think there's anything fucked up or fanatical about that. I don't believe in censorship or revisionism. It was part of the public record once, and it should remain so. I've seen the power to edit the record used for ill *far* more often than I've seen it used for good, and I'm not going to start down that path. Once posted, everything on my site stays posted. It's my site, so deal with it.

The key point here is that you don't have to forget in order to forgive. If this had reached the courts, which it nearly did, that would be part of the public record and would remain so with nothing I could do about it even if I wanted to. It would also, rightly, be considered part of the *past* once the offender had been punished or made reparations appropriately. We need to teach people to forgive *without* forgetting, because forgetting is antithetical to learning.

Valid questions. Yes, I did some homework to convince myself that the story was valid, but that would have made the anecdote too long. ;)

I've long since gotten used to the idea that everything I say online - going back to Usenet days and even before - will stay with me forever. Some times you just have to remind people that it was X years ago and people/opinions change. Would you take advice from someone in mid-life whose opinions hadn't changed since their teens?

That's all garden-variety stuff by now, but I did have a more interesting case come up on my website. I had occasion to write about someone who was trying to scam people with an online "contest" that was rigged. Yes, I named names, especially after the guy (who went by more than one name BTW) tried to intimidate me with fake legal threats. Years later, I got email saying that he'd reformed, he was trying to get a job, but potential employers would Google for his name and find my site. Tough luck, I thought, and continued to think as the pleas kept coming every few months for years. What finally got my attention was when he mentioned that he now had a family. This little piece of history, no matter how valid, was now starting to affect *other people* who were completely innocent. While I don't believe in censorship, I do believe in the validity of the "statute of limitations" concept so I decided on a compromise. The article about this guy is still on my site, you can even find it by searching there, but you can't find it by searching on Google. (Robots.txt plus referer blocking specific to that post, for those who care.)

The lesson is that the existence of information and the ease with which it may be looked up are two different things. Dirt is just too easy to find, for the same reasons that gold is too hard: search engines' evaluation of "importance" or "relevance" doesn't always match any sane human's. While it should be *possible* to find someone's decade-old forum posts, perhaps it's not quite right for the most inflammatory thing they ever said to be the very first thing that shows up in a casual search . . . and it often will be, because controversy drives higher rankings. Making stuff just a little bit harder to find, like we all do here with low-rated comments and like I basically did in this little anecdote, deserves more frequent consideration as an alternative to deletion.

Thank you for making that point. Marketing and finance and such are all wonderful optimizations of the wealth-creation process, but there has to be some actual wealth-creation to optimize. How many people in the US are actually creating wealth, instead of figuring out how to persuade others to spend some of theirs? It's no wonder we have booms and busts, when 90% of the "wealth" out there is total speculative bovine-excrement. BTW, I do know not all readers here are from the US, but I am referring here specifically to the US; as a citizen here all my life, I'm painfully aware that just about every other country has a much more realistic view of the relationship between real value and mere dollar signs. As far as I can tell, Wall Street and Silicon Valley are both turning the US into a big Golgafrincham Ark, full of people whose much-vaunted skills are only of value within a totally inbred economic system increasingly divorced from the actual needs or wants of anyone not born into that economic and sociopolitical environment. JeffH (disclaimer: we've met) had it right: that sucks. Even if we don't personally have the skills to do things that result in direct benefits *somewhere* on a sane Maslow Hierarchy, we can at least turn those skills toward taking the tools that have been developed to serve the vapid goals of "social" media and try to re-orient them toward more productive purposes.

...is that *no* single-figure-of-merit benchmark is going to be worth anything. Sandia's "Graph 500" Johnny-come-lately isn't going to be any better than Linpack that way, and will just skew the results towards a different not-generally-useful architecture. A far better idea has been around for over five years: the HPC Challenge benchmark. It consists of seven different tests (Linpack is just one) which stress different aspects of system design. Anybody who knows anything about building big systems would identify some mix of these tests that best approximates their own workload, use that as a starting point for looking at likely alternatives, and then remember that it's just a starting point. The only benchmark that really matters is the one that you run yourself on your own application, but that can be a very expensive and time-consuming exercise so these lists can be a good way to figure out which systems deserve that more extended analysis. Linpack, on the other hand, isn't even useful for that. What's sad is that some people either didn't know (which says something about how we train engineers) or didn't care until a Chinese system found its way to the top (which says something even worse).