Comment Re:Another vote for NoSQL and some experience (Score 1) 235
I have seen these kinds of situations happen a lot (I'm a statistician who works on computationally-intensive physical science applications), and the best solution I have seen was a BerkeleyDB setup. One group I work with had a very, very large number of ASCII data files (order of 10-100 million) in a directory tree. One of their researchers consolidated them to a BerkeleyDB, which greatly improved data management and access [...] I think the general idea of a key-value store that lets you keep your data in the original structure would work well.
A file system *is* a key-value store.
I suspect those 100,000,000 files were in fact tiny pieces of data which didn't make sense to access using normal tools (from ls to MS Word). That the conversion worked out for *you* doesn't mean that it would be useful to convert *every* set of files into a BerkeleyDB. Especially not sets of (say) 500 files, 10GB each.
I completely agree. If you have a lot of small datasets that break ls and such (as was the case in my situation), BerkeleyDB provided a great solution. If you have a smaller set of very large files, a different solution is needed (perhaps just the file system with some kind of automated indexing).
Comment Another vote for NoSQL and some experience (Score 2, Informative) 235
How Do I Get Open Source Programs Written For Me? 285
Small Asteroid On Collision Course With Earth 397
Nobel Prize For Medicine Awarded, Physics Soon To Follow 135
On Fourth Launch Attempt, SpaceX Falcon 1 Reaches Orbit 518
Wall Street's Collapse Is Computer Science's Gain 435
Comment Check with university libraries (Score 1) 4
NYT Ponders the Future of Solaris In a Linux/Windows World 340
Microsoft Blesses LGPL, Joins Apache Foundation 425
Amazon Insists Publishers Use Their On-Demand Printer 182
The Reality Distortion Field Is Real 270
Creationism Museum Opening in Kentucky 1166
Two Ways Not To Handle Free Speech 686
Update: 02/11 04:25 GMT by KD : updated Nick Gisburne link to new account.