Please create an account to participate in the Slashdot moderation system

 



Forgot your password?
typodupeerror

Comment Wrong questions. More details needed. (Score 5, Informative) 217 217

You're not asking the right questions:

The first correct question is why on earth would someone need to access half a petabyte? In most cases the commonly accessed data is less than 1%. That's the amount of data that realistically needs to reside on disk. It never is more than 10% on such a large dataset. Everything else would be better placed on tape. Tiered storage is the answer to the first question. You have RAM, solid/flash storage (PCI based), fast disks, slow high capacity disks and tape. Choose your tiering wisely.

The second question you need to ask is how the customer needs to access that large datastore. In most cases you need serious metadata in parallel with that data. For Petabytes of data you cannot in most cases just use an intelligent tree structure. You need a web-site or an app to search that data and get the required "blob". For such an app you need a large database since you have 5M objects with searchable metadata (at 200MB/blob).

The third question is why do you have SAN as a premise? Do you want to put a clustered filesystem with 5-10 nodes? Probably Isilon or Oracle ZS3-2/ZS4-4 are your answer.

Fourth question: what are the requirements? (How many simultaneous clients? IOPS? Bandwidth? ACL support? Auditing? AD integration? Performance tuning?)

Fifth question: There is no such thing as 100% availability. The term disaster in Disaster Recovery is correctly placed. Set reasonable SLA expectations. If you go for five-nine availability it will triple the cost of the project. Keep in mind that synchronous replication is distance limited. Typically, for a small performance cost, the radius is 150 miles and everything above impacts a lot.

Even if you solve the problems above, if you want to share it via NFS/CIFS or something else you're going to run into troubles. Since CIFS was not realistically designed for clustered operation regardless of the distributed FS underneath the CIFS server, you get locking issues. Windows Explorer is a good example since it creates thumbs.db files, leaves them open and when you want to delete the folder you cannot unless you magically ask the same node that was serving you when it created the Thumbs.DB file. Apparently, the POSIX lock is transferred to the other server and stops you from deleting, but when Windows Explorer asks the other node who has the lock on the file you get screwed since the other server doesn't know. Posix locks are different from Windows locks. It affects all Likewise based products from EMC (VNX filler, Isilon, etc.) and it also affects the CIFS product from NetApp. I'm not sure about Samba CTDB though.
I would design a storage based on ZFS for the main tiers, exported via NFSv4 to the front-end nodes and have QFS on top of the whole thing in order to push rarely accessed data to Tape. The fronted nodes would be accessed via WebDAV by a portal in which you can also query the metadata with a serious DB behind it.

I've installed Isilon storage for 6000 xendesktop clients that all log-on at 9AM, i've worked on an SL8500, Exadata, various NetApp and Sun storages and I can tell you that you need to do a study. Have simulations with commodity hardware on smaller datasets to figure out the performance requirements and optimal access method (NAS, Web, etc.). Extrapolate the numbers, double them and ask for POC and demos from vendors, be it IBM, EMC, Oracle, NetApp or HP. Make sure that in the future, when you'll need 2PB you can expand in an affordable manner. Take care since vendors like IBM tend to use the least upgradable solution. They will do a demo with something that can hold 0,6PB in their max configuration and if you'll need to go larger you'll need a brand new solution from another vendor.

It's not worth doing it yourself since it will be time-consuming (at least 500 man-hours until production) and with at least 1 full-time employees for the storage. But if you must, look at Nexenta and the hardware that they recommend.

And remember to test DR failover scenarios.

Good luck!

Data Storage

Ask Slashdot: How Do You Store a Half-Petabyte of Data? (And Back It Up?) 217 217

An anonymous reader writes: My workplace has recently had two internal groups step forward with a request for almost a half-petabyte of disk to store data. The first is a research project that will computationally analyze a quarter petabyte of data in 100-200MB blobs. The second is looking to archive an ever increasing amount of mixed media. Buying a SAN large enough for these tasks is easy, but how do you present it back to the clients? And how do you back it up? Both projects have expressed a preference for a single human-navigable directory tree. The solution should involve clustered servers providing the connectivity between storage and client so that there is no system downtime. Many SAN solutions have a maximum volume limit of only 16TB, which means some sort of volume concatenation or spanning would be required, but is that recommended? Is anyone out there managing gigantic storage needs like this? How did you do it? What worked, what failed, and what would you do differently?
Software

Ask Slashdot: User-Friendly, Version-Preserving File Sharing For Linux? 212 212

petherfile writes: I've been a professional with Microsoft stuff for more than 10 years and I'm a bit sick of it to be honest. The one that's got me stuck is really not where I expected it to be. You can use a combination of DFS and VSS to create a file share where users can put whatever files they are working on that is both redundant and has "previous versions" of files they can recover. That is, users have a highly available network location where they can "go back" to how their file was an hour ago. How do you do that with Linux?

This is a highly desirable situation for users. I know there are nice document management things out there that make sharepoint look silly, but I just want a simple file share, not a document management utility. I've found versioning file systems for Linux that do what Microsoft does with VSS so much better (for having previous version of files available.) I've found distributed file systems for Linux that make DFS look like a bad joke. Unfortunately, they seem to be mutually exclusive. Is there something simple I have missed?

Comment Re:Germany should pay war reparations for WWII (Score 4, Insightful) 743 743

This kind of ridiculous stunt is why the Germans are sick and tired of giving Greece money. They've been model world citizens and have been subsidizing Greece for decades, and trying to use this now is the ultimate in spoiled screaming teenager tactics. Nobody bankrupted Greece except Greece - as the Nordics, who actually got their shit together, very painfully, like to point out.

If I remember correctly, it was the 3rd party auditors that made the economical recommendations that led Greece to bankruptcy. In a perfect world, the financial institutions and auditors that pushed Greece onto such a road would pay for the economical disaster that they directly contributed to. But I guess that they're busy giving bonuses to C*Os. If your financial consultant (or tax consultant) makes wrong calculations/projections/recommendations for you and puts you into default, wouldn't you seek compensation from him? You did pay him to give you realistic results. How can one country's rating go down from AAA to Junk in one day?

Germany are somewhat dour and grumpy parents, and a Grexit now is much less harmful to Eurozone than it would have been two years ago, so being kicked out of the house isn't out of the question at all. I wouldn't push it too hard.

You're claiming that it's not fair, but the IMF and ECB gave Greece loans at rates that are not sustainable. I can get an EURO credit at a lower rate than Greece has. Furthermore, for Germany it's win/win. They bought out a lot of Greek companies for pennies. Think of OTE that was bought by Deutsche Telekom. I personally feel like this is looting and not helping out. Private corporations from the US, UK and Germany (financial and audit) bankrupted Greece with bad advice, while earning serious money for it (think Deloitte, S&P, etc.). When the bubble burst, the Greek government received help at ridiculously high rates from a few countries and multi-national institutions. Then came the major companies from those countries and bought everything for pennies. Afterwards, they are still complaining that the Greek can't make the payments.

I'm not German or Greek, but have been following this for years in the Economist and Bloomberg, and I know lazy scammers trying to wheedle more money rather than earn it.

I see your problem right there: you're reading it from Economist or Bloomberg. How about checking out the bare survival conditions of a lot of Greek citizens? Should Greece abandon them because Germany said austerity is the way? The Greek government's responsibility is to it's citizens. P.S.: I'm not Greek or German either. I don't live in Greece or Germany, but I try to get my news from newspapers that aren't necessarily in New York, London, Frankfurt, Tokyo or Hong Kong.

Comment Re:Two answers (Score 1) 162 162

Literally filed or overcrowded with drones: Never, what would they be doing?

This reminds me of a classical line: "there is a world market for maybe five computers"

  • Surveillance & Patrolling (public, intelligence services, military and private)
  • Transport (think of those Pizza delivery SLAs going down to 10 minutes or Amazon)
  • Leisure
  • Emergency Services Support (Search & Rescue, Disaster Evaluation & Reconnaissance, etc.)
  • Gardening, Crop Spray-ing, etc.
  • IT Infrastructure Support for massive events (think wireless at a concert)
  • Scientific measurements for weather/pollution/etc.
EU

ESA Complete Spaceplane Test Flight; IXV Safely Returns To Earth 56 56

hypnosec writes The European Space Agency has successfully completed the first test flight of its Intermediate eXperimental Vehicle (IXV), as planned, wherein it saw the wingless spaceplane land in one piece in the Pacific Ocean. A Vega VV04 rocket took the IXV to an altitude of 340 km, from which it separated and continued up to 412 km. Reentering from this suborbital path, it recorded a vast amount of data from more than 300 advanced and conventional sensors. According to ESA the spaceplane few east around the globe during its descent and finally landed safely in the the Pacific Ocean west of the Galapagos Islands at about 15:20 GMT.

Comment Re:Absolutely correct! (Score 1) 155 155

Would you please point to a study that states that fracking wells have a higher incidence of water contamination than normal classic oil or gas wells? Traditionally Romanian gas exploration has used hydraulic fracturing. The only difference is that we are now drilling deeper, as well as horizontally and we can exploit more from an existing deposit. To give you an idea: right now, out of all the electricity produced in Romania, only 39% is carbon producing (coal, heating oil, gas), the rest is non carbon producing (hydro, wind, nuclear, photovoltaic, biomass). You can see the real-time information on http://www.transelectrica.ro/w... . You an also see historical values http://www.transelectrica.ro/w... .
Romania has gone through a complete overhaul of it's energy sources in the past 20 years. We have an installed capacity of 23GW with a power usage between 4GW (low point in summer) and 9GW (max point in winter). In the past 10 years we've added 2,5GW of wind turbines (completely absent until then), and 1GW of photovoltaic. Since we still need gas (for now) and have ample reserves, why should we import from our "old adversary" instead of using our own?

Businesses

Why CurrentC Will Beat Out Apple Pay 631 631

itwbennett writes Working closely with VISA, Apple solved many complex security issues making in-person payments safer than ever. But it's that close relationship with the credit card companies that may be Apple Pay's downfall. A competing solution called CurrentC has recently gained a lot of press as backers of the project moved to block NFC payments (Apple Pay, Google Wallet, etc.) at their retail terminals. The merchants designing or backing CurrentC reads like a greatest hits list of retail outfits and leading the way is the biggest of them all, Walmart. The retailers have joined together to create a platform that is independent of the credit card companies and their profit-robbing transaction fees. Hooking directly to your bank account rather than a credit or debit card, CurrentC will use good old ACH to transfer money from your account to the merchant's bank account at little to no cost.

Comment Re:Switch to Solaris... seriously... (Score 1) 195 195

I remember seeing at some point numbers. It didn't impress in a single thread, but could easily saturate a 10Gb link in multi-threaded tests. They tested an FTP server on a T2plus. Regarding cores, we have anything from dual UltraSPARC IIIi to T4 based systems including some M-class. I believe the T3-4 has the highest number of cores. It should be 64 cores and 512 threads, but a single Solaris instance can only see 256. I believe that the M9000 and M9000-64 should have the same problem, but the biggest M series I've worked with is M8000.

Comment Re:Switch to Solaris... seriously... (Score 0) 195 195

While I'm a Solaris admin for some time, I can tell you that it's not the best TCP/IP stack. It does have all the bells and whistles, but it's not even close to the speed of FreeBSD. It's actually not even in the same ballpark as FreeBSD. It's probably Linux fast if you tune it properly. It does have cool configuration, virtual switches, link aggregations, hardware crypto that can be usable by OpenSSL, OpenSSH, and ipsec but it's not even close speed-wise. But the cost of all those features basically means that it has mediocre performance for simple, yet performance-hungry scenarios.

Comment Re:Baseline power? (Score 2) 365 365

Sunny days they make tons of "free" electricity.

On cold dark winter nights, where does the power come from?

They can build backup plants that run on coal/gas typically operating under nameplate capacity or they can buy nuke power from the French.

Oh, the irony...

You've got it. What I don't understand is why nuclear electricity is put in the same basket as coal and gas plants. The incidents that Nuclear has gone through in the past 60 years only reinforce my view that it's a safe solution. If given all the fsck-ups that gave us Chernobyl, Fukushima and 3 Mile Island that's all that happened I think that it's pretty much OK. I'm saying this because coal/thermal have their exhaust pipe problems which affect a much greater percent of the population and hydro is in general an ecological mess that also involves massive population relocation.

Comment Re:So No then (Score 2) 464 464

Nothing is laptop hardware in that machine. Like previous Mac Pros it has workstation cpu (Xeon), workstation graphics (FireMV) and workstation RAM (registered, ECC). Indeed, the mac mini has a laptop CPU and SO-DIMMs for memory, but we're talking about the Mac Pro.
Furthermore, I don't get the "doubts about the thunderbolt displays". Thunderbolt can act as a simple mini-display port (with audio also). So go grab your $150 Dell Display Port monitor and plug it in. All it takes is a $8 mini-display port M to display port M cable. If you want to use the more advanced features of thunderbolt, it's a matter of taste, but for a lot of external hardware USB is not an option even in it's 3rd incarnation.

Comment Re:and... (Score 1) 464 464

Stop scratching the machine if you don't want scratches on it. My workstation is always on, and I think that except for dusting it, I haven't actually touched it in over 1 year. Now going to serious stuff...
Upgrades are allowed. It features 6 Thunderbolt ports so you can add as many 10GigE, FC, HBA, high performance external directly attached storage arrays, Video Capture controllers as you want. There are a few thunderbolt to pci-express 2.0 8x adaptors available if you want to use your own hardware.
I guess that the only non-upgradable parts are the video cards. I think that they are swap-able but due to their proprietary format there would be no 3rd party alternatives.

Internet Explorer

Microsoft Boasts of Tiny Energy Saving With IE 243 243

judgecorp writes "Microsoft has sponsored research that indicates that its Internet Explorer browser uses less power than the competition, Firefox and Google (there's no explanation of what causes the difference). However, the difference in power use is not really significant — it's about one Watt when browsing. Browsing for 20 hours at this rate, the IE user would save enough power to make a cup of tea, compared with Firefox and Chrome users. That Microsoft commissioned and published the report seems to indicate a certain desperation to Microsoft's IE marketing efforts."

"You're a creature of the night, Michael. Wait'll Mom hears about this." -- from the movie "The Lost Boys"

Working...