There are a lot of naysayers on here, but since you did not specify the capacities you are handling I'm going to assume that you are working with hundreds of terabytes, the scale at which using a tape library starts to really make economic sense. Any kind of "use tape to complement hard disk storage" scheme will use a robot-driven tape library. If you aren't in this class of solution, then the other posters are right, do not even consider going down this path, the expense ($10K USD entry level) is not worth it.
What you are looking for is called a Hierarchical Storage Management solution. They are all proprietary software (the hardware part of the solution is pretty much functionally interchangeable), there is no production-grade open source offering, which is unfortunate. The proprietary ones I know of don't allow hooks to programmatically customize inject/retrieval policies and operations, the primary reason to want an open source alternative (though Tivoli Storage Manager has an extensive API that someone could use to roll their own HSM with its own API complete with programmatic hooks).
If you find these financial requirements too onerous, then as a middle ground solution I recommend you get commercial-grade hard disks like Western Digital Black with 5-year warranties to hold everything you have, a single tape drive for traditional backups, some software that supports incremental backups, and climate controlled storage for tapes.
"Over the next two years, H-P will work with members of the Linux community to develop a version of that operating system with reliability features like those that come with the version of the Unix operating system that powers some of its Itanium-based machines. That will run on servers powered by x86 chips, such as Intel's Xeon or AMD's Opteron line.""
Link to Original Source
Link to Original Source
Before a rookie who takes this seriously mouths off to management that they've done this, here is how this tactic works out in my prior life as an employee, before I started my own business. YMMV.
After I had stacked up three years of living expenses, on top of zero debt of any kind, I found that my demeanor in negotiations changed. Not obviously, and not obnoxiously. This, I found by accident, was far more effective than simply blurting out that I could give a damn because I was debt free. The other side of the negotiating table can sense the subtle shift, likely because without that debt clouding my judgement in the back of my mind, I could think without emotion about the negotiation itself. This business-like focus on the merits of my contributions to the bottom line were effective in securing what I wanted.
Years later, I appended additional rules to the two shown above, which other US-based Slashdotters might find even more effective after securing the first two rules.
- Research your health insurance plan options available as a private entity, and be sure to set aside enough in your contingency budget to accomodate that amount. Speak to the billing staff of medical professionals you use today, and ask them which insurance plans (not companies) are the easiest to work with, and ask insurance brokers which plans they use for their own family.
- Especially if you have family to look after, secure a long-term disability insurance policy and term life policy on your own. Read up on these insurance products at Consumer Reports.
After this point, as long as you focus on keeping yourself sharp and relevant to the market (when I was an employee I did this by continuously updating my resume on the job sites with each project I finished, comparing against skill sets listed in open positions I would be interested in if I was looking, and staying on friendly terms with the recruiters that kept calling me), you pretty much call the shots in compensation negotiations as long as you stay within market range.
The standard explanation proffered by the HFT owners and customers is they "add more liquidity". This is repeated so many times that laypeople buy it; see typical comments in this thread like "we have traded wider spreads for higher instability". This is not the entire story: the liquidity is for them, not for you . That there is sometimes a spillover liquidity and spread improvement for participants in the wider market is merely a convenient observation suitable for PR. The past and ongoing flash crashes demonstrate that when the liquidity trades against them, they pull this vaunted liquidity quicker than you can blink, literally. They're not going to leave money on the table supplying liquidity into the market if they don't have to.
Another oft-made claim is "anyone is welcome to do what we do, there are no barriers to entry". That is not quite the entire story as well. The defining feature of an HFT firm over the retail investor apart from scale (you need accredited investor-scale financial depth just to ante up the money to the exchange to cover their risk for you fracking up your code and making market on your fracked up orders they then have to make good upon) is access, as the articles this story links to amply documents. They are quite different from most market participants. While it is true that one doesn't have to have special institutional privileges and access to buy these newfangled digital-age "exchange seats", and "merely satisfying" some financial and technical criteria make these seats putatively easier to obtain than the old seats, make no mistake about it, they are more privileged than the old school NYSE exchange seat holders: they enjoy special access to the markets that "non-seat holders" do not, namely preferential positioning in the order flow inspection pipeline, or put another way, they enjoy market making access without market making responsibilities. Just because you no longer have to have a hallowed name descending from the Mayflower, a family history intertwined with the exchange, and an imposing granite edifice for offices to qualify for an exchange seat that buys access to the order flow doesn't mean that preferential access is open to everyone. The day the exchanges open up the HFT level and quality of tick access for the same price as 15-minute delayed ticker quotes, would be the day that I withdraw this observation.
If you chafe at these new special breed of privileged market participants, then an old school remedy is still available: with privileged market access, comes market making responsibilities and market making regulatory oversight. Perhaps not as much responsibility as the exchanges, but definitely more than those without the preferential access, commensurate with their impact upon the market as shown by the flash crashes. Let them have the special access, but make good on the liquidity and spread claims with regulatory enforcement; that is, they continue eating at the trough even when the liquidity and spread moves against them. It didn't stop the old school market makers from coming up with different licenses to print money, so they'll still make great bank (though they'll bitch like a platoon of coked-up noob IB's at Penthouse for having to run through regulatory hoops that didn't exist before, instead of spending that time cranking the next batch of algorithms onto FPGAs), but coupling privileged market access with market making responsibilities did truly impart long-term benefits to participants in the wider market. Arguable if the benefit was proportional, but as long as we will tolerate differential access, we might as well at least maintain the marginal benefits of status quo ante, eh?
You rarely have mail stolen...
This should be emphasized. I visited a gold mine in the US once. Was astounded when they told us they mail their raw ingots (that contain gold, silver and platinum all mixed together) to their refiner by USPS. They matter-of-factly told us that only USPS had the kind of government-force-backed security and guarantees that made transporting around >$100K bars every day feasible.
It is fascinating that business professors are not inundated with requests from programmers clamoring for just an idea person to pair up with to change the world.
That indicates there is a supply-demand imbalance between idea people and programming people, that all these supposedly business-oriented idea people are overlooking. An imbalance that is classically rectified by raising the value of programmers, and/or lowering the value of idea people until the supply-demand curves intersect at a more mutually-acceptable point.
In any case, trying to discuss whether the idea or the perspiration is more important vastly oversimplifies the complexity of a business. I suggest that the more relevant matter to pursue in a technical forum is, why are so many programmers such poor negotiators that as a group, they have come to be perceived as a commodity in the mainstream?
USPS, UPS and FedEx will like this IF they are involved.
Point out how much they will save on payroll, and in the US, payroll and medical and retirement benefits. You'll have them beating a path to your door (provided you solve the engineering and reliability problems others have pointed out in this thread).