It's hilarious watching people argue over a topic that has already been shown to be a non-issue. The EIA (US) and German statistics show that, in aggregate, wind-energy sources produce a relatively steady amount of power. Individual turbines and even whole wind farms might not be deterministic, but all the wind farms taken together... are.

-Matt

Except you can decode the salt trivially if you took a cab ride that happens to be in the data set and you recorded the license and medallion number. At which point the salt is useless.

-Matt

And... that's it? What did SMART say? Did you actually wear the SSDs out as-per the wear indicator? Or did you hit a bug in the samsung controller before the wear-indicator maxed out?

To be fair, the precise situation you describe, particularly if you did not retune the RAID-6 setup or the mysql server, and if the server was fsync()ing on every transaction (instead of e.g. syncing on a fixed time-frame as postgres can be programmed to do)... that could result in el-cheapo samsungs not being able to do any write-combining and cause a 256:1 write-amplication of the data.

With proper tuning the write-amplication could easily be reduced to 4:1 and you would probably be able to run the server with SSDs. Maybe use Intel or Crucial though, and not Samsung. It isn't just the controller that matters... just using stock firmware doesn't really net you a good, robust SSD and there aren't too many real vendors who work on the firmware vs just OEM whatever was supplied with the controller. Intel is probably one of the better ones. They actually fix bugs, as does Crucial. Samsung... I dunno.

-Matt

Yes, but it's a well-known problem. Pretty much the only thing that will write inefficiently to a SSD (i.e. cause a huge amount of write amplification) is going to be a database whos records are updated (effectively) randomly. And that's pretty much it. Nearly all other access patterns through a modern filesystem will be relatively SSD-efficient. (keyword: modern filesystem).

In the past various issues could cause excessive write amplification. For example, filesystems in partitions that weren't 4K-aligned, filesystems using a too-small a block size, less efficient write-combining algorithms in earlier SSD firmwares. All of those issues, on a modern system, have basically been solved.

-Matt

I have around 30 ranging from 40G to 512G, all of them are still intact including the original Intel 40G SSDs I bought way at the beginning of the SSD era. Nominal linux/bsd use cases, workstation-level paging, some modest-but-well-managed SSD-as-a-HDD-cache use cases. So far wearout rate is far lower than originally anticipated.

I'm not surprised that some people complain about wear-out problems, it depends heavily on the environment and use cases and people who are heavy users who are not cognizant of how they are using their SSDs could easily get into trouble.

For the typical consumer however, the SSD will easily outlast the machine. Even for a pro-sumer doing heavy video editing. Which, strangely enough, means that fewer PCs get sold because many consumers use failed or failing HDDs as an excuse to buy a new machine, and that is no longer the case if a SSD has been stuffed into it.

A more pertinent question is what the unpowered shelf-life for typical SSDs is. I don't know anyone who's done good tests (storing a SSD in a hot area unpowered to simulate a longer shelf time). Flash has historically been rated for 10-years data retention but as the technology gets better it should presumably be possible to retrieve the data after a long period on a freshly written (only a few erase cycles) SSD. HDDs which have been operational for a time have horrible unpowered shelf lives... a bit unclear why, but any HDD I've ever put on the shelf (for 6-12 months) that I try to put back into a machine will typically spin-up, but then fail within a few months after that.

-Matt

Google must know by now how bad a light its broken permission system is putting on Android. I can't run half the android apps I want to run on any of my Android devices any more because of the permissions they want. And a lot of the ones that I intentionally do not upgrade no longer work. It's making my three android devices useless and almost worthless.

I'm flabbergasted that there are full-on idiots in the Google command chain who are unwilling to address such a severe and obvious problem. Truly flabbergasted. Has Google gone insane?

I've already stated but I will again... when the iPhone-6 comes out, I'll be moving over to it from my perfectly working but horribly insecure Motorola Razr. At least then I can browse my facebook account from my phone without it sucking up all the stuff I've tried so hard to keep partitioned off of it. As it stands now, I can't even run customized UIs on my Android because the g*d* program insists on advertising on my notifications screen, even though I bought the paid-for version.

At least with iOS I don't have to worry about all this in-the-face crap ruining the experience.

-Matt

Where do you get this nonsense from?

-Matt

The spreads are smaller because of computerized trading, *NOT* because of HFT. HFT itself, verses normal computerized trading and non-HFT computer trading, is not going to have a big impact on the spread. In fact, HFT algorithms themselves do not really work all that well if there is any significant spread. They require volume to operate... no volume, no HFT.

-Matt

I think reality tends to trump thought experiments. I don't dislike the 1-second auction idea, I think it would work quite well. But I disagree that IEX's ability to stop the HFTs cold is a fluke that will disappear as their volume goes up. Their reasoning is sound and obvious and immediately solves the biggest problem that money managers have these days when trying to buy or sell large amounts of stock. I don't see how volume changes the equation at all.

Besides, his paper does not appear to say what you summarized at least in regards to IEX. It simply states that IEX is solving one aspect of the problem. It's pretty easy to argue that the piece they are solving is the biggest piece of the pie. Personally speaking, I don't care about the aspects of HFT which only involve standard arbitrage.

In terms of HFT... it was obviously fraudulent from the day it started beind used. The SEC should have acted immediately and didn't. Companies were spending hundreds of millions of dollars on infrastructure to get sub-millisecond transit improvements and they were lying to our faces talking about improving liquidity, magically, well in excess of the capital they actually had in play, when it was obvious that they were only exploiting flaws in the system.

It was a failure of the financial media as much as it was a failure of the SEC, but the SEC *should* have acted immediately and they didn't. And the result is a major loss of trust in the mechanisms of the stock market to the point where many retail investors who didn't understand the low scale of the fraud exited the market and stayed out of the market when they should have stayed in. I'm not going to make excuses for those people, I certainly wasn't scared away, but the general public deserves better than what the media and the government has handed to them over the last ~6 years.

-Matt

What, you mean pensions that companies are unable to make good on? Sounds to me like they SHOULD be eliminated.

And you forgot the single biggest reason why poor people stay poor. It's a four-letter word. D.E.B.T. #1 reason. Not 'Jim Crow' laws, regulatory capture (huh?), the destruction of unions, or anything else. You seem to have a chip on your shoulder yet you don't know the #1 reason for why poor people stay poor?

-Matt

In fact, the biggest money maker for a high frequency trader *IS* front-running someone else's order. Other profits from HFT algorithms, such as simple arbitrage, are a lot lower than profits from front-running opportunities. I very much doubt that the infrastructure investments HFT firms have made would be worth doing without the front-running component, so it will be interesting to see what happens down the line.

The front-running is possible when the other person's order is placed on multiple exchanges but not delayed so it hits them all at the same time. If the order hits, say, the BATS exchange first and there is a 100ms delay before it hits, say, the NYSE, then HFT algorithms will see the order on BATS and will be able to front-run the order on the NYSE using their superior network connections to the NYSE to get ahead of the original order that is still in-transit to the NYSE.

-Matt

This might be a surprise to you, but it has already happened. The growth in dark pools is directly attributable to major investment managers exiting the public markets and doing their trading in the pools in order to avoid the HFTs and other shenanigans (both real and imagined). The public exchanges have lost a huge amount of business over the last few years... hoist by their own petard, so to speak. At least to a degree.

Insofar as regular investors go, HFT doesn't really have much of an effect so there's no reason to take our business elsewhere. Losing a penny here and there, when it happens at all, is nothing compared to what trading cost us even 10 years ago. Our trades are simply too small for the HFTs to be able to act on.

But if you are really worried, just use limit orders that are slightly outside the bid/ask range (i.e. so they don't fill instantly) and wait for the fill. Nobody can front-run a small order so making it public beforehand prevents the HFTs from being able to do anything with it. You have to be a bit patient, but that's all. (this advise does not apply to thinly-traded names, only to liquid names).

-Matt

Hedge funds tend to be used by the affluent, not by regular people with regular incomes. In addition, hedge funds usually lock your funds up... there may be only a few times a year where you can cash out and you have to give them notice (3-6 months, depending). Only investors with serious excess capital that they can afford to put aside for a few years uses a hedge fund.

And even then, hedge funds are hardly any guarantee of good returns. As a class Hedge funds returns have been horrible over the last year, for example. And while the media likes to hype up the returns that some hedge funds have been able to make, they tend to ignore the larger number of hedge funds that underperform the market.

-Matt

That is total and complete nonsense. Ultimately the stock price is a reflection of the company, not the other way around. If a stock winds up being mis-priced due to an out-of-control trading program (and I've seen that happen plenty of times), it doesn't stay that way for long as investors pounce on it. You should count your blessings when it happens, because being able to buy a stock that an out of control trading program drops $10 in 60 seconds is virtually guaranteed profit.

-Matt

There are a number of good solutions available. If anyone actually read the book, defeating the HFTs basically comes down to adding a delay to multi-exchange transactions such that the transaction reaches each exchange at the same time.

The real problem here is that the regular exchanges prioritized their own profits over their duty to provide a fair market to participants. That much is obvious, and frankly I think there should be criminal prosecutions for what they did (I doubt it will happen though).

This sort of things has happened in other areas. There are several market reports that sell early copies to a select group of clients. One just recently was selling an 'electronic' version even earlier than its main group of clients. It turned out that the HFTs who purchased the ultra-early version were using the data to front-run the normal clients (who in turn were trying to front-run regular investors reacting to the report when it goes public).

The instant it was revealed what the early-early group was doing, the regular clients stopped trading on the early news and the early-early group were suddenly not able to make any money front-running the regular clients. The producer of the report was forced to retract the early-early report or lose *all* of their regular clients who were tired of being front-run.

The stock exchanges are engaged in the same sort of crap with the HFTs, selling them special access and trade types that other investors do not have. Now the exchanges are in a position of having to deny that they are making things unfair when it is obvious that they are making things unfair.

This just goes to show how convoluted things can become. But once it gets into the light of day, corrective action can happen pretty quickly.

If our regulatory agencies were more competent, this would have been dealt with years ago instead of letting it fester as long as it has.

-Matt