I should have thought things out a bit better with the stddev example - and realized that it does indeed have a reasonable closed form. Good catch.

Complex data mining is hard everywhere, that's true. The problem is that even straightforward data mining is hard once the dataset sizes reach into the hundred-millions or billions or trillions in size (implying absolute dataset sizes of terabytes or more). For google it's webpages, for biology labs it's sequences.

The big killer is the cost of transferring data, which is how traditional data systems are built. A remote host has some software set up, and you send it some data, and it processes it and returns it to you. The distinction with Hadoop is that you keep the data on distributed hosts and send the code (which is typically a lot smaller).

The point stands that incremental update of queries on mutation is not a generally solvable problem: it'll still require the addition of new constructs and the limitation of existing constructs in SQL (e.g. ordering). Hadoop approaches the issue from the other end of the spectrum: focusing on a framework that models distributable algorithms directly using a small set of primitive operators (specifically, "map" and "reduce").

-Laxitive

In situations where you are using Hadoop, your "primary" data store should BE the HDFS store you are using to analyze it. That's a big part of the actual efficiency proposition of Hadoop.

The big trick with the "big data" approaches is to recognize that you keep _everything_ distributed, _all the time_. Your input dataset is not "copied into the system" for some particular analysis task, it _exists in the system_ from the time you acquire it, and the analysis results from it are kept distributed. It's only at specific points in time (exporting data to send to someone external, importing data into your infrastructure) that you should be messing around with copying stuff in and out of HDFS.

-Laxitive

There are some serious technical challenges to overcome when you think about actually implementing something like this.

Take something like "select stddev(column) from table" - there's no way to get an incremental update on that expression given the original data state and a point mutation to one of the entries for the column. Any change cascades globally, and is hard to recompute on the fly without scanning all the values again.

This issue is also present in queries using ordered results (as changes to a single value participating in the ordering would affect the global ordering of results for that query).

The issue that "Big Data" presents is really the need to run -global- data analysis on extremely large datasets, utilizing data parallelism to extract performance from a cluster of machines.

What you're suggesting (basically a functional reactive framework for querying volatile persistent data), would still involve a number of limitations over the SQL model: basically disallowing the usage of any truly global algorithm across large datasets. Tools like Hadoop get around these limitations by taking the focus away from the data model (which is what SQL excels in dealing with), and putting it on providing an expressive framework for describing distributable computations (which SQL is not so great at dealing with).

-Laxitive

The codename for this processor, was "Ming Mecca".

-Laxitive

Well, not really. It's the same as operating systems 'overcommitting' memory by giving each process a full virtual address space and filling it on the go. Operating systems solve this problem by... well.. using paging.

The paging approach works well for systems where you expect the in-memory working set to be tight. Mainly you'll see a graceful degradation in performance as you actually start hitting real memory limits and paging comes into effect.

Eventually, I think that can be resolved by taking a hybrid approach: wait until memory pressure builds and paging hits performance more than you'd like, then auto-migrate machines off the host as necessary. You get the best of both worlds: oversubscription when resource usage is low and performance is not affected, and on-demand resource allocation when resources are known to be needed.

-Laxitive

You will not be fired for exercising workplace rights. You'll be fired for "not being a team player".

-Laxitive

It's not that 3d user interfaces have been fully explored, but that simulated 3d interfaces on 2d desktops have some fundamental limitations. We already have some amount of simulated pseudo-depth: windows can lie on top of other windows, etc.

The problem is that by the time you get around to interacting with something, you're interacting with a 2d euclidean plane which presents a projection of some 3d model. It doesn't make the plane 3d. You can't reach around and touch the "middle" of an 3d object projected onto a 2d plane. That's a problem. These might be somewhat ameliorated by true 3d interfaces (where the display itself is 3d), but that tech has yet to mature.

If you think about it, even the way we work on our typical desk is mostly 2d, from a topological perspective. I have a pile of papers and some random crap lying around my desk. When I go to grab a document to work on, I don't just reach into the middle of a stack and pull out the right one. I don't have that capability. I need to go and start flipping pages, basically morphing my 2d topology to reveal some object hidden in 3d, and only then interact with it.

That's not to say that all 3d effects and stuff are useless. Simulated 3d is a great way of providing visual cues that we have been training ourselves on since we opened our eyes. That can be a very important aspect of intuitive interfaces.. but fundamentally it acts as a visual highlight. The goodness or badness of any particular 3d interface depends entirely on how effectively the _2d_ projection is.

Thirdly, "true" 3d is actually too limiting. We are forced to live in a 3d world, but our computers give us access to many more dimensions, weirder dimensions, than that. We can provide 2d projections of abstract non-fixed-dimensional objects, like n-ary trees (e.g. filesystems). An example of a projection of that abstract object to a 2d interface would be spotlight. It provides a 2d textbox which behaves in strange and weird ways - a 2d textbox that projects 2d manipulations (type some characters), into an arbitrary traversal of the tree. Compare the utility of that to the utility of a "true" 3d rendered filesystem. What value would that add? Sure, it would look neat, but what extra thing would you gain from it?

There's nothing magic about 3d. Computers operate above and beyond limitations of 3 dimensions, and are currently constrained to expose their behaviour through primarily 2d interfaces. Simulating 3d on top of 2d user interfaces, aside from the "visual cue" aspect, is kind of an arbitrary choice.. not necessarily the best one.

-Laxitive

"How cool would that be?"

I don't know. Depends on how good the movie is.

I see that you have spent some time developing deep experience with strawmen. Good for you.

Anyway, I'll go ahead and give you an A for effort. Let's play smug one liner vs. amused mockery again sometime :)

-Laxitive

Ah, I see, you must be referring to the Delhi high court's support of caste discrimination. I'm having a bit of trouble finding examples of that, though. Could you point me to some of your examples?

I tried searching for '...' in google, but it doesn't yield too much in the way of results relating to the delhi high court.

Thanks in advance!

-Laxitive

I'm sure if you lend the Indians your time machine, they can go back in time and fix that issue. Until then, I guess they'll have to just live with outlawing caste discrimination in the constitution and then slowly working to change public attitudes.

Or perhaps you've discovered a way to fix the issue with smug off-topic one-liners?

Do tell. I eagerly await your insight into the issue.

-Laxitive

In experimental science, this is not uncommon. Using different methods of analyzing the same subject is, in other words, using (relatively) independent methods to analyze that subject. Using multiple independent methods and combining their results is a good thing, because it avoids experimental error and potential systemic biases that exist in every observational setup.

That said, I don't want to get into an actual discussion about the actual paper in question because I have not read the relevant hacked personal e-mails with their full context and interpreted their significance (and I likely won't have time in the near future given the pressures of day to day life). I am not particularly inclined to start implying conclusions and accusations based off of an incomplete and shoddy reading of a few out-of-context paragraphs. I am neither willing to vouch for or defend, or attack a particular piece of research until I am reasonably well informed about how that research was conducted.

There seem to be many people, however, who are willing to do exactly that.

-Laxitive