I do agree that a language-to-language rewrite would yield impressive gains... but that's not the whole of it. Cassandra is an edge case
C++ isn't necessarily the best choice for everything, just like a Mclaren F1 isn't the optimum choice to pick up groceries. But if your requirements dictate that performance is a chief priority, it most certainly is.
I've written many Java and C++ systems at scale. Java simply does not excel at maximizing the use of system resources, predictable real-time performance nor high uptime. Stomp your feet all you want and pretend it's not true if you like. Java trades off performance to provide features to developers that they cannot override. Fact.
Java is fine for 99% of most everything ever written. Honest. Cookbook blog: great! That 1% though where every bit matters, that's when you take off the training wheels and code as low as you can tolerate or afford.
What Java zealots in the Cassandra and Hadoop communities kept boasting was the idea that vertical performance doesn't matter anymore. Solve all of the problems with JVM unreliability and poor performance under the umbrella of big data and more hardware. This makes sense at a few dozen servers. It's insane when you start considering scale at 100s or 1000s.
I hope DataStax considers making Cassandra more cost effective. The simplest way is to get rid of the JVM and give me a machine code binary. I'd really like to throw 128GB of RAM to my nodes, but Java won't let me.