Ok, I actually think you, me, and Theo all agree :)

1) We don't think a specific technical change would have _prevented_ the issue.

2) We all agree that better software engineering practices would have found this bug sooner. Maybe even prevented it from ever getting checked in (e.g. suppose the codebase was using malloc primitives that that static analysis tools could "see across", and that the code was analysis clean. Could this bug have existed?)

Who has claimed that using the system allocator, all else being equal, would have prevented heartbleed?

Who has claimed that heartbleed was an allocation bug?

I understand what freelists are and do.

The point here is that rigorous software engineering practices -- including the use of evil allocators or static analyzers that could actually understand they were looking at heap routines -- would have pointed out that the code implicated in heartbleed was unreliable and incorrect.

If you read the link you pointed at, after making a modification to OpenSSL such that coverity could understand that the custom allocator was really just doing memory allocation, Coverity reported 173 additional "use after free" bugs.

There are bugs from years ago showing that openSSL fails with a system allocator.

Don't you suppose that in the process of fixing such bugs, it is likely that correctness issues like this one would have been caught?

Actually, it is you who are wrong.

Theo's point from the beginning is that a custom allocator was used here, which removed any beneficial effects of both good platform allocators AND "evil" allocator tools.

His response was a specific circumstance of the poor software engineering practices behind openSSL.

Furthermore, at some point, openSSL became behaviorally dependant on its own allocator -- that is, when you tried to use a system allocator, it broke -- because it wasn't handing you back unmodified memory contents you had just freed.

This dependency was known and documented. And not fixed.

IMO, using a custom allocator is a bit like doing your own crypto. "Normal people" shouldn't do it.

If you look at what open SSL is

1) crypto software
2) that is on by default
3) that listens to the public internet
4) that accepts data under the control of attackers ... you should already be squarely in the land of "doing every possible software engineering best practice possible". This is software that needs to be written differently than "normal" software; held to a higher standard, and correct for correctness sake.

I would say that, "taking a hard dependence on my own custom allocator" and not investigating _why_ the platform allocator can no longer be used to give correct behavior is a _worst practice_. And its especially damning given how critical and predisposed to exploitability something like openSSL is.

Yet that is what the openSSL team did. And they knew it. And they didn't care. And it caught up with them.

The point of Theo's remarks is not to say "using a system allocator would have prevented bad code from being exploitable". The point is "having an engineering culture that ran tests using a system allocator and a debugging allocator would have prevented this bad code from staying around as long as it did"

Let people swap the "fast" allocator back in at runtime, if you must. But make damn sure the code is correct enough to pass on "correctness checking" allocators.

This bug would have been utterly trivial to detect when introduced had the OpenSSL developers bothered testing with a normal malloc (not even a security focused malloc, just one that frees memory every now and again). Instead, it lay dormant for years until I went looking for a way to disable their Heartbleed accelerating custom allocator.

it's a very good read.

Sinews (aka "tendons") are bundles of fibrous collagen bound together with an organic glue of proteins and polysaccharides. Sinews can be pounded to extract those collagen fibers, and then those fibers can be spun into cordage of any desired length.

The process is exactly the same as spinning short wool fibers into skeins of yarn, or transforming cotton bolls into cotton thread. The fibers are bundled together and twisted so they lock together and the axis of the resulting cord cuts across the axis of orientation of the fiber, producing a very strong thread. As the fibers are locked together into a thread, you continually add more bundles of fiber to the loose end. You finish by tying off the end of the thread you've created, or twisting the thread into a multi-strand rope.

Collagen fiber from sinew is an excellent cordage material, but less available in large quantities than plant fibers. For that reason you don't see sinew ropes. Although such a thing would be physically possible, sinew is a costly material so it is only used in specialized, low volume applications like fishing line and bowstrings.

Primitive people are every bit as smart as engineers who design microchips or airplanes; they just express that ingenuity through materials they can harvest and process themselves.

Well, my first choice would be to use surplus and scavenged materials, like polyester or silk. In the long run as these materials become more difficult to find, I'd go for hemp or flax. Just about any fiber can be spun into a workable cordage. Shredded animal sinew yields extremely strong cordage.

He MIGHT let the NSA do it, OR he MIGHT NOT. That's a credible a statement as anyone could make.

You can always concoct a situation in a scenario where your skills aren't important.

You're a farmer? Seems like your skills would be useful but wait -- what if the neighboring tribe burns all your crops and steals your seeds?

You're an emergency room physician? How will that help you when bandits club you to death in your sleep?

Agreed, but your hypothetical persons with first-hand knowledge of managing large numbers of draft animals is likely to be in short supply *in the stipulated scenario*.

My point is *about* the limitations of simplistic models. In the simplistic model, a computer science major can do computer science -- and nothing else. In the simplistic model you can obtain precisely what you need, which is either a two hundred year-old soldier or a historian who specializes in the logistics of pre-mechanized armies. But chances are *in our scenario* people with precisely such skills will be hard to find as unicorns, and people with CS degrees will be common as muck. So, do you look for a historian, or someone with a degree in a somewhat math-y field who happens to have a little of both common sense and imagination?

This is actually a situation which is less exotic than you might think. When you hire an employee, it's often the case that you've got a round hole to fill and a bin full of square pegs. None of the candidates are exactly what you're looking for, so you have to imagine how the candidates you *do* have might adapt.

Right. And this is different from the pre-apocalyptic use of whatever your academic specialization is, how? You get out of school and you have to apply your ivory tower training in idealized problems to messy real-world problems. Does that mean that the ivory tower training is useless, and that the time would have been better spent just getting real world experience? Of course not.

When my dad had a heart attack, my oldest brother was going into his senior year as civil engineering student. He quit school and got a job selling restaurant and food service equipment. He did very well at it, probably made more money than he would have as a civil engineer. That was mainly his people skills, but his engineering training made him the go-to guy for large projects. You might not think there is such a thing as a large restaurant supply project, but it turns out that if you're opening a new theme park and you've got to figure out how to feed a couple million visitors a year, it's very useful to have an engineer who understands food service.

That's the hallmark of a good engineer. A good engineer doesn't just apply his skills, he finds ways of making his skills applicable.

Please, give me some credit for not being stupid. Anyhow, you're just making my point.

Never said it did.

Not before I put an arrow between his eyes. I can not only shoot a primitive bow pretty well, I could make one, including the bowstring, with nothing but a knife. If I didn't, then I'd have to fall back on my boxing and (admittedly rusty) judo skills.

It's a common misconception that people capable of unusual intellectual feats must necessarily be physically helpless, hopelessly specialized, and oblivious to everything around them.

Naturally, my model would not starve the farmers. The real challenge is figuring out how to stop the bandits who are starving the farmers. Fortunately, you only need about seven samurai for that.

So when you break your leg, you're going to have your witch doctor set it for you?

Vaccines and antibiotics are not high tech -- by which I mean something that requires an extensive and intact industrial infrastructure to produce. Crude replacements could be created by someone with 21st C scientific knowledge and the kind of technology that would have been available to 18th C gentleman scientists.

As for other drugs, a doctor could work with herbalists. Willow bark replaces aspirin; foxglove replaces digitalis; Ephedra sinica replaces pseudoephedrine; absinthe replaces anti-worm medications. A herbalist working under medical supervision is a lot better than nothing.

Which, it turns out, has very little to do with actual computers.

The intellectual skills involved in CS could, with not much difficulty, be turned to other kinds of problem solving such as operations research. Seriously, you're going to leave questions like how to most efficiently distribute scarce resources such as food to someone with a *business* degree? As a computer scientist, I'd create a model of the underlying problem, develop alternative algorithms, then show how those algorithms and model apply the real world problem. I use computer science every time I come home from grocery shopping. As I remove items from the bags I stage them by where they are eventually going to go. Why? Because efficient sorting algorithms eliminate lots of entropy early on. Consequently I only open my refrigerator *once*.

Computer science is essentially about figuring out the resources needed to accomplish things. If you want to figure out how much fodder it would take to move your draft animal powered army over a certain distance, you *could* consult a historian who specialized in the logistics of pre-mechanized warfare who'd tell you how Viscount Howe did it in the New Jersey Campaign of 1776-1777. Or you could find some CS graduate who pulled at least a "B" in algorithms to figure it out for you.

As for experts in gymel -- a technique for singing polyphony with one voice -- it's worth considering that the technique was developed in a period of human history that would be considered apocalyptically awful by modern standards. Even when times are violent, disordered, and desperately poor people still need art and music, and if we're stipulating that apocalyptic == "no computers", that means no iPods either. So it seems quite plausible to me that experts in gymel might find their services *more* in demand in a post-apocalyptic world.