SecurityFocus Responds To ESR Column On OSS Security

RabbitInTheMoon writes: "Elias Levy, moderator of BUGTRAQ and Chief Technology Officer of SecurityFocus.com has written an article about Open Source Software security in response to the recent message from ESR published here. He makes some very interesting points. Maybe this will clear up some of the misconceptions about open source security. "

The other benefit of OSS

ESR forgot to mention one of the strongest sides of OSS: when there's a flaw in the security, you can fix it. And even when you can't there are always others who will do it for you. Even if you read from every damn newspaper on the Earth that a Microsoft product you're using has a backdoor, you can't do anything about it except upgrading to Apache.

Rehashing the same old stuff

This article presents nothing new; neither did ESR's. Both articles simply rehash the same old arguments we've been hearing for years. I suppose these things do have to be occasionally restated, but I just feel the urge to say "bleaagh."

Elias Levy seems to be saying that security through obscurity does work, because he thinks that everyone is lazy/dumb.

Personally, looking at the evidence of the last few years, I'm not so sure I agree with him. Yes, *theoretically* open source software could go unfixed for a long time, but what we have seen is that it seems to be much more common for closed source software to have holes in it for years.

Regarding his point that black-hats will find the holes in open source: who do you think is finding the holes in closed source stuff? I'd still rather have the "window of opportunity" be of shorter duration, and I think I get that with open source.

Open source is not a panacea, but it's still safer in the long run, IMO.

New XFMail home page [slappy.org]

/bin/tcsh: Try it; you'll like it.

Let's take this one at a time

1) Is anyone readining it?

The evidence says yes. There was an attempt to post a trojan in open source recently (I lost the URL- help, anyone?). It lasted _ten_ _hours_ before it was discovered. And this question applies doubly so to closed source- as not just anyone can read it, only employees can read it. And most companies that I know of don't implement any sort of code review, so often a peice of code is only ever read by one person.

2)
Are they qualified to review the code? Some yes, some no. Once again, the exact same question can be applied to close source. Opps, most closed source isn't reviewed, even by unqualified programmers!

3)
It's easy to hide vulnerabilities in complex, poorly documented source code. This I'd agree with- especially _unintentional_ vulnerabilities. On the other hand, it's also hard to _maintain_ such code- and in the open source world, over time it tends to get reimplemented. Sendmail worries you? Try using smail or qmail instead. And, if anything, complex, poorly documented source code is more likely in closed source projects where the assumption that only the original writter will ever see the code, and as they already understand it, comments and simplicitly are optional. Besides, writting comments and refactoring code takes time, and I have a deadline this week...

4)
There is no strong gaurentee that source code and binaries have any relationship. Ah yes, Ken Thompson's paper. Thompson's paper assumes only one compiler, ever. You always compile gcc with only gcc- never Sun's cc, or IBM's xlc. It also assumes that any version of the compiler will recognize any other version of the compiler- so gcc 1.0 would recognize the source code to gcc 2.4, AND be able to insert the back door correctly into the produced binary! It'd also have to recognize the source code to tools like objdump and gdb as well, and insert the proper back doors into _them_ as well. I knew Ken Thompson was legendary- but this defies the imagination.

And, once again, there's no evidence of this on the closed source world either. Prove to me that Visual C++ isn't inserting back doors into Windows...

5) Open source makes it easy for the bad guys to find the vulnerabilities.

I was wondering when this chestnut would show up. The implicit assumption here is that the bad guys can't disassemble code- an assumption proveably false. There was a famous quote an IRA terrorist reportedly once said to Prime Minister Thatcher once- "We only have to be lucky once. You have to be lucky all the time." Security analysts face the same problem- the crackers only have to find _one_ vulnerability, while the security people have to find _all_ of them. You're not making the cracker's lives much difficult, but you're making the white hat's lives all but impossible.

Open source is not a security silver bullet- and I don't know of anyone outside a few anonymous cowards who claims that. Open source _is_ better than closed source for security. No ifs, ands, or buts.

Re:ESR can't have it both ways

What the article omitted is the issue of how quickly bugs get fixed. Certainly patching things on the fly is much easier with open source, and there are alternate ways to get fixes. This, of course, can be a weakness, if you're not careful who you get your fixes from. But overall I think it's a strength.

Re sendmail and the DEBUG (and WIZ) commands: those commands were quite well known in the 1980's, but one has to look at the times to understand the issue in context. Until the Morris worm, people never really paid too much attention to back doors; the Arpanet was much more of a friendly club than the Internet of today. Also, sendmail wasn't really open source back then anyway; the vendor supplied it as a binary (often with proprietary extensions). You could plug in your own version of sendmail if you were motivated to, but you'd often lose something in the process, such as YP username resolution or some such. I still remember binary patching sendmail to turn off the DEBUG and WIZ commands because they were so obviously braindead (no particular kudos for that; just find the DEBUG and WIZ strings and overwrite them with nulls).

I don't think that "more people checking source" = "more out there introducing bugs". Most people who hack the source are doing it for their local site. They may well be introducing bugs, but they're not spreading very far. Certainly they're not likely to propagate back to the master source tree, assuming that the people controlling the master source are anywhere on the ball. Most people also don't need to get open source packages via back channels; it's just too easy to get it officially (most of the major Linux distributions include a ridiculous number of packages). One can argue about where they get their source -- with some validity, at least in principle -- but for players such as Red Hat, SuSE, and the like they employ enough known clued people so that there's a good chance that they're getting their software from reasonably trustworthy sources. Along those lines, I'd be more worried about warez; that always comes from back channels, and there are too many places where it's easy to stick nasties on (just do a rogue installer that installs a virus along with the real package).

Aside from moral issues, the problem with vetting access to open source software is the usual chicken and egg problem. How do you verify someone's bona fides if they don't get a chance to do something in the first place? For the maintainer of a package deciding whether to accept a change it's reasonable that that maintainer use whatever standards he or she wants. For simply inspecting the code, or making local modifications, why go through all this?

Re:98% In A Prog Course != The Ability To Read Cod

Don't worry, I have many years of experience but I still can't just glance at code and know what's going on. Firstly it's time consuming. Secondly, unless you have direction (you want to find out who a small piece works), it's easily to be overwhelmed, especially for bigger projects - you need to apply the "divide and conquer" approach. The smaller your ulterior motive, the harder it is to get into and understand somebody else's code, let alone start finding security bugs. In some ways, the best approach is to review the code with another person... you can discuss it as you go and motivate each other.

We do code reviews at work and they can take hours. That's just small sections of code - no more than a few hundred lines. Looking at somebody else's code takes sometime just to get up to speed.

I got really annoyed with Mozilla crashing on my SMP machine a couple of months ago. I downloaded the source, compiled it and then starting running it through Visual C++. Guess what, after spending a whole Saturday doing that I give up (don't give me a hard time, I know I didn't stick at it long enough to get anywhere useful). It takes a long time, and I don't have the time. I already work 8 to 12 hours a day, after which I want to get away from the computer and spend time with my girlfriend, etc.

So, my point. Although open source means that there is the potential for more eyes looking at the code, does this really happen? How many people just extract and compile, or just install a package? That's effectively no different to closed-source binary distributions. About as close as I've got to the source is when I've had to set a pre-processor define in a kernel header, or when I've got unresolved linker errors building the kernel. The rest of the time I can't be bothered... it's too much effort and I just want it to install and work.

Backdoors everywhere

Depending on how active the development is the code may be found in a day, or a year or even more. No-one knows as this has never been done before.

How do you know?
Maybe it's part of Microsoft's policy to introduce such an innocent backdoor with every program. And maybe some open source superhacker is grinning to himself knowing that nobody noticed his supercleverly-disguised harmless backdoor in the Linux kernel.

Even if we didn't find it, it doesn't mean it isn't there :) .
__

Problems in Levy's article.

Hi Elias,

I'd like to point out a few problems with your comment.

The Gauntlet firewall published by Trusted Information Systems was not an Open Source program. It's what we call "disclosed source-code", and that's very important because that difference means that nobody had much reason to read it or work on it. The software license didn't provide them any incentive to do so - you would have only been fixing bugs in a program that somebody else has an exclusive right to sell. Who wants to be the unpaid employee of another company? With real Open Source, you have the same right to sell the program as anyone else, or to distribute it for free, for that matter, and thus you aren't some company's unpaid dupe. For an explanation of what Open Source is, see The Open Source Definition [perens.com] .

At the time of the Morris Internet worm, the BSD software distribution of which Sendmail is a part was under a restrictive license and required an expensive ATT Unix license before you could get the system. This is also not what we today know as Open Source. Besides, you are writing about the epochal Internet virus, and few people even considered Internet security before that event.

Yes, all compilers have a bootstrap problem. One can avoid it by compiling the compiler with another compiler, once in a while, and then compiling the result with itself. This method can also be used to detect the Trojan: compare the generated executable with one that doesn't have another compiler in its heritage - if there's a significant difference, look for a Trojan there.

Most users do not compile their own applications, but they get them from a trusted source who has compiled them and cryptographicaly signed them. You might not be aware that in all Linux distributions of any import, the packager does compile all programs. If there is a trojan slipped in, you can trace it to the person who compiled the program and bring charges if necessary.

And what good would it do anyone to grep through source code for strcpy()? We've already done that ourselves, and have fixed obvious problems.

Sure, it's no guarantee, but it's much better than the alternative, which lets Microsoft embed snide comments (if they really aren't trap-doors, embedding a trap-door would be as easy) in their software and have them undiscovered for years.

Thanks

Bruce Perens

All about nuances

In the end they don't really disagree; where ESR says Open Source security would be better almost by definition, whereas Elias Levy notes that the 'open source way' is potentially better but it all depends on the many eyes actually watching/being able to watch for harmful code (the cc malicious chicken and egg problem aside; but proprietary software doesn't seem to offer any advantage here).

Then again, there's mostly a small team of hard core developers for any open source project, and especially the (mostly technical) security related stuff. Any nasty stuff would probably have to be done by someone inside such a team. An outsider trying to submit something 'bad' would very probably be noticed by one of the core members.

So if, as a user, you don't want to code everything yourself, it all comes down, I think, to trust. The only question in the open source vs proprietary case is: who do you trust more in the end, a proprietary developer team or an open one.

Security by Obscurity could maybe temporarily help the Proprietary case, but the 'Exploit-Found' scenerio would always turn out better for Open Source: more fixers, hence quicker fixes. And the fix-it-yourself option.

Re:Effective Security.

I disagree though. It does raise the bar for the average "black hat" to create a _viable_ exploit. While it is mindnumbingly dull for most "white hats." Sure, you can look for certain key strings with a hex editor, but it is not comparable to looking at the entire code and seeing it in context. Certainly if you look at the actual number of published exploits for NT they are relatively few. So we must either conclude that your average black hat has a number of NT exploits which the informed public is unaware of, or they simply don't have it. In either case though, a) neither the admins nor microsoft is doing much about it b) Microsoft gets away with it because the hacking incidence of NT isn't much worse than Unix (and some would argue better) c) a relatively small group of intelligent and motivated individuals can punch holes in almost every NT box on the internet. In other words, the possibility of a highly sucessful systematic attack against supposedly secure NT installations is not exactly outlandish.

Re:Effective Security.

The recent DOS attacks involved large numbers of infected Linux systems being controlled remotely to help with the attack. Look int he recent news for articles.

I am differentiating between the Linux kernel and the distributions. I fully realize that the actual implimentation of Linux in the various distributions is less than secure. I'm asking you to enumerate what bugs specifically in the kernel were discovered by blackhats before the whitehats. The mainstream press is useless when it comes to technical details such as this.

There is no such backdoor, and the continued insistence to refer to it as such simply points out the FUD factor in the Linux world.

Umm, no. The backdoor exists, just it was not all it was cracked up to be. I believe I alluded to this in my previous comment too. Nonetheless, it is not terribly relevant to my point. Microsoft screwed up. The public did not notice it for years. The degree of the severity of the bug is essentially irrelevant, it was certainly significant enough to get noticed. It certainly casts doubt on Microsoft's auditing practices.

As for being a member of the Linux FUD community, my record speaks otherwise.

FYI

FYI: http://www.techweb.com/wire/story/TWB20000417S0001
hardly irrelevant, or non-existant.

Effective Security.

As far as the Open Source advocates go, I generally find ESR the most levelheaded. ESR is probably right insofar as a blatant backdoor with "Netscape engineers are weenies!" would never escape scrutiny in something such as the Linux kernel. However, his claims were a bit too broad to be digested meaningfully by the masses. Levy addressed ESR's claims. Levy was not claiming "security through obscurity" in and of itself is sufficient. Quite the contrary, he said that many black hats can operate a hex editor and find bugs that way. What he did say was that closed source can offer a significant obstacle to discovery of trivial bugs by black hats. Although it might be obvious when you think about it, many Open Source people hold it as an article of faith that if you take the same source, any source, and Open Source it, it automatically becomes effectively more secure in, say, 6 months. This is simply not the case when you look at the empirical evidence. In other words, if you own some source code to an application, "opening" your code may hurt or it may help your effective security. The change in security is contingent on the specific situation.

For example, do you really believe that Mozilla is any more secure than Netscape? It obviously contains hundreds of thousands of bugs still. Open source has yet to resolve even more obvious stability bugs, so I think it is reasonable to assume there are significant security issues there as well. Not enough qualified people are truely spending the time to examine and fix it. So what we have is source with bugs, but a situation where any blackhat hacker can run grep/sed/awk/perl/etc on it to look for trivial bugs. If this same source were closed, it _would_ raise the bar for creating a viable exploit significantly.

On the other end of the scale, we have something like Linux's kernel. Thousands of qualified people really do work and look at the code. The size is managable. The code is easy to understand. The code is modular. All this works in Linux's favor. I sincerely believe the Linux kernel in and of itself (e.g., not the thousands of binaries that come with Linux distros) to be more secure than NT's.

To make a long story short, the change in security is contingent on the situation. That being said, I do think Open source affords significantly improved security against highly systematic attacks against dedicated attackers. The more reviewed Open source code (e.g., Linux) is at the very least a moving target. The odds of a single blackhat exploiting a bug en masse before the thousands of white hats can close it is quite slim. In other words, although Linux and NT may appear equally secure today, this is just against your average black hat. Your average black hat really isn't all that intelligent or motivated. So Microsoft can afford much less secure code due to their closed source nature and still maintain apparently equal security. Some organization, let's say the KGG, could throw enough brains at Microsoft binaries to create a program to silently scan and backdoor every Microsoft server on the internet, using this as a gateway to more sensitive internal company data (e.g., many companies have worthless firewalls)...A few admins may notice something foul, but many don't fully understand the security model. Fewer yet have the skill to reverse engineer such an attack even partway. And virtually no one other than the big bad evil blackhat group would have the resources or the time to create a working exploit. Consequently, Microsoft can never be made to look sufficiently foolish to force them to do anything. Operations cannot shut down just because of suspected bugs. It would continue getting exploited.

The bottom line is that apparent security (e.g., the number of known NT exploits vs known Linux) and ultimate security (in scenario's such as the one described above) are different....

...gotta run. bye

Actually, yes he can

whereas it's actually "review by any boob that can manage to do ftp:"

Yes, any boob can review it. So what?

Furthermore, the more people there are checking the source, the more there are out there introducing bugs, for the hell of it.

Ah, I see your problem. You are assuming that since "any boob" can review it, that means "any boob's" modifications will be fed back into the main source tree. Fortunately, you are mistaken. The decision to accept a change is made by the project's maintainer. It's not, as you seem to imagine, some kind of free-for-all where anyone can change anything.

It would seem far more useful to me if source was "open" in the sense that you could get a copy of the code on production of a reasonable description of what you planned to do with it, what improvement you wanted to make, plus at least two references from people who were prepared to establish your bona fides. Kind of like the criteria for getting a reader's ticket to a law library.

How could that possibly be, as you say, "more useful"? Why would you ever want to place any restrictions -- at all -- on who is allowed to know and understand things? If all of those things are needed just to read about law (not just to legislate and judge it), then it sounds like the software profession is far more advanced and reasonable than the law profession. You should be copying us, not vice-versa.

Restrictions should only be placed upon those who create and modify software (and that's the case, with both open source and closed source). With open source or custom software, the end user has the final decision on how strict or lax those restrictions are. With shrink-wrapped closed source software, the end user has no control of those restrictions, or even knowledge of what those restrictions are or if they even exist.

---

Peer review vs. security audit

Peer review of software is not as crucial as in physics. Everyone wants to check the theory of cold fusion or the proof of Fermat's last theorem because they are used as building blocks for further research. When Fermat's last theorem was proved, it opened up more possibilities. Software can also be used as building blocks but new source code almost never cause a revolution. People also prefer to reinvent the wheel instead of reusing source code.

However, open source software can be audited and that's what some white hats do. The Linux Security Audit Project is actively searching for holes by reading the source code. This includes lots of gifted programmers who can smell a hole from far away.

I'm sure that commercial software is also audited inside the companies but close software gives you false sense of security. It's easier for you to make sloppier code and leave temporary holes because "nobody knows about them anyway." But if you know that bad guys are going to read your source code and exploit it, you really concentrate more on security. And even if bad guys are not going to read your code, you don't want to be laughed at for leaving holes the size of Titanic.

But while peer review is not as common as many think, it doesn't mean that it's useless on unexistant. How many of you have actually checked Andrew Wiles's proof of Fermat's last theorem? Only a handful of extremely intelligent and gifted mathematicians can do that and have done it and that's enough for the whole "community" to trust the theorem. So even if only a handful of programmers check important source code, it's enough if those guys are as gifted as Linus Torvalds or Alan Cox.

Liability shift?

If I download, say, a GPL'd firewall who is legally responsible when it let's through an attacker?

My common-sense approach would be to blame the network admin unless it was a bug of such an egregious nature that it constituted negligence on the part of the company. Many products (see: Microsoft) have just these kinds of bugs in them. So a smart network admin gets something Open Source.

Could a sufficiently crafty company claim that, since you have the source, certifying that it is secure and bug-free is up to you? And, if so, will we see companies moving to Open Source releases for liability protection?
--

What about FreeBSD and TrustedBSD

The article does not mention FreeBSD [freebsd.org] or TrustedBSD [trustedbsd.org]. Both of these make a big thing of security, including reviews of software. TrustedBSD is even going for Orange Book B1 certification.

Paul.

OK, all together now, let's repeat the obvious...

Yeah, this article says nothing we don't already know.

• "Just because the source is available, doesn't mean anyone is reading it."

Yeah, no kidding. But when the source is closed, I guaranteeno one is reading it. Just because the author and many of the developers he knows are not over-conscientious and don't read the source doesn't mean that there aren't many of us out there who do. I personally review the code to almost every piece of software I use regularly, to the best of my ability. Yeah, I may not be "qualified" to "judge" something like Sendmail, but at least I can have the piece of mind that its developers are not trying to pull a fast one on me, as Microsoft did. If I don't feel "qualified" to judge some code, I reread it until I am. Maybe that's just me. No wait, that's not just me -- that's a lot of people out there, and that's why open source works.

• Open Source makes it easy for the bad guys to find vulnerabilities.

And leaving your car parked in a parking lot makes it easy for car thieves to find. What is that supposed to mean? The issue is not, and never was, the "bad guys" finding vulnerabilities. Last thing I heard, the bad guys find vulnerabilities in closed source stuff all the time. The issues are prevention and the ability to fix bugs as they are noticed. I can fix the bugs myself if I find them, I can apply patches myself, I don't have to wait for a new version or a binary patch to replace the compromised DLL's or shared library.

darren

Cthulhu for President! [cthulhu.org]

Re:Backdoor Test?

This has happened with sendmail wu-ftp several years ago. It was released as a "new" version and it was even distributed from the wu-archive server.

Re:Who's this "you"?

MS's hotfixes and service packs are also free.

Unless they are sold as upgrades like Windows98/2000 :-)

Sorry couldn't resist. Actually I think Streetlawyer has a point.
"Open source" does not help you if you can't (or dont have time to) understand the source.

Most of the time "Open source" is not worth a shit without a maintainer. (Nor is closed source SW BTW)

I must know where to get the *latest* version, I must know where to find *reliable* developers. I must know how to handle "Fix it *now*, cost is not important" situations.

I need a solution that does not require an expert for daily use

And "I" in this case is anyone who is computer savvy, but makes a living with running computers, not by making them run.

Some OSS fulfills those needs, some commercial SW does too. Neither open or closed source rubbish does.

Re:The other benefit of OSS

With the same day's workaround being "delete the file and the functionality it comes with".
That would be much like "yes, we're aware of the latest sendmail root-shell exploit. The fix is to rpm -e sendmail and lose your mail".
Fixing a security leak caused by bugs can take a while (needs to be tracked down etc.) - fixing an INTENTIONAL BACKDOOR, especially if you know what to look for (such as the weenies string) is a matter of a few minutes, and I don't see why they didn't release a fixed dll.

Let's get this straight

Sure, the source code is available. But is anyone reading it?
They can or cannot. But if you don't build they won't come. Today how many programmers work with open source? It's kinda like saying in the early years of the car that it's better to have a horse, since with a car you didn't had roads to go everywhere. True, but this doesn't mean that the concept of a car is wrong. We can get to a point that there are more OSS than programmers, but as more and more companies adopt OSS, and put theis programmers to review the software they will use, this problem will be solved.

Even if people are reviewing the code, that doesn't mean they're qualified to do so.
True, and this is different of closed source how? But them you're comparing the knowledgement of the security review team (if there is one) on the company to the knowledgement of the entire world. Where do you thing would lie more people prone to find errors?

It is easy to hide vulnerabilities in complex, little understood and undocumented source code.
Yep, right again. But remember Netscape. How many years have they spend with a codebase hard to work with and extend? Slow and insecure? Why did this change when they openned the source? Because new people, excited to contribute needed something simples. Have you ever worked in a big software company, with 1st layer and 2nd layer managers? It's "do this, fix this and make it work". Building a house in sand.
People joke that Microsoft doesn't want to open Windows because people would laugh at its code. Truth is, OSS is different, you don't have so much tight control as in a company. Some people say to not mix code and politics, but we must, to understand OSS's potential. Understand how different things are in a company and in a mailing list. What would happen if Netscape decided to not use Gecko? Think about that. I was reading about a table bug in Netscape 4 that was discovered in Netscape 1!

There is no strong guarantee that source code and binaries of an application have any real relationship.
Bullshit. Trusted sources... The example given here was to an extreme. (it is, for me, the greatest hack I know of). If you don't trust the source, just compile the source. Truth is, we need better tools for software deployment. Ok if a bug is discovered a patch is issued in days. And then what? How many people among the users will issue the patch? Most of them will wait till the next version. We need something down to the OS level that could automatically update software and libraries from a trusted source. But then again, this is a problem with most programs and OSs.

Open Source makes it easy for the bad guys to find vulnerabilities.
Bullshit. Most of the vulnerabilities are related to input, when the software comunicates with the world. When it reads a file, accepts a data packet, etc. This in any software. You just focus in the 1% of the code that has something to do with external data. Data validation to avoid buffer overflow, invalid commands or characters, etc... This in an open or closed software.

But make no mistake, simply being open source is no guarantee of security.
Did anyone say that you just have to open the source to be safe? You mostly touched points that are related to both systems, but are easily addressably by OSS. It doesn't mean that all the open software out there that is open is ineherently better than a similar closed one, but that is uses a better method of development and debug proccess.

--MSM

Backdoor Test?

One interesting, and usefull, thing to do would be to intentionally put a harmless (say deleting a specific file that has almost no chance of existing like /usr/adfasdf.txt) peice of malicious code in one of the large open-source software packages such as Apache or Samba. Depending on how active the development is the code may be found in a day, or a year or even more. No-one knows as this has never been done before. But someone should try, if only to test if the usual security through peer-review will work at all.

After that a similar test (perferably a whole bunch to be statisticly valid) on some closed source software would be in order. Any MS programmers here?