Interbase Backdoor, Secret for Six Years, Revealed in Source

Diesel Dave writes "CERT Advisory CA-2001-01 announced today that the Interbase server database contains a compiled-in back door account. The thing is, it was not the result of a malicious code infection, but a direct addition by the original Borland/Inprise authors done before the program was released as open source." The backdoor was installed sometime between 1992 and 1994, and has been included in every version of Interbase during that time.

Re:Why the surprise?

If you feel so strongly that every open source program should go through a security audit, then when is the last time you volunteered to do one? Opensource is about people volunteering their time which is often in competition with their real jobs, lives, families etc. In a perfect world, all software would go through a security audit, but it is not going to happen.

At least with opensource, things like this get found. Obviously Borland's security audit didn't find it when they originally released this as a commercial product! If it wasn't for opensource, this would probably still be being silently exploited by the original programmers and the few people they told.

Re:Why the surprise?

Borland was relying on security via obscurity on this one. I don't know why no-one took this up as an issue. Perhaps I will volunteer to security audit this code (it doesn't look like much) but I am honestly of the belief that there are companies out there relying on this software to run their business. Surely they have a responsibility to contribute back to a project that they are making money from. So if you're a company and you give half a damn about security, take some of the responsibility and pay for a security audit on the source! It's in your own interests.

Re:Why the surprise?

OpenBSD has been undergoing a security aduit for years. A couple months ago they were able to claim there had been no known root hacks in the current release for 3 years. (That is they were able to fix root hacks before they were discovered for the last 3 years). Well sometime this summer someone discovered a root hack in the released system, despite all those audits. (To be fair, they had fixed that hole in the unreleased code stream, nobody realized it was exploitable at the time though so there was no hurry to release it early).

Audits are good, but they take time. OpenBSD has proven they take a lot of time. There is no open source project with as much work in security auditing as openBSD. (Probably no closed source project either). No open source project cares are much, yet they can't always get it right despite 5 years of work. To criticie any other project for not discovereing all secuirity holes is a mistake. Even if the openBSD audit team had decided to work on this with as much effort as went into openBSD there is no reason to belive they would have discovered this sooner.

root=backdoor? (was Re:Open source = no backdoor)

A back door can be good thing on the local level. ie a sysadmin who can unlock a workstation even when the user has forgotten the password.

By this (implied) definition, 'root' is a backdoor. If I accept that definition, then this becomes a question about the 'domain' of a backdoor. I.e. how many people should know of the existence and details of a given backdoor, and how 'editable' is the backdoor.

In the case of root, the existence of the backdoor is well known, but the details (password) are nominally only known by a few people. On some systems, the 'root' name is changed to something else (e.g. toor) for obscurity reasons.

In the case of Inprise, the existence and details of the backdoor were known to external persons (developers) but unknown by the actual user and the details are unchangable without source code. (note: it looks like a quick fix here would be to edit the backdoor details in the source and recompile). This was entirely 'security by obscurity' and, now that the cat is out of the bag, almost every user of the software is at risk.

Point to be made here: Opening the source code simply made it much easier to find the backdoor. Overall, I think that this is a good thing. There may be some hackers out there who knew of this backdoor for many years. Now we have the knowledge and impetus to clean it up.

I don't think that this was a malicious backdoor. The design of the software seemed to require it (oops!). The big mistake is that nobody who had access questioned it's existence. The lesson to be learned is that people who have access to source code and see this sort of stuff should make waves to open up the process.

The best gemeric solution is to remove the need for internal 'backdoors' in code. That being infeasible, the software should be changed so that the details of the backdoor are editable by the end-user (or randomized on every start of the software). Obviously, the user has to be made aware of the need to edit this data. That solution, of course, has its own security implications (exercise for the reader).
`ø,,ø!

Re:Open source = no backdoor

I have two machines linked together by an crossover ethernet cable. Can you hack into that network? I'd be impressed if you could

A fairly simple manner of splitting the cable and installing my own junction, or attaching my laptop to one of your machines via a serial port /joke

Anyway, as soon as I saw your comment, I got into your master server (which I noticed connected to the Internet on 127.0.0.1 hah!!), and have told the police about your massive pr0n and war3z collection! You should now notice your hard disk is thrashing as my rm -r * takes affect suX0r!

Whoops! Hangon? Why is MY disk thrashing ... aargh!!

Re:Hits on port 3050/tcp already on the increase

Which is, of course, the complete opposite of what you said.

Which is why I like /. comments, because no mistake ever goes uncorrected. I had assumed from reading the security notification that the password was placed in the source just before it had been open sourced. As the yanks say, my bad. It was placed in the original program years ago, but only opensourced one year ago, and that was what led to the backdoor being discovered, I've got that now. I wonder how many people have taken advantage of this over the years.

the AC

Here's a buffer overflow

Well it took 20 minutes but if you grab the file interbase/qli/dtr.c from the firebird cvs you will see one of the very first things it does in main is:

SCHAR home_directory[256];
...
#ifdef UNIX
/* If a Unix system, get home directory from environment */
startup_file = getenv("HOME");
if (startup_file == NULL)
{
startup_file = ".qli_startup";
}
else
{
strcpy(home_directory, startup_file);
strcat(home_directory, "/.qli_startup");
startup_file = home_directory;
}
#endif

That's called a "buffer overflow" and I doubt it is the only one. Just a short grep over the files gives an idea here. 642 strcpy's, 139 strcat and 945 sprintf's. The first thing to do is replace those with safe alternatives (strncpy, strncat, snprintf) and then the fun begins. And I just know that next week I'm gunna be asked to install an Interbase server :)

Are there any *good* choices for Interbase users?

Anybody running a pre-open-source Interbase seems to have only really unpleasant choices:

• Use a binary-only patch (if it's even available for the version they're running) which fixes the problem and trust that they really did remove the backdoor and didn't just replace it with a different one (which I know I wouldn't be willing to do given the fact that they put it in there in the first place)
• Spend an unknown amount of time and effort (and as we all know, time = $) to update to a new version which they know can be trusted (because they can compile it themselves)
• Switch to a different database altogether
• Leave it as-is and hope nobody notices

I'm glad I'm not in that position.

Re:https??

Turns out that a plain http [cert.org] transfer works as well.
-russ

Re:Dogma

Notice how many years it took ANYONE to discover this.

Correction: how many years it took anyone to discover and announce this. Just because it was only now announced doesn't mean someone didn't know about it two years ago and kept quiet about it.

Re:Why the surprise?

The bug in question was a one byte overflow in ftpd. The guy who invented [pulhas.org] one byte overflows had this to say:

Conclusions could be drawn from this nearly impossible to exploit situation.
Although I would be surprised to hear of anyone having applied this technique
to a real world vulnerability, it for sure proves us that there is no such
thing as a big or small overflow, nor is there such thing as a big or small
vulnerability. Any flaw is exploitable, all you need is to find out how.

So even he didn't think this would ever happen and the bug in ftpd was a direct result of this. No one knew it was there because no-one knew that such a bug even existed (and if it did it was most probably not possible to exploit). That is definitely not the case here. This is an obvious flaw in security written by a programmer who obviously never thought the code would be open sourced. It should have been one of those things that you picked up on the first day and said "this is bad, you never should have done this."

Re:Here's a buffer overflow

OK, first a comment that you keep saying that it only took you 20 minutes to find the hole. Yet buffer overflows are well understood and strcpy and strcat are obvious red flags (sprintf does not necessarily mean buffer overflows with correct format strings). As you've shown, a quick grep will give you some clue where to look. You could even almost say that use of these functions is an error in the code. Yet the backdoor you are berating people for not finding is not an error, it is deliberately written into the program, more than likely using perfectly valid code. To find that backdoor mean understanding that piece of code and probably large pieces of code around it and, since you don't know in advance where to look, that means that the whole of the code has to be understood (though not necessarily by one person) to be sure there are no backdoors. Even then, if the understanding is held between more than one person, there may be an interaction between the parts which results in an unlocated problem.

Add into this that this will be a HUGE source base with many many lines of code, that open source contributors generally want to produce things and not be reading over other peoples code and that reading other peoples code (and that usually includes the "you" from >6 months ago) sucks sucks sucks

But those criticisms aside, it does indicate that open source probably does need to consider security more. Especially when inheriting code from closed source projects but just as importantly for exisitng open source projects. It seems that openBSD is doing a good job of auditing their code. While I wouldn't even think of saying that open source projects *must* do x or y, perhaps a central security auditing helping project which ranks other projects on their security and offers suggestions on common security errors and auditing methodology. Projects could apply these techniques or not as they desired but the end user could check the security status by going to the security site. Interbase would have been ranked red_unsecure_not-yet-audited, sendmail could be blue_unsecure_script-kiddie-heaven etc.

My second comment is more a query. Are there header files available which make sure that strcpy and friends can't be used? It would go a way to helping if you could use these headers and WARNING:STRCPY USED. COMPILE ABORTED would pop up as appropriate. It wouldn't be a final fix but it would help and might get programmers out of the habit of using these awful functions in the first place.

Finally, with the front page story yesterday being about OOP, this is clearly the kind of thing where OOP helps. A good string class will take you a long way. Also, OOP is more easy to read and understand in small chunks so it's easier to audit (and easier to get people to audit)

Rich

Re:Why the surprise?

Hmmm... While I agree with the idea that perhaps more people should be checking out the source code of the open source apps they use, I think you missed the point.

The backdoor was introduced in the commercial version of the software. It's only now that it is open source that we could even see the error. The people paying for the 'presumably...high-quality app' you extoll the virtue of were receiving the backdoor-enabled product. Rather than being a failure of open-source software, I'd say this one was a sucess. I only wonder what other kind of 'crap' exists in all those apps whose sources are closed.

Um, how do you know it worked?

Just because you didn't know about the backdoor doesn't mean that some cracker didn't know about it.

Re:Are there any *good* choices for Interbase user

Hi

Have a closer look ;-)

The code is intialised to the variables in the .h file, and when the server starts up it repaces them with random data using chars with ascii values 1-255

So every time the server starts up you get a different random password.

I've posted somewhere else, a bit about how this was done just prior to christmas, to fix the problem, and not introduce any unknowns.

A more perminant fix will be applied, we found it when we were doing a review of the security

There are problems, but in Firebird we have several people who do crypto/PKI things for their day job and we were doing a security review, that in part explains how we've found these. It also places us in a good position to fix these things. As far as Borland are concerned, they seem to be ignoring us,

They wouldn't tell Jim they were working on a patch for prior versions of InterBase, so he felt compelled to write his own.

But for now it's a good time to keep your Firebird/InterBase server locked behind a firewall

Cheers

Mark O'Donohue
--
Your database needs YOU!
http://firebird.sourceforge.net

Re:Open source = no backdoor

Is there a good use for back doors?

I can't think of one. The CERT advisory makes it sound like this particular one is there because the design of the system requires it:

It turns out the LOCKSMITH is an entity needed to allow "authorized" interaction with the security accounts database between services. This LOCKSMITH is the user account in question compiled into the code with full-access to the security accounts database by default.

So, at least it doesn't seem to be a Borland/Inprise employee being sneaky. But still, leaving such a gaping hole in the software, even by design, it stupid. Especially considering the password for said account is hard coded! I can't imagine that idea passing the giggle test for any security expert.

Recent MS break in?

Borland was able to keep this secret for years, or the developers of borland.

Since the source was released, it's obvious that the developers that added the backdoor have already left borland, since it wasn't removed, and the other developers haven't noticed that there is a backdoor.

So, If it can go undetected even if the whole world has access to the source. So might this indicate that there is a very certain possibility that the crackers who broke into MS DID backdoor the source?

Extra info

Some extra info (mostly non-technical, but detailing the discovery and subsequent Borland (non)response) is available at the Interbase Developer Iniative [interbase2000.com].

BTW, it seems that, as usual, they were not very concerned.

Re:More juice ... I like this part

From the webpage [interbase2000.org]:
For security reasons, the patch is available only as a binary and you will be required to register for this download.

Nice, eh?

M.

what did you expect?

Most of the old school software houses have compiled in some back door or provided an hidden way to get access to users systems all over the years. In my opinion it's common practice. They just love to have this kind of control/power over consumers.
Loosing this kind of control is one among other things that make industry afraid of going open...

Other Borland Products

Makes me wonder how many back doors are there in other Borland's products, specially those intended for app development. Is it possible that a back door could be compiled into every Delphi/C++ builder/Jbuilder app ever written, or at least the apps compiled with Standard versions, which don't provide the source of the libs?
Has something like that ever happened before?

Re:Open source = no backdoor

Oh bullshit. There are security flaws found all the time in Open Source products, many of them quite old. If careless coding can create a security flaw on accident that can slip past so-called "peer review", then certainly a reasonably intelligent person could slip in a very subtle backdoor that is infinititely harder to detect. About all you can really say generally about Open source security is that an ultra-trivial backdoor opened with a string like "I AM BACKDOOR" is unlikely, because even the casual reader it apt to notice.

Re:Backdoors vs. default passwords

On the contrary, there is a huge difference. The default passwords are documented, and easily changed. This backdoor was undocumented and would require a recompile to change.

Of course, any computer is only as secure as its administrator.

Re:Hits on port 3050/tcp already on the increase

...and why it existed for years in open source before being discovered.

Correction... Note that the blurb above says "...a direct addition by the original Borland/Inprise authors done before the program was released as open source." This wasn't done after the Open Source release.

Furthermore, Interbase has only been under an Open ource license for less than a year. Inprise was considering the move around last December [slashdot.org], and was finally (although missing parts and amidst great controversy which eventually forked the code [sourceforge.net]) released under an Open Source license around July 2000 [slashdot.org]

So, the thing is from what I can see, this is an instance where an Open Source release allowed a security hole, hidden for years as closed source, to be found finally. Which is, of course, the complete opposite of what you said.

Why the surprise?

Many people seem surprised that it took so long to find the backdoor. Their logic is that since it is opensource and has countless eyes looking at it, then it should have been noticed much sooner. What they don't realize is that a project like this is usually in the range of hundreds of thousands to millions of lines of code and when a developer goes into a project of that scale, he/she does not read everything, but only enough to learn the overall structure of the program, then zeroes in on sections that have been identified to need work or may contain known bugs.

If anyone truly believes that things like this should be found faster, they should try reading through this amount of code. When their heads stop spinning they will probably have a change of heart.

This is serious fuel for open source

From what I understand, this security hole have been there for years. This was (mostly) harmless as long as the machines were not connected to a global network (well, it could be used to do a lot of harm, but for someone that already have access to the network where the database run. Anyone technically given access to the internal network of a company can do a lot of harm, anyway. Most of internal security is security-thought-obscurity. Hence, when you know how to search...)

What most guys don't realise is that many many closed-source software that currently run on many computers contains such backdoors, generally implanted to ease remote maintenance (and cut down costs). I, for one, would be _very_ surprised if there was no such backdoor in the various incantations of proprietary operating systems.

Cheers,

--fred

One year since source release..

You can download the surce Here [sourceforge.net]

According to the page it was registered at Source Forge on 2000-Jan-28 15:37
--
Why pay for drugs when you can get Linux for free ?

Re:Hits on port 3050/tcp already on the increase

Wow.

Even more... If you read the saga of the backdoor here [interbase2000.com], it seems that not only was the backdoor known about by Inprise R & D engineers-- but that when the original creators of Interbase (no longer a part of Inprise, but now part of the Firebird development fork) brought the security breach to their attention engineers at Inprise were forbidden to speak to them .

And furthermore, as they realized that not only was this in the Open Source release, this backdoor was also in the last 3 closed source versions of the database. So they fixed the Firebird source, but also-- even with the company itself forbidding its own engineers to contact these people-- they wrote a binary patch program to disable the backdoor on previous versions.

Imagine that. Even while being slapped in the face, these guys fixed their product for them.

Re:what did you expect?

I agree that many software houses do this, but I doubt it is for control or power. How many stupid users are out there who mess up their systems or forget their passwords. They end up calling tech support and expect to be able to get stuff fixed. These users just don't realize that if the tech support guys can get in then it is a security risk. But then again, not much of reality makes sense to the suits...

Not surprised that it took time to be found

Lots of people here are apparently surprised that it took so long for this backdoor to be found. I thought I'd try to present an explanation.

1. Interbase wasn't officially released under an open source license until last summer. I at least, did not spend any serious time with it until the license was correct.

2. The open source interbase got off to a very slow start. Here's why:

- Borland didn't release all the tools required to build and test interbase code.
- Many of the original developers had left Borland, meaning that there was a shortage of mentors for new developers.
- Borland yanked startup funding at the last minute from the group that was going to take over the management of the code base, causing many to question interbase's future.
- Documentation of the code base is still unfinished.
- The codebase is large and complex.

Independent interbase builds (firebird on sourceforge) didn't start happening until very recently. In my mind they found this bug faster than I would have expected.

-OT

Dogma

Uh. First off, that doesn't mean open source products are any more secure. Second, many of them do not involve buffer overflows at all, but rather race conditions, poor checking of passwords, fundamentally flawed security architecture, terribly stupid flaws (remember phf?), etc. Third, more difficult for whom and in what way?

It would take a hacker a significant amount of time to discover a properly hidden and hardcoded backdoor in a closed source product. Notice how many years it took ANYONE to discover this. That is "difficult", or rather time consuming for the hacker. You might say it's easy to reproduce, but that's true for literally hundreds of Open Source security flaws. Once a hacker discovers a means and releases an exploit, the work is done. It doesn't matter to the hax0r, aka script kiddy, if exploit.c sends "LET ME IN BACKDOOR" or a bunch of machine code to the target host. Furthermore, it's quite easy to test for the existence (or at least the probable existence) of a security flaw via improper bounds checking. In other words, you just send a bunch of different programs extra long strings on various inputs until something crashes, then you simply do the work to make the exploit happen. Compare this with trying to find a well hidden backdoor in a closed source product, you either try to reverse engineer the binary or you can try brute force. In either case, it's much harder to detect.

So the question remains, easier for whom and how is that relevant? It's really not terribly relevant if you ask me. The question is how secure is YOUR product at the end of the day in YOUR environment for YOUR needs. If you start overgeneralizing by saying "Open Source is secure, Closed Source is not" then you're making a fundamental mistake. Rhetoric and dogma are not conducive to practical security.

Re:Open source = no backdoor

It was checked - that's how the hole was found. You can't security audit code in a short period of time - it takes a while. Anyway, it was because of the source release that this was found. Otherwise, this _never_ would have been fixed.

Re:Are there any *good* choices for Interbase user

Firebird doesn't have the problem!? Then why on their web page [interbase2000.com] do they have the advisory? And what is this code that I just pulled from the CVS doing?

char *PWD_ls_user()
{
if (strcmp(ls_user,"Firebird ")==0)
{
mk_pwd(ls_user);
}
return ls_user;
}

char *PWD_ls_pw()
{
if (strcmp(ls_pw,"Phoenix")==0)
{
mk_pwd(ls_pw);
}
return ls_pw;
}

Perhaps you mean it doesn't use the same backdoor password? If you are using firebird I would suggest you change these lines in interbase/jrd/pwd.c to something else for the time being (note *QUICKFIX* only). If there are any developers of firebird around I wouldn't mind hearing reasons why this isn't the same problem? What's more, the "solution" described on the home page, namely "change super secret backdoor password to something else" won't work. That's security through obscurity in the perfect form.