Continuing Security Concerns at DoubleClick

In 1999, DoubleClick bought the Abacus database, which got them a ton of data about our personal buying habits. They've promised not to correlate it with their banner-ad database, but that's not the concern this week. This week, the concern is their network security. Last week Thursday, the French site Kitetoa discovered three separate security issues on DoubleClick's network; the company deleted the evidence of one immediately, but left the servers up until Monday, when they mostly closed the other two. There are numerous other issues but the question on everyone's mind should be, how long and how far has DoubleClick been penetrated? And how long can we expect it to continue?

As I write, I'm aware of two security holes in their network, which is an improvement over last night, when the number was three. Unfortunately, this does not mean they are now 33% more secure. I don't have the background to be sure exactly how significant the remaining problems are, but I'll share what I know.

Unfortunately, DoubleClick's Chief Privacy Officer was not available by press time to respond to questions. We have offered the company a chance to respond, and we hope they'll take that opportunity to clear up some questions. Meanwhile, here's their official statement on the matter as of today:

"Over the last week there have been unsuccessful attempts made to hack into DoubleClick's servers. Those situations were immediately corrected," said Jules Polonetsky, Chief Privacy Officer, DoubleClick. "DoubleClick is now undergoing a comprehensive security audit, including the expertise of external security professionals and engineers, to fully ensure the continued integrity of our servers."

Now here's the history of DoubleClick security since last week, as far as I can tell.

Kitetoa ("KITE-a-toe-a") is a group of white-hat hackers who publish together under that pseudonym. They broke the news on their website simultaneously with transfert.net, and spoke with someone at DoubleClick to make sure the company knew to patch the holes. (I left several messages to this same contact, but he did not return my calls.) They continue to update their site with more DoubleClick security news.

The first IIS vulnerability is the commonly-known unicode bug, which lets you read and write files with the same permissions as the webserver, typically "IUSR."

Using this vulnerability, Kitetoa discovered the second security issue, which is that someone else had compromised the DoubleClick corporate webserver at some time in the past. The file eeyehack.exe was left on www.doubleclick.net. This is a backdoor written by the white-hat hackers at eEye, which opens port 6969 for attackers to telnet in.

DoubleClick assures us that eeyehack.exe could never have been executed, because that directory had script access disabled.

But I spoke with Marc Maiffret, Chief Hacking Officer of eEye (the people who brought you this port of nmap to Windows NT, by the way). He points out that the same backdoor could have been copied elsewhere, too, possibly into directories that allowed execution. I've asked DoubleClick whether they checked this; no answer at press time.

It's a separate question whether a cracker could have gotten SYSTEM level access through some other hole. With just IUSR level access, probably not much could have been done. There's no evidence that higher-level access was obtained ... but absence of evidence is not evidence of absence.

What concerns many people is that the eeyehack.exe file that was visible had a modification date of 1999. We know this date is not accurate, because the exploit that writes that file did not exist until last November. But that odd date does raise questions about how long DoubleClick's network has had these vulnerabilities.

The nightmare hypothetical is that a cracker has had access to DoubleClick's networks for the last couple of months or years, and has been reading the data they have been collecting about banner ad clicks. Or, worse, has had access to the Abacus database. Let me emphasize that I know of zero evidence that this has actually happened. But the potential for enormous privacy violations, with this company more than almost any other, is very serious.

DoubleClick assures me there is a good reason for the 1999 date on the backdoor program, but my question about it goes unanswered at press time.

The third hole is almost exactly one year old, and it allows ASP source code to be read. This alarms people because the server named AbacusOnline.DoubleClick.net was shown to be vulnerable. I verified this myself and learned the SQL passwords that go with two usernames, "gcolon" and "aowebuser."

Asked to estimate whether a determined cracker could have made use of those SQL passwords, Kitetoa guessed it was "85% certain" - for whatever that's worth.

Not that SQL should have been stored in the ASP code anyway. Marc Maiffret comments, "One of the better ways to secure SQL login information, within an ASP file, would be to store all of the login information and functionality within a COM object. This is not a silver bullet solution but it does provide you with much better security than storing things in plain text in a .asp file."

Was that database accessible from outside DoubleClick's network? Not that I could tell, but I didn't try very hard. What data was in that database? Considering the wealth of data that Abacus has collected on our purchasing habits, we might justifiably be concerned about a machine named "AbacusDirect."

Again, DoubleClick denies that this is a problem, saying it was a programmer's machine which "was not connected to consumer or production data in any way." We have to trust them on that.

That's what was known as of Monday, and you may have seen that in the MSNBC story, InternetNews story, or the ZDNet story.

But the problems continue. Kitetoa has continued scanning DoubleClick's networks, and continues to turn up vulnerable servers. A half-dozen servers of unknown significance are still Unicode-able. And - let me check - yes, I can still read ASP source code on travel.doubleclick.net.

As well as www.doubleclick.net. The company said Monday that the holes had been fixed on that, their main corporate website. But as of five minutes ago, I was still able to read ASP source code on that server using the year-old exploit. (I did not see any more SQL passwords, for whatever that's worth.)

DoubleClick's Chief Privacy Officer Jules Polonetsky claimed on Monday that "Even a partial breach of a noncritical server is unusual," a claim which looks more dubious every day.

Maybe none of these servers has any valuable data on them. But to a serious cracker, breaking into them could easily have provided the necessary opportunity to reach further into DoubleClick's systems. Being inside the firewall, setting up trojans, sniffing network traffic from other machines - these are all reasons why unauthorized access of any machine must be taken very seriously.

And there's more. A security writer from transfert.net - a sort of French "Wired" - was able to snoop around one of DoubleClick's internal mail systems, webmail.doubleclick.net. It's fixed today, but he sent me a partial list of employees whose mail files were in a directory. (He could not read the mail itself, and DoubleClick denies that mail could have been, at any point, read.)

Here's the big picture. Where long-term security is concerned, process is always more important than the individual problems. We can learn more about DoubleClick's security by observing their response to security reports.

I would have hoped to see servers taken down. As Kitetoa remarked to me, "I think they should have closed all the servers for an hour or two while they fixed the problem." Or, if necessary, longer. Tough decision, but better than possibly exposing data.

I would have hoped that DoubleClick would not announce to the media that everything was fine until they actually knew that this was true. At this point, we have to trust them when they say that attackers could not have seriously compromised their data.

And I would have hoped that they would start tackling problems sooner than they did. Vulnerabilities about which they were informed last Thursday were not fixed until Monday. (The vulnerabilities themselves, of course, are not new, and some of them are a year old.)

And more vulnerabilities continue to crop up. Paranos (based in Paris) found one just by using search engines. The funny thing they found was a database of DoubleClick employees in the U.K. who attended last year's Christmas party.

Less funny is the list of people Paranos found who apparently filled out a form in conjunction with FoodTV, the "Outfit Your Kitchen Sweepstakes."

It took about two hours after I notified their PR contact of the URL before it was removed. It was stored in what appears to be a directory of backup data which was never intended to be public: http://www2.doubleclick.net/live2/chefs/ foodtv-bak/form/chefs.txt.

This isn't a vulnerability, but it is indicative of the company's security process. DoubleClick should have policies in place to prohibit posting backup data on public websites; it should enforce those policies; and when it was done anyway, it should have found the leak with by internal audit.

This promotion ran in 1999. Remember, kids, data you give to corporations never goes away, and may pop back up at any time!

I picked one of the phone numbers for my state and gave it a call. Kathy Bankes, the wife of one of the people on the list, answered the phone. She said she was "very leery with the computer," and then proceeded to give me a common-sense understanding of the state of security on the internet.

"They say things are supposed to be secure," she said, "but I don't care how secure anything is - if somebody knows how to get in, they're going to get in if they have the technology."

"Should we worry about it or not?" she asks. I would say yes, especially when the company that owns an enormous customer purchasing database has problem after problem with its security. Maybe I'm naive to think that privacy promises mean anything in the real world, or that crackers can be fended off by a big corporation.

When I'd expressed my surprise at all this to Kitetoa, he'd just chuckled and said, "You'd be surprised what you can find on the internet. It's all like this."

Kathy, too, was pretty sure that there were people who have all this private information anyway. She very sensibly pointed out that credit card and other personal data can be stolen in the real world just as easily as the internet.

And she answered her own question for me: "There's really nothing you can do. I don't feel secure anyways."

Way of the Future

Detailed consumer profiling is going to be a fundamental part of our future, and we ought to be developing ways to get around it, limit it, etc., instead of yammering out the usual boilerplate about how DoubleClick is the devil because they know if we like golf or not.

With the increasing sophisitication of profiling technology (and the databases that drive it), as well as the sketchiness of existing laws on the subject, it won't be long before every major company has a detailed consumer database. We complain and complain about privacy issues, but they don't know anything that we haven't made known to them. If they send us sailing magazines, it's because we clicked something somewhere to indicate that we were interested in it. We know what happens when we click those things. Everyone knows. My grandma knows.

You are being watched. Act accordingly.

Poof!

You know there's a pretty easy way to block doubleclick. In windows edit your c:\windows\hosts file to include:

127.0.0.1 doubleclick.net

There's also the hosts file in linux that can do the same thing. No more worries about doubleclick!

When are we going to get it right.

My lisp instructor recently gave a keynote speech at Los Alamos entitled When are we going to get it right [lanl.gov]. The speech does a great job covering many of the security issues regarding double click and just the overall state of security on the web. Its a good watch if you have the time. You can see the whole thing in realvideo from the link above.

-gerbik

For the love of god, opt out

If you value not being tracked, you really should opt out of Doubleclick's tracking. For those not familiar, they use cookies attached to every ad in order to track which of their member sites you view. If this bothers you just a little, go to

http://www.doubleclick.net:80/us/corporate/privacy /opt-out.asp?asp_object_1=& [doubleclick.net]

--

Re:security through obscurity

"If you focus enough distributed processing power on any security problem, like, say, through posting it to slashdot, you will overwhelm and outpace the sincere efforts to patch the problem that are being undertaken by the hapless victims."

The company has now known about the problem for 5.5 days. I had debated how long to give the company to fix stuff before posting this, but since it was already picked up by MSNBC and other media two days ago, I don't really feel it's an issue anymore.

Jamie McCarthy

...and the big deal is?

Hey,

Is it just me, or is this just another company getting hacked? So it happened to be an advertising company. Big deal. This hardly seems slashdot-worthy; web servers are compromised all the time. Most of DoubleClick's data is just IPs and lists of websites.

It isn't automatically a big conspiracy, spying on you. Do you really think that, if hackers compromised doubleclick's servers they'd be looking for your information? Well, let me tell you this: They won't. To think that they are is paranoia taken to it's extremes.

So a website has a security bug or two.Why not just inform the site owners, and give them a chance to fix it, instead of proclaiming it loud and clear to the world? It seems helpful to no-one.

Just my $0.02

Michael

...another comment from Michael Tandy.

He acted responsibly

He didn't just say "oh, it's been published by others, I'll go ahead and break the story too." He gave them an additional two days to get back to him, and the fact that they didn't suggests that they probably never will. After 5.5 days, if a company hasn't responded to a message, they have probably chosen to ignore it.

I am actually quite impressed with the journalisitic merits of this article, and I'm happy that Slashdot has started putting more research into their stories.

Re: Menthol is not a disability!

"...people who cannot use editors because of disabilities, menthal or physical."

I know, some people don't prefer the cool, crisp taste of menthol cigarettes. That's their choice. But to say it's a disability? I've smoke menthols before, and I can use an editor just fine.

I'm really beginning to doubt the level of intelligence displayed here on Slashdot.

--SC