
A Day with an ISP Spam Investigator 167
scumbucket writes "Network World Fusion has an interesting article about an abuse investigator for ISP Earthlink and his job of tracking down spammers. It's nice to see that major ISP's are making an effort to shut spammers down and kick them off of their networks."
A yawner (Score:5, Insightful)
Where's the beef?
Re:A yawner (Score:5, Insightful)
What the article describes is entirely re-active. In no way is it pro-active: pro-active costs money, and keeps the spammers from signing up in the first place to send the spam. This is typical Earthlink, whose focus is on making the weekly progress reports their departments favor as taught by the "WISE" management techniques so favored by their Scientology educated president and his top staff.
It's not evil, but given their history of blowing off complaints for months or even years until faced with real consequences such as a Usenet Death Penalty where all posts from Earthlink would be actively cancelled, it's not topnotch.
Re:A yawner (Score:2)
Re:A yawner (Score:2)
Reed was one of their founders and got arrested for running a pyramid scheme. Given the financing misbehaviors of many start-up ISP's, it's not real surprising that one of their early managers also ran pyramid schemes, but that seems to be extremely common for the upper level Scientology members. They had a bunch of their upper
Re: (Score:2)
Re:A yawner (Score:2)
Re:A yawner (Score:2)
Most of what was said is really good for the non-geeks in our midst. You know as well as I do that nearly everyone hates spam, hates those who commit fraud, etc. This is showing that someone behind the scenes is doing work on t
Self interest (Score:4, Insightful)
Re:Self interest (Score:2, Insightful)
Re:Self interest (What is the Cost?) (Score:5, Interesting)
I've heard many a system admin complain about the "cost of spam" to their networks, but have not seen a quantification of that cost. Given that spams are so small (the ones that I get average 4kB/spam), the storage costs of saving every spam (at 1$/GB) are about only 4 micro$/spam and the transfer costs (at $3/GB of transfer to pick a Google figure) are only 12 micro$/spam. Even CPU time is cheap. If a $2000 server CPU can handle only 10 messages per second (an underestimate?) then the cost in CPU time is only about 6 micro$/spam. In total, a million spams would cost an ISP maybe $20 or $30 which is far less that the burdened labor cost of one hour of a technician's time.
What am I missing here? Can any admins tell me the true dollar cost per spam? The only other reason, that I can think of, is that Earthlink fights spam to avoid blacklisting because blacklisting would drive up support costs when a million customers call at ask why their emails aren't getting through.
Re:Self interest (What is the Cost?) (Score:2, Interesting)
Part of the cost is also due to filtering and to the extra admin costs for implementing enough capacity to hold the extra spam..
Re:Self interest (What is the Cost?) (Score:4, Insightful)
Re:Self interest (What is the Cost?) (Score:4, Insightful)
Which I had to do. (Score:2)
And I have to make sure it is patched.
And I had to adjust the email server's threads (default set to either 2 or 4) for handling incoming email (increased to 50).
And tuning of SpamAssassin.
Re:Self interest (What is the Cost?) (Score:2)
Re:Self interest (What is the Cost?) (Score:2, Insightful)
Re:Self interest (What is the Cost?) (Score:4, Insightful)
but if its 30 $ per day, its 10k per year.
further more you have to spend time and energy you have to spend sorting the mail. this is, ive heard, quite expensive in CPU time.
The best filters catch 99.9% of spam and only make 1 mistake in a thousand. ( i don't even think that they are that good).
1000 emplyoees gets 5 mails aday for a year thats 1.8 million mails, thats 1800 mails per year that goes down the drain. im not sure what that costs, but some of the are prolly quite expensive.
This is not absolute facts nor close, but my point is that the price of spam is more than the price of reciving spam.
Re:Self interest (What is the Cost?) (Score:3, Informative)
Let's assume it takes a user only 1 second to determine if a piece of mail is spam, and deal with it, and let's assume the average user's time is worth $20 per hour. A million spams then cost the users:
$5555 = 1 second * 1 million / 3600 seconds in an hour * $20
You're right, the ISPs scared of being blacklisted. But they also
Re:Self interest (What is the Cost?) (Score:5, Interesting)
I used to work at a small ISP -- lets say 5000 customers. We were getting lots of complaints [slashdot.org] about spam, so we decided to put in better spam filtering. That required a bigger server. Then the mail server went down for half an hour [slashdot.org] because of the volume of incoming spam, and there was a suddenly a big rush on getting the new server up and running.
The server was the cheap part: let's say $2000 (all figures Canadian) for the box, rackmount, hard drives, yadda blah. Thank God for Free software, because FreeBSD and SpamAssassin saved our asses. It took me, conservatively, three full days to set up and get it more or less right; I was doing a lot of learning on the job, and the regular sysadmin was away.
Now, don't forget that we were down for half an hour. This was from roughly 9am to 9:30am on that day, so that's a busy fucking time for us. There were tons of calls and only three people to handle them; fortunately, I was pressed into service trying to fix things, and wasn't on the phones. We probably lost a couple customers then, but most people were pretty understanding, especially when they were told it was fuckwad spammers who were causing the problem.
Complaints were a huge deal, both before and after the filtering was put in place; I was dealing with most of them, because I was doing abuse duties, and it wasn't fun. Complaints before the new server was installed went, "Why am I getting all this spam? Why can't you stop it?" Complaints afterward went, "Why am I still getting all this spam? Why isn't your filtering working? What do you mean, I have to set up my mail program to do more work?" (We set the threshold rather high, thinking that customers could use filtering in their mail client to set their own tolerance level. Ha! It is to laugh. Ever tried filtering on random headers in Outlook Express 5.0?)
Plus, there was maintenance of the server and software; upgrades were never fun; false positives happened and were dealt with; and now, my sources tell me, they've graduated to buying dual-fucking-xeon processors in order to handle spam filtering. Fuck me!
But hey, we were after a dollar cost, and I did get sidetracked. We already said $2k for the server. Three days of my time, $400 (deal!). Half an hour when everything in the company came to a halt because no one could send mail or do anything but answer the phones: $500, and that's probably very conservative. Two customers' worth of lost revenue for a year: say another $500. Spam complaints before took, oh, probably a good five solid days of my time: $650. Afterward was probably the same, so another $650. I know of at least one customer we lost afterward when the spam filtering wasn't the magic bullet I kept trying to tell them didn't exist, so $250. Bandwidth for all the spam we were accepting but kept from reaching the customers: let's say $50, for a nice round total of $5000.
Now this is very, very rough back-of-the-envelope calculations for a small dialup ISP I no longer work at; the managers there could probably tell you more about lost good will and so on. More importantly, it doesn't tell you about ongoing costs; that's just a snapshot from when I worked there. But that was $5000 spent by an ISP that was going down the tubes (true story), just to keep up (barely) with a denial-of-service attack that was slowly grinding us into the floor. I can't even imagine what it's like for AOL or Hotmail. Nor will we ever know what that time and effort and money might have done if it wasn't being spent on spam.
Goddamn fuckwad spammers piss me off.
Re: (Score:2)
Re:Self interest (What is the Cost?) (Score:2)
As far as Perl and speed goes, from what I remember it wasn't much of an issue; we used the spamd c-based daemon to pass email to just-the-one copy of SpamAssassin, and it wasn't that bad. There was a bit of delay, but it was nothing like before when we were using Procmail. One bad entry in Procmail could bring the whole thing grinding to a halt.
Re:Self interest (What is the Cost?) (Score:2, Insightful)
Its not just the total number of received messages that affect cost. Delivery rate causes problems with network availability. Because of distributed attacks and mail bombs, we have to be a
You've got a niche business (Score:2)
Re:Self interest (What is the Cost?) (Score:2)
Re:Self interest (What is the Cost?) (Score:5, Insightful)
More recently, the local ISP for which I often do admin work had to build three new incoming mail servers and purchase spam and virus filter software for each machine at the rate of at least $6000 ea. plus subscription. Without these machines, user mail spools were filling up with spam and viruses; the older the account the worse off it was. Ask these folks how much it costs.
I have seen spam perform the equivalent of DoS floods: causing servers to crash, filling up T1s, causing CPU loads on older but otherwise working machines to hit 98%, and more. I host a domain which sees 28,000 spams per week on average. We employ RBLs in our fight against spam, as well as blocking a number of countries known for delivering nothing legitimate to our servers.
We see the shit come from all directions. In one night I observed a spam run against a hosted domain attempt to deliver 5,821 messages -- all forensically identical -- in less than 100 seconds from roughly 15 sources.
Why should it be the burden of the ISP to provide extra bandwidth, CPU processing power, memory, and storage space just to accomodate what it clearly a theft of services? The dual 66MHz SPARC system that was running an ISP back in 1995 is still running, and in a normal environment handles incoming and outgoing email just fine. Without the introduction of a front-end server, or replacement altogether (money spent no matter how you look at it) the machine often ran at 75% load or more during times when historically it ran no more than 30%.
The attitude of "well, it's going to happen anyway, might as well deal with it" is garbage. Adopting such an attitude in the face of a hurricane, the forces of which cannot be stopped, is fully acceptable. But in the face of spam which should not exist in the first place, this attitude is comparable to rolling over and taking it right up the rectum rather than dealing with the source.
Customer Satisfaction vs. ISP Burden (Score:2)
It's also because tracing spammers sufficiently well that you can haul them into court and force them to pay is usually a lot more expensive, has a low probability of success, and if they're in the US where yo
Re:Customer Satisfaction vs. ISP Burden (Score:3, Insightful)
Horseshit. That is along the same lines as the police department telling a hotel manager that he should bullet-proof the glass and walls in his establishment to help with the onslaught of drive-by shootings.
A
Re:Customer Satisfaction vs. ISP Burden (Score:2)
Re:Self interest (What is the Cost?) (Score:2)
Powerpoint format [hserus.net]
Steve Atkins presentation to the ASRG: Google cache as HTML [216.239.59.104]
Same as powerpoint [word-to-the-wise.com]
A graph [nixcartel.org] of a random minute at a large email provider.
Each point is one host.
Those numbers are all very very real.
Dumb business attracts dumb users (Score:4, Funny)
Spammers are stupider than I realized.
Re:Dumb business attracts dumb users (Score:3, Informative)
Insecure passwords at ISP (Score:2)
Did anyone else see the implications of that? It says, "Earthlink admins know your password." Every security system I know stores passwords using a one-way hash. It is supposed to be impossible for an administrator to discover the password from the stored data. But this admin just admitted he is that checking the cleartext passwords. Make certain you use a d
Re:Insecure passwords at ISP (Score:2)
As a consultant, I tell companies that they have to trust their admins. If they do not, then the company will have problems making IT an asset rather than an obstacle. I built an incredible single-signon system, and all passwords are encrypted for storage, but an admin could modify the program to save the unencrypted passwords. I wonder that com
Earthlink SpamBlocking (Score:2)
(The "high" setting requires whitelisting everybody, which means giving Earthlink my address book. Even I do not have my address book; I just search for a previous email from somebody and click Reply.)
Whatever Earthlink is paying for the SpamBlock is too much if it does not do anything.
How to solve the Spam problem (Score:5, Interesting)
Clean up your act guys. When you're costing the world this much money, it just isn't funny anymore.
Re:How to solve the Spam problem (Score:2)
I agree it is an American problem. Most of the statistics says so.
For those who say it comes from outside the US, like China or Korea, please thing about it for a moment: What are they advertizing?
Even if the messages are coming from overseas IP Addresses, the content advertizes US-centric products, for example, cheap mortgage will not help somebody in France, or Egypt or India, even if they wanted to. Yet, they have to pay to get SPAM because unlike North America, their slow dialup connections are m
Re:How to solve the Spam problem (Score:5, Funny)
"not whimpering soccer players'
American Football - British rugby (but with pads so the ickle Americans don't get hurt. Everyone say awwwwww)
Re:How to solve the Spam problem (Score:2)
Re:How to solve the Spam problem (Score:4, Insightful)
everyone is first in line to expect us to fix it.
Don't you think we should, if the problem is coming from us?
Re:How to solve the Spam problem (Score:2)
Where exactly do you get this notion? You must not include either of the rainforests as "big forests". I would also say that most of the "big forests" are more likely in the northern portion of N. America mostly in Canada.
I by no means say we are innocent. I also think Bush is an idiot, but we cannot accept the brunt of blame for a spam filled mailbox. I also never said anything concerning how we as a country have less or more broadband tha
Re:How to solve the Spam problem (Score:2)
Can't blame ya guys at all on that one.
Deterrence? (Score:2)
"Serial spammers who have been kicked off the EarthLink network once will often jump back on, creating as many as four or five fraudulent accounts per day using stolen credit cards
So - Earhtlink are fining victims of stolen credit cards, in other words!. Possibly a more long-lasting approach would be to geo-
Re:Deterrence? (Score:2)
Why can't Earthlink ban certain MAC addresses from its network? Surely the way to stop a repeat offender re-registering is to use MAC addresses (which are each unique to the unit) to ban his computer or router.
Sure, a technically minded user can change a MAC address but its a delay, and not always easy. Spammers aren't the brightest bunch.. hell, most of 'em can't even spell viagra!
Re:Deterrence? (Score:2)
Re:Deterrence? (Score:2)
So is your Sig, actually, I've bookmarked it to read at length later)
Re:Deterrence? (Score:2)
But at which cost? (Score:2, Interesting)
(Page 2)...One notorious spammer, whom EarthLink helped put behind bars, repeatedly used the names of sports such as baseball and football as his password...
I thought that passwords SHOULD not be easily unencrypted... or do they monitor them before encryption?
Re:But at which cost? (Score:2)
The only passwords we did not have access to were employee accounts, who we *could* have access to, they'd just be notified instantly.
Re:But at which cost? (Score:2)
Re:But at which cost? (Score:2)
Granted, this was over four years ago. I'm not sure their policies remain the same.
WTF? Stolen credit cards? (Score:2)
So, Earthlink finds a spammer using a stolen credit card. Wouldn't they send the phone number and the credit card info to the FBI? Wouldn't the FBI trace that phone number to a physical location and arrest the spammer for fraud?
Re: (Score:2, Funny)
Just like the USPO (Score:5, Insightful)
Job security? :) (Score:3, Insightful)
Gotta agree with you there. Particularly at an ISP.
If you KNOW your actions are ineffective, wouldn't you re-evaluate your approach and look for more effective actions?
Say
From the article: "Yet canceling a spammer's
Re:Job security? :) (Score:2)
The characteristics of spam. (Score:2)
I'm guessing the article was about dial-up accounts because I don't see anyone opening 4 or 5 dsl accounts a day.
So, the easy solution is to block port 25 from your dial up accounts. Or, at the very least, limit the out-bound connections on port 25 from those accounts. Either by number of connections (limit the number of spam messages sent) or by a fixed number of destinations (a lot of spam can be sent to a few a
Re:The characteristics of spam. (Score:3, Insightful)
Because T-1's cost more and require physical loc. (Score:4, Informative)
Because, unless you have a peering agreement, you are connecting to an ISP.
"You call the phone company (any phone company) and say you want a data T1 connection."
Okay. That's a chunk of money and it has a physical connection point that is recorded. It is completely different than a dial-up account.
"They give it to you and give you some IP addresses."
From their block. That means that they are your upstream provider. If someone complains about your behaviour, they will complain to your upstream provider who will then cut you off (or not).
"They do not process email for you, they do not give you web space and they do not respond to complaints about what you are doing with your T1."
They do respond to complaints about what you are doing.
"I expect this holds true for any sort of data connection from a telecommunications provider that is not providing any additional services, which means if you call SBC to get an OC48 they aren't going to ask you what you plan to do with it."
That is correct. They will not. But you ARE plugged into THEIR network.
One end of the line terminates at your location, the other end terminates at the phone company's location.
So, traffic coming from your line goes through the phone company's network. And people can see who licensed that IP range to you. They will complain to your upstream provider.
Re:Because T-1's cost more and require physical lo (Score:2)
Not. Not ever. We went through a long period where some folks at SpamCop decided we were spammers because of a subscribe-to newsletter that they didn't remember or appreciate. So, we were spammers. Lots of complaints were received here. Lots of complaints were sent to McLeod USA (who we had a T1 from at the time).
Re: (Score:2)
Re:Just like the USPO (Score:2)
How do you know that it's completely uneffective? Spam might be bad now, but without ISPs shutting spammers down, it could be even worse.
Abuse (Score:5, Interesting)
It's the major
Which is a shame; KISA (.kr equivalent of the FCC/ACA/etc) have got a great early-warning system set up, which shows transit traffic between
About 40% of my current spam corpus is from korea, the other 60% is about 30/30/40% china, uu.net, and comcast/verizon open proxies.
Re:Abuse (Score:4, Informative)
Re:Abuse (Score:2)
Re:Abuse (Score:2)
Additionally, whilst the US is sitting around 70%, the majority of US sourced spam is sent through bogus proxies, many of which are in
Re:Abuse (Score:2)
Those evil young males (Score:5, Funny)
Yes, the typical spammer is a slashdot-reading geek who lives with his parents. ... Reminds me of a thing I read earlier warning parents about signs of their child engaging in dangerous hacking, such as use of Linux or requests for better hardware.
Just what a geek needs, another reason for parents to be suspicious of his computer usage. Help! I'm a computer addicted teenager who can't stop sending spam, and this is really a cry for help!
Re:Those evil young males (Score:3, Informative)
Very sensible (Score:4, Funny)
"He only reads the content of an e-mail in extreme cases, he says."
I've always found it safest to avoid reading email, unless I'm feeling really daring...
White Hat or Censorship? (Score:3, Interesting)
Re:White Hat or Censorship? (Score:5, Insightful)
Oh, and it's not censorship. He's not a government or publisher. The spammer can find other places to publish his work other than my mailbox. (Just like wannabe painters can't exhibit in my living room.)
Re:White Hat or Censorship? (Score:3, Funny)
Oh, real nice for you to tell me that now! I was all packed and ready to go.
And how did I get modded insightful? The site I linked to is SatireWire. I'm beginning to thing that some people don't RTFA I link to!
decent post.. (Score:5, Funny)
While he believes his job is important, Rush doesn't take the role of Internet cop too seriously. But he admits with a chuckle that his favorite computer game at the moment is called City of Heroes.
I'd sit back all day and play CoH, and tell my boss "Sure, I killed off 800 spammers today. But 30,000 more popped up. Guess I'll be seeing you monday."
I need me one of those gigs. Anyone offering?
Interesting article... (Score:5, Interesting)
Incidentally, this bit:
was interesting to me. This sounds like the oft [icx.net]-repeated [apfn.org] assertion that a US flag with a fringe in a courtroom means that you're under Admiralty law [fourwinds10.com], not the law of the United States, and that anyone who appears before that court has lost most of their rights. Of course, They [bible-prophecy.com] don't want you to know this...or that England still owns the US [davidicke.net], or that there is a subtle yet vitally important difference [sweetliberty.org] between the United States and the United States of America that means you are 0wn3d by the government...
I tell you, there are worlds upon worlds of free entertainment out there on the Internet.
Re:Interesting article... (Score:3)
Re:Interesting article... (Score:2)
Re:Interesting article... (Score:2)
Re:Interesting article... (Score:2)
Their not really serious (Score:2, Informative)
If they were really serious about curbing spam they would implement greylisting and greet_pause features in their MTAs. Both of these would block 99% of the spam being sent. The remaining spammers would then be mu
Re:Their not really serious (Score:2)
memory lane (Score:4, Insightful)
About 50% of my time was indoors, pulling street-by-street printouts off our Tandem system and cleaning up/verifying account info by going back to original install paperwork. The rest of my time was spent climbing poles, verifying hookups and disconnecting the "non-subscribers." After a year of that, we had enough info to deliver numbers to the statehouse: 4% of all cable viewers weren't paying us for the service. That was enough for the legislators, and cable theft became a mid-range misdemeanor.
So then I started going after the midnight installers offering people "free HBO forever" at the low low price of $100 (or whatever). That was kinda fun...serveral times I was just hours behind these guys, removing service drops while the resident stood by watching, moaning eulogies for their recently departed 100 bucks.
I'm surprised that more ISPs don't have employees like the guy in TFA (or perhaps I'm surprised that we don't hear more about them)...losses due to spam are real, no? [In the case of cable, the "losses" were 99% paper; there was no extra drain on bandwidth, no guarentee these folks would have been paying us otherwise, and no real loss on the converters they were using (our collections folks did just fine charging 4X the cost for unreturned equipment). The only true "loss" was in tech-time, for the rare hookup that caused interference on a distribution line or radiated enough signal to breach FCC rules.]
Is the reason for this apparent lack of interest on the part of ISPs similar to that of the credit card companies during the early online days? Rather than appear inept at providing decent system integrity (easily spoofed card numbers, pitiful account verification, etc.), fraud and abuse were handled quietly, with costs taken off the bottom line. Or is the apparent less-than-vigorous investigation of spammers just part of the "?" step in the profit! formula...where bandwidth lost = cost of investigatory personnel, so screw the inconvenience to customers?
Passwords? (Score:4, Insightful)
Re:Passwords? (Score:3, Informative)
Re:Passwords? (Score:2)
Re:Passwords? (Score:2)
Usernames, passwords, email addresses and IP addresses are EarthLink's property and EarthLink may alter or replace them at any time.
So they could probably claim ownership over your password (as it exists within Midas, not over the password itself) and could justify allowing empl
Re:Passwords? (Score:4, Informative)
Re:Passwords? (Score:2)
Re:Passwords? (Score:3, Informative)
It depends entirely upon the ISP, but yes, at most large ISPs, employees can view your password. It makes tech support MUCH easier when dealing with stupid people. If this bothers you, call your ISP and ask them, and if they don't encrypt their passwords, switch to an ISP that does.
Re:Passwords? (Score:2)
Re:Passwords? (Score:2)
If the server supports it. Most don't.
Earthlink (Score:2, Informative)
Now Zapp, you may ask: "What has that to do with anything?"
If you really don't know what staunch dfenders of free speech the Scientolgy[tm] "Church" is you might find some interesting reading at this link [skeptic.com].
If you want to dig deeper then Xenu [xenu.net] can guide you.
Re:Earthlink (Score:3, Insightful)
Precisely. I worked at Earthlink for over a year, and the only time I heard anybody mention anything related to Scientology while I worked there was a couple of crazy nutball customers.
Yawn... (Score:2)
This was pedantic in the extreme.
MTA should bounce body free emails (Score:2)
Re:Spamming as a job. (Score:2, Funny)
Re:Spamming as a job. (Score:3, Informative)
But even assuming they only made 50c per 1000 emails, when you're sending out 10+ million emails per day that's still $5,000+ per day or $1,825,000+ a year. Even at 1c per 1000 mails they still make $36,500+ per year.
Re:Welcome to the Global Economy. (Score:2, Insightful)
Re:Spamming as a job. (Score:2)
Re:First Post! (Score:2, Informative)
and do not object to Google's policies
But many of us will not send mail to gmail.com
Problem 1: Gmail is nearly immortal
Google offers 1 gig of storage, which is many times the storage offered by Yahoo or Hotmail, or other Internet service providers that we know about. The powerful searching encourages account holders to never delete anything. It takes three clicks to put a message into the trash, and more effort to delete this message. It's much easier t
Re:Yeah, right. (Score:2)
Please note that searching n.a.n-a.sightings for just "earthlink" will also give you a ton of examples of spam that was sent to Earthlink accounts, spam that was sent using a forged Earthlink return address, and completely non-Earthlink-related spam sightings posted by people with Earthlink e-mail addresses.
For example, please actually look at the first result of your search and explain to me h