AOL Blocks Links from LiveJournal

Evan Martin writes "LiveJournal.com is an open-source weblog site with over a million users, some of whom use AOL. Last week, AOL began blocking all HTTP requests with "www.livejournal.com" Referer headers. This is a common practice by image hosting sites to prevent off-site linking of their images and 'bandwidth theft'. However, in AOL's case, they're blocking everything, not just images, effectively breaking all links to any AOL member's site--but only from LiveJournal. To be clear: nobody on LiveJournal can even make a link to any AOL member site without getting a '404 Not Found' error. We've also heard reports of the same thing happening on AOL properties (Netscape, Compuserve). This concerns us because we have to deal with the support requests: it worked in the past for our users, and it continues to work for other sites, so our users think it's our fault."

Martin continues: "We've tried to contact AOL three different ways, all without success. We've also told our users to contact their tech support. At one point, an AOL staffer pointed out that FTP access still worked (which is probably because FTP has no "Referrer" concept), and so, as an interim fix, we're rewriting all HTTP URLs to use FTP on the AOL properties where that works instead. This means that users can again host their images on the AOL webspace they're paying for, but more importantly, it means they can simply link to their webpage.

We wouldn't be so upset if they were simply blocking images. Bandwidth use is a valid concern, after all, and we even provide step-by-step instructions for people to configure their webservers to prevent image "theft". However, because they're blocking all access, including regular links, this looks like it's either a mistake, or something more insidious (the conspiracy theorists have pointed out that AOL has just launched their own competing weblog product, also based on "journals").

Although CI Host sued AOL recently for being blocked, we really don't want to do that. We still suspect that this was all just a mistake, and hopefully, by making this public, we'll manage to get their attention, since all our previous attempts have failed."

hopefully

[jellomizer]

Hopefully this is a temporary block giving them enough time to increase their bandwidth to the correct systems. And right now they are blocking everything so they can come up with a game plan.

Litigate

[Bruha]

All I can say is begin the complaint process with AOL. Get a someone who'll file the proper paperwork and maybe file a lawsuit to get things put back the way they're supposed to be unless it's a genuine mistake on AOL.

Also put up a message on your support lines with Steve Case's phone # to call him for support :)

tinyurl?

[Wavicle]

Could you get around this using tinyurl [tinyurl.com]? I'm not sure if it changes the HTTP_REFERRER or not.

Sorta related

[Tyler Eaves]

Any else noticed that members.aol.com is sending an invalid content-type header?

I've seen iso8859 and text/iso8859-html, neither of which Firebird likes...

They block slashdot too.

[Anonymous Coward]

Well not the whole AOL network, but the former mozilla division blocked links from slashdot (and still does), (Example [mozilla.org]). Any sites that cause major bandwidth use should be blocked, I'm sure some frequest slashdotters get the infamous Pink page of death.

Wait a minute

[s20451]

Anytime there's an article that whines about deep linking, a few dozen people post replies saying that the company could use the referer header to block all such requests. Now that a company is actually doing it, it's suddenly a bad idea. Which is it -- good technical solution or bad censorship?

I should also point out that some sites automatically block slashdot.org referers as a matter of self protection.

And AOL wonders why.....

[HutchGeek]

Seriously - they wonder why they get such a bad rap from the internet community at large. Most likey what has happend is that "Upper Management" made the decision to do it for some reason (although the journal conspiracy sounds quite probable), and they did't bother to ask the "real staff" what kind of an impact it would have. Now, once again, they've managed to piss a whole lot of people off. Makes you wonder what else they've blocked (censored) that thier users don't know about. I've heard rumblings on NANOG that they are trying to whitelist thier email too. There's a bright idea - a customer base the size AOL has, and their gonna whitelist mailservers. and my cutsomers wonder why I get ready to slap them when they suggest using AOL for a provider.

What's AOL's stated policy...

[militantbob]

...on usage of the customer webspace? Does it have to be a full site, or can it be a storage place for images/files linked to from another site? Consumers are paying for the AOL service, and getting AOL webspace as part of the deal - are there limitations on its utilization?

AOL blows

[destiney]

Pretty soon AOL will have blocked all of it's lusers from the entire web.

Common Decency Dictates..

[Plix]

That should AOL continue to block deep-linking (which they have the right to do so assuming that there is no contradictory clause in a user's contract) they should at least redirect users to a page explaining what is actually going on rather than leaving them to complain to LJ support.

Company People

[philipkd]

Why can't we treat companies like people? If a friend of your group all of a sudden has all this money, but abuses it by playing little Napoleon, why can't you treat him with disrespect and ostracize him?

When AOL needed help setting up their blogging software, who did they talk to? People like Dave Winer and other members of the net community.

So shouldn't there be some sort of Karma here where we, the blogging community, ostracize a bad player. They do it to spammers all the time, why not to the big guys. They'll eventually realize that it's not profitable to do so, and conform.

We could choose to disallow AOL urls into weblogs. We could prevent anybody with an AOL account having an RSS feed to a Blogger or LiveJournal. We could ban them from our conferences. Sounds like we're being assholes or "closed" by doing so, but I think it's important for people to check the bully to in the long-term enable the most openess possible.

Re:Will this be what kills the referer header?

[mosschops]

I use a program [admuncher.com] which allows me to change the referer header to appear to be from the same site I'm browsing. Not only does that protect my privacy but it works with the vast majority of sites that clamp down on content linking / bandwidth stealing. Sometimes simply removing the referer will kick you back out to an index page, which is a pain.

Re:hopefully

[EvilStein]

Heh. Uh, AOL has more than enough bandwidth to handle freekin *livejournal linkage*....

When I worked there (Netscape), we had 5 OC-48s in our building alone. OC-12s & DS-3 circuits for "redundancy."

I highly doubt that it has anything to do with bandwidth.

Killing referers kills EVERYTHING

[strredwolf]

Unfortunately, killing the referer header breaks alot of sites which are blocking image pointing. We (KeenSpace) just put in header checking. We do it so that if a request for an image isn't from a webpage we host (eazy stuff to do), it's 404'ed.

We cut our bandwith by 50% that way.

Re:Company People

[Anonymous Coward]

It's too hard to get a substatial number of sites involved. I'd join a major effort like that, but I sure wouldn't be an early adopter.

Also, I'd block **AA sites from access my site before AOL and have it show them a note saying they're prohibited from visiting my site and any attempt to get around it violates the DMCA.

sure

[SHEENmaster]

AOL invented blanket blocking. I'm blocked from their SMTP server, and I've heard several different justifications for it.

I suspect that they are like SCO, in that no one with any self respect or knowledge will work for them. The first time I complained about being blocked, they replied that no one there knew how to allow a server on a "dynamic" subnet. (Dynamic my shiny metal ass.) Later, I heard that no one knew how to allow one ip address while blocking the rest of the subnet. As a result, I'm being accused of the half a billion pieces of spam my ISP's other customers send to AOL.

Let's hope that broadband finally kills those bastards off. I hope their stock falls so much in value that they start using outstanding shares as toilet paper. (I'd pay to use it as toilet paper, but they want a lot more than it's worth...)

Re:Gee alot of ISP's use referrer blocking.

[dasmegabyte]

Cheapasses is right. You get what you pay for, and if you're paying for impossible things, you should beware the catch. There is no such thing as unlimited bandwidth, or unlimited space...merely limits that you haven't hit yet.

A t1 line is still over $700 per month, so burstable bandwidth starts at more than $2 per gigabyte. People who are on better pipes pay way less, of course, but then again they need to maintain them, and technicians start at about $25 per hour. Servers need to be powered, backed up and maintained to prevent hackattacks. So when somebody offers you unlimited bandwidth, unlimited space, unlimited email with 24x7 support for a pretty number like $7.77 or $5.55 or whatever, they're basically lying to you.

Check your AUP. Somewhere in there you'll find a line saying that your unlimited bandwidth can be terminated at any time if you use too much of it. Unlimited really means "We're not telling you the limits. But you'll know when you hit them." Generally because your site takes off. You get popular, people start laughing at your jokes and caring about your weblog. Then your provider cuts the cord. Sucks, don't it?

See, ISPs at all levels make money by overselling. They tell you you have a T1, when really it's fractional. They tell you you have 256 kbit upstream, then it maxes at 192. The most egregious example of this is the El-Cheapo webhost, an animal I despised so much that I started my own crummy service [webslum.org] to combat it. If you have the know-how, and you have the time, I suggest you do the same. It can be a lot of fun and offsets the cost of big web projects [dasmegabyte.org]. Just don't harbor any dreams of getting stinko rich.

I remember the first time I had a site get "overnight popular." It was a certain web comic [somethingpositive.net] that we begged to come on board. In about two weeks ge went from moving 2 gig a month to over 50. And because we small timers get the short end of the bandwidth stick, his bill was about $200. Not his bill FROM us, but the bill TO us from our host for just his transfer. We didn't mark it up. That's a lot of money when you're a hobbiest. Shit, that was as much as we paid for everybody else's bandwidth that month.

We have a policy of not touching people's sites or restricting tranfer, but if we hadn't known the guy (and known he was good for the money, which his new fans donated in droves, we even threw in $30), we probably would have had to use the "no contract" clause and take the site offline. Damned if I'm paying for somebody else's popularity...

HEH.

[Gnea]

Am I the only one here that remembers AOL from back in the day? I'm talking 1994 here. Had it for a month, then they cancelled my account behind my back with no prior warning because i downloaded too much stuff. Back then, all they had was email. Now AOL is getting all restrictive again. This doesn't surprize me too much, but it'd be nice if they would keep the 'net a friendly place. I guess the current neophytes have prevented such action.

Simple..

[adeyadey]

If you have paid for the space, and they put a block on, sue them..

Re:Will this be what kills the referer header?

[Anonymous Coward]

Be careful, there are lots of paranoid log readers out there that investigate abnormal Googlebot traffic (the ip ranges it uses are quite well known).

Re:sure

[Anonymous Coward]

I suspect that they are like SCO, in that no one with any self respect or knowledge will work for them.

No, actually there are a lot of folks with self respect and knowledge working there... it just seems the folks implementing stuff like this don't bother to consult us first. :( I'd love to print out this article and tape it to the door of the person responsible for this...

Re:Good.

[jrockway]

I've never considered slashdot a blog. To me a blog is a personal journal/website akin to the journals at slashdot. I don't read those very often :)

Re:Killing referers kills EVERYTHING

[Anonymous Coward]

which is the main reason i dont goto any site on keenspace, one less visitor for you, one less comic for me. oh well, theres plenty more sites that dont block empty referrals.

you may have cut half your bandwidth, but youll notice you also cut a third of your user base. hey, you might try blocking images for EVERYONE, then youd REALLY save some serious bandwidth!
somehow i just dont envision there being millions of sites posting entire comic strips on their pages linking to your sites images...

Re:sure

[dtfinch]

You're not alone. Some of my customers are AOL users. When they send me emails asking for tech support, I usually have to reply through AOL Instant Messenger.

Bloggers ate my samwich

[kobotronic]

Really, I can't say I'm surprised that AOL would want to block image inline image traffic from blog sites, as that shit eats your bandwidth like nobody's business.

I "run" a (dormant) photo website on a commercial hosting service. I pay about twenty bucks a month for the diskspace and capped bandwidth - a reasonable amount, I think, which allows me to serve my users without garish adbanner detritus.

The ordinary site traffic is reasonably stable and keeps well below my bandwidth cap, but parasitic inline traffic comes on top of that, drawing close to redline.

I'm very seriously considering blocking livejournal and any other blog site I can think of, as their users frequently inline my images, eating a little of my bandwidth each time one of their blog pages are loaded. I have some car photos which about fifty retarded pimply teens have inlined on their pages for apparently decorative purposes.

I'm much too busy to go out and chase down every offender, but at the same time I've been reluctant to activate a simple block rule to get rid of the inline traffic once and for all. I guess I should follow AOL's example, eh?