AOL Blocks Links from LiveJournal

Evan Martin writes "LiveJournal.com is an open-source weblog site with over a million users, some of whom use AOL. Last week, AOL began blocking all HTTP requests with "www.livejournal.com" Referer headers. This is a common practice by image hosting sites to prevent off-site linking of their images and 'bandwidth theft'. However, in AOL's case, they're blocking everything, not just images, effectively breaking all links to any AOL member's site--but only from LiveJournal. To be clear: nobody on LiveJournal can even make a link to any AOL member site without getting a '404 Not Found' error. We've also heard reports of the same thing happening on AOL properties (Netscape, Compuserve). This concerns us because we have to deal with the support requests: it worked in the past for our users, and it continues to work for other sites, so our users think it's our fault."

Martin continues: "We've tried to contact AOL three different ways, all without success. We've also told our users to contact their tech support. At one point, an AOL staffer pointed out that FTP access still worked (which is probably because FTP has no "Referrer" concept), and so, as an interim fix, we're rewriting all HTTP URLs to use FTP on the AOL properties where that works instead. This means that users can again host their images on the AOL webspace they're paying for, but more importantly, it means they can simply link to their webpage.

We wouldn't be so upset if they were simply blocking images. Bandwidth use is a valid concern, after all, and we even provide step-by-step instructions for people to configure their webservers to prevent image "theft". However, because they're blocking all access, including regular links, this looks like it's either a mistake, or something more insidious (the conspiracy theorists have pointed out that AOL has just launched their own competing weblog product, also based on "journals").

Although CI Host sued AOL recently for being blocked, we really don't want to do that. We still suspect that this was all just a mistake, and hopefully, by making this public, we'll manage to get their attention, since all our previous attempts have failed."

Will this be what kills the referer header?

[Anonymous Coward]

It's optional, so browsers don't need to send it. Mozilla/Firebird/etc (and Opera) can be easily modified to not send one, and the Google Toolbar could probably support blocking them, too (since IE isn't being updated). AOL is a big enough presence that this could have a significant impact on peoples' browsing.

AOL and Blogs

[zangdesign]

Actually, you may want to investigate whether or not AOL has gone live with their blog offering ( article here [washingtonpost.com]). If so, it may be viewed as an intentional act.

Well played...

[tempest303]

This is a really level-headed, well played move on LJ's part - primarily because they're following the universal principle of assuming stupidity before malice. ;)

Why is it...

[pongo000]

...that people bend over backwards to accomodate companies with draconian policies like AOL? If I were running an ISP, the loss of a few customers because they suddenly discovered they could no longer send e-mail to AOL customers through no fault of my own would most likely be offset by new customers who understand that the earth does not revolve around AOL. So they're blocking incoming HTTP traffic based on referrer? Are there not more pressing problems to attend to rather than trying to please the AOL gods?

I'm not saying AOL is in the right. I'm simply saying that AOL (and companies like them) should be made to lie in the bed they make for themselves. Only when AOL customers start to be inconvenienced by AOL's own policies (rather than third parties patching together "workarounds" in a misguided attempt to protect the integrity of AOL) will they realize what AOL is up to...

mail as well?

[m0i]

"And now we have a request from an AOL user that suddenly they stopped getting LJ emails. They say AOL did just add some new spam filters, so that may relate.

It almost makes you think that they don't like us..."
AOLers are only getting sanitized Internet to the company's liking... Those who are not happy should switch.

A very good point

[pgrote]

Many times in a large organization changes are made where the impact across the enterprise isn't realized either through poor planning or lack of full testing. Sometimes you just miss something.

I do like their approach of hitting up the Slashdot crowd looking for more information and passing on what they have.

More companies should do like you said ... try cooperation and information sharing rather than decalring war.

Re:F12

[randyest]

Well, if I were an AOL user right now (quite a stretch to imagine, I'm afraid), I'd be loving to check my contract to see if it provides for such limitations on user personal space. If not, I'd call to complain (I'd probably do that anyway, right before I called my new ISP). I know that my cable modem "free" hosted space included has a very specific contract that limits the monthly bandwidth usage, but does not mention anything about blocking access for links or clicks from other domains. I wouldn't be happy if they suddenly started returning 404 Page Not Found errors when anyone linked or clicked from foo.com.

Maybe they can claim technical difficulties if called on it, or maybe the contract does let them yank their users around like this. I don't know. Does anyone here use AOL and will admit it long enogh to post a link or copy of the appropriate contract?

Of course, there's the tinfoil-hat theory that AOL is planning to start thier own blogging service and wants to drag LiveJournal down from it's #1 spot a bit. Seems like a particularly blatant and non-clever way to do this though, or maybe that's the "beauty" of the whole ingenious plot? ;)

Finally, I'm going to hope the /. editors checked this out somehow. I don't have AOL space. Can anyone else verify this is true and not just some EBKAC or hoax?

Re:Wait a minute

[isorox]

My Libertarian side says AOL are free to do whatever the hell they want, it's their server. If you ask the AOL server for a page and it send you the goatse man, thats fine, thats their right. Vote with your wallet and dont buy their service.

My more centrist side says this could be abusing a monopoly (or at least dominant position), OK they dont have a microsoft style monopoly, but they do have the monopoly over Joe Stupid.

My cynical side says who gives a flying fuck

Re:Wait a minute

[infornogr]

I regret to inform you that all Slashdot comments are not posted by the same person. Not only are there different people on Slashdot, some of them have different opinions. It's a shocking revelation, I know.

Referrer Header

[Mrs. Grundy]

I, for one, like the referer heading. It is useful to see where traffic is coming from and it really stinks that AOL is going to encourage people to mess with it, remove it, or spoof it. This will be the ONLY result of AOL's action. They may get a short break from livejournal links but people will work around it. The internet is about linking after all. If AOL want's to invent their own thing with their own rules they should make their own little private net like they used to have and they can remain one tight, happy, cloistered little clique. Of course if the referer header becomes useless maybe it would be a good opportunity to fix one of the most influential spelling errors in recent time and start using the refeRRer header instead.

Re:Wait a minute

[randyest]

Um, I don't think this is really about deep linking. There's not much meat in the story summary, but as I understand it they are blocking access to AOL users' hosted web space (the "free" 20-50MB or whatever, usually with a monthly BW limit, that comes with most ISP accounts.)

Seems to me that this comes down to a question of the wording in the AOL user contract -- if it allows this, then the folks are SOL and chould change ISPs. If it doesn't allow this (such as my cable modem accont, which only mentions limits on total space and bandwidth, NOT referer) AOL should stop doing it, and if they want to revise the contract, do so before trying it again.

Leave it to AOL...

[TheHawke]

to use sledgehammer tactics when it comes to something they don't like..
Stupid, stupid, stupid..

Re:Wait a minute

[danielsfca2]

Deep linking seems to be a totally different issue here. Your average AOL user who wants to place a link on their LiveJournal to "members.aol.com/aoluser" would derive little value from a link to "www.aol.com" or even "members.aol.com"!

This is webspace that AOL gives its users as part of their paid service. When you pay for webspace, the general idea is that it supports these things called hyperlinks. It stands to reason that you or anyone else should be allowed to link to your website from any other website. Any deviation from this traditional behavior should be documented in their terms of service, and is very shortsighted and/or stupid, as it threatens the very nature of the WWW, much like restrictions/penalties on linking to sites that are deemed undesirable.

Re:Will this be what kills the referer header?

[fruitbatUK]

It may be optional, but in my experience of using JunkBuster there are some sites which won't work without a referrer header coming from their site.

If the referrer header does die and those sites have to reconfigure their systems I don't see that as a bad thing. Why should I be treated any differently based on the previous site I have visited, or because I want to keep my browsing history private?

Re:Quick fix for HREFs viewed by MSIE

[jamie]

Dear God in heaven, anyone who does this is nuts.

Load a javascript off the website of someone else you don't trust? Cmon now... let's just invite a stranger into your home to watch all your websurfing, or post the contents of your cookie file to your LJ.

Anthony, I'm sure you're a nice guy and all, but would you trust a random stranger's javascript on every one of your webpages?

(The space added to the URL you pasted in is added to every long word at the 50-character mark, to make sure idiots can't break your browser rendering by typing very long words into their comments.)

Re:Will this be what kills the referer header?

[Anonymous Coward]

Well, I'd consider the referer headed "dead" if all browsers started just sending them a referrer based on their own domain. That'd get around most problems.

Re:Quick fix for HREFs viewed by MSIE

[inertia187]

Dear God in heaven, anyone who does this is nuts.

I have no problem releasing the script for anyone to use. The script comments say as much. But for the average user, it's easier to just dump HTML on a page. If you know how, by all means, copy the script on your own server.

Anyway, this should be interesting. Carry on.

Re:Solution:

[Mage Powers]

thats a little pie-in-the-sky, Are you sure the linkers are linking thier own stuff? bloggers don't make things, they link things.

Re:geocities does same thing

[Anonymous Coward]

No...they don't. Feel free to at least read the Slashdot summary before posting next time.

Preventing a slashdotting

[Kallahar]

I believe sites that have been slashdotted in the past have done this same thing to prevent their server from getting flooded. I think it's AOL's right to do this, they don't want livejournal linking to them. The polite thing to do would be to say why in the error page though, not just give a 404.

On a technical note, you can set up a page with a META Refresh which will clear the referrer (a HTTP server transfer will keep the original referrer intact though)

Re:Solution:

[carsont]

Don't use an ISP that is "broken". AOL has little to recommend it.

Well, for one thing, AOL has been "broken" in many ways for many years, and yet they still have an enormous and loyal user base. So does Windows, for that matter.

The problem is that for the average AOL user, who to put it bluntly is probably both too stupid to figure this out on their own and too lazy to read LiveJournal's explanation, it will appear just as likely that LiveJournal is "broken", not AOL. They will squeal "OMG WTF IT DONT WORK!!!!!1!!!11!!! :-(" and leap into the open arms of AOL's competing service. Whether or not AOL planned it this way is irrelevant.

This is how the fragile and complex interoperability between pieces of computer software, which is opaque to most users, can subvert the workings of the free marketplace; if company A sabotages their product so that it won't work with company B's product, it is easy for customers to be fooled into blaming company B.

Microsoft did this with their implementation of Java, and probably many other times. I doubt if this is some deliberate strategy on AOL's part, but the result will probably be the same regardless.

There may be method to this madness...

[Anonymous Coward]

If AOL is having a problem with people chewing up a huge amount of bandwidth using AOL as an imageserver for LiveServer, blocking all file types would be necessary unless AOL wants to screw with the content. The reason? IE doesn't care what you call an image - it can be "hotpr0n.html" and IE is "smart" enough to figure out it's a jpeg and display it. Plus, the pr0nmongers could always make AOL hosted iframes for their images, so even if AOL could spare the computing power to analyse every document it serves (to see if it's really an image) it wouldn't help. I've dealt with pr0nmongers before - they're very clever monkeys. And if it's wares or mp3s, that's even worse - you can fix the pr0nmongers with a simple apache mod to add a space to the beginning of every non-jpeg/gif/pdf/etc document so that mislabeled images will not display in a standard browser, but people hosting/collecting mp3s and wares will adapt. Anyone familiar with the "Iria" user-agent? If not, you don't know what AOL is dealing with...

Re:Quick fix for HREFs viewed by MSIE

[Anonymous Coward]

Uh, dumbass, the script is perfectly downloadable, savable, and viewable. HTTP is neat like that.

I'm reminded why i stopped playing the slashdot game after i won (karma cap)--too many ignorant idiots.

Re:tinyurl?

[lspd]

The trick is for the relay site to use a meta refresh instead of a HTTP redirect. It always blocks out the referrer.

Doesn't work for images, but who cares?

Re:hopefully

[Stephen Samuel]

It's clearly not bandwidth for another reason: If it was a bandwidth issue, they'd be blocking everything (or, at least, a lot more than just one blogging site) -- and as the previous reply pointed out, they've got lots of bandwidth available. I doubt that LiveJournal is that high a volume site.

On the other hand, this really stinks of anti-competetive practices in the light of the fact that they're just now moving into the blogging market. If LiveJournal can come up with a lawyer (other than Boies) willing to work on a contingincy basis, I expect that they could get a nice legal settlement.

Leave them feedback

[BrynM]

You can leave them feedback regarding this decision at the AOL Feedback Page [aol.com]. Let them know that this decision will impact more than they expect it will and that you are disappointed with their actions. You don't have to be a customer. Most of us are technology experts who have influence on their potential and existing customer base. I've included my submission as an example, but please write your own.

I find it disappointing that you are blocking content linked from such a popular site as LiveJournal based on referrer headers (see this slashdot discussion: http://slashdot.org/article.pl?sid=03/08/29/205724 2&mode=thread&tid=120&tid=187&tid=96&tid=9 9). I have recommended your services in the past, but that will stop now. Further, I will inform AOL customers whom I have recommended AOL to that their content is not available to sites that AOL seems to disapprove of for no stated reason.

Changing your policy regarding this may defer my judgement about your "service", but your reputation has been irrecoverably tarnished in my view. Additionally your subsidaries, such as Netscape, will no longer be recommended by me.

You may be thinking "Who cares? This is someone who isn't even our customer", but I have become the technology "guru" for over 100 people in my personal life and have input on technology decisions at my workplace. What should worry you is that for every peice of feedback you recieve like mine, there are hundreds of technologically literate people who will simply downplay or berate your services and will not do you the professional courtesy of informing you.

Formerly Respectfully Yours,
BrynM

Re:Wait a minute

[kaltkalt]

A good technical solution and an act of bad censorship are not mutually exclusive descriptions. Yeah, one has the word good and the other has the word bad, but those adjectives modify different things. In other words, AOL's actions can be both a good technical solution and an act of bad censorship. For example, the attacks on September 11th were a good technical solution to killing lots of people one considers to be "infidels" but also a very bad, evil, horrible act. Point: your dichotomy is false. If you're gonna bash the colletive slashdot mindset, do so in an intelligible way.

Re:Wait a minute

[Felinoid]

And then there are those who actually like Microsoft GASP and yes Microsoft has it's own Zellots GASP HORROR.. They just want us to believe they aren't bies.

But back to the point.

Using header information is like locking a door. It's not a bad idea in itself.

"I don't want people walking into my home so I'll sue anyone who lists my home address."
Sigh.. Dipstick LOCK YOUR DOOR.

But now what happends if someone starts locking OTHER PEOPLS doors.
If some jerk adds a lock to your house in effectively locking you out. If you object your against locks? No your against the application of locking someone out of there own home.

But this is AoLs site so let's bring this more to reality.

If your landlord locks you out of the appartment your renting.... He hasn't kicked you out he is still collecting rent he just won't let you inside.
Again not anti-landlord not anti-lock. In California you have to file a 30 day notice to evict a tenent so just locking them out is illegal.
(Unless the tenent is a nusence then you get a 3 day notice)
IANAL but was a landlord.

Now lets actually bring this entirely to reality.
Landlord dosen't lock the tenent out.

The AoL landlords are dictating to the tentent/users that they can't invite friends from Livejernal.

In the real world that would get the city on your butt and ACLU.

Then then someone says "Well make up your minds are you pro-locks or anti-locks?"

Re:Wait a minute

[ralphus]

Which is it -- good technical solution or bad censorship?

It is *both* a good technical solution and bad censorship.

A "technical solution" does not carry any moral or wider value judgements with it. It's essentally binary logic and a problem being addressed can be approached with the cold hard pragmatism of doing a math problem. It is objective.

The moral status of censorship is subjective. AOL's perfectly happy with the solution, or at least they are for the time being. Those being censored will usually be perfectly unhappy with it on the other hand.

You've related two concepts that aren't related at all, it's really apples and oranges.

Re:Here's An Idea..

[DanBrusca]

You miss one of the key aspects to Livejournal, the whole community aspect to it. LJ lets you incorporate your friend's journal entries into your own pages, find people with similar interests etc. That kind of thing just isn't available if you take the DIY approach.

Re:F12

[OhioJoe]

I've been saying the same thing as well, to friends and family just getting into the 'internet' scene. I explained that AOL is one room of a colossal mansion (the mansion being the Internet) that has a 'little taste' of most things on all the other rooms, but they don't advertise that it's a 'little taste' and instead portray it as the real thing.

I just was setting up some technical consultation with a towing company owner and asked him what ISP he was using. He answered "AOL", because my son gets into those chat things". I recall years ago after having used IRC, going into AOL's chat rooms and couldn't believe what I was seeing. Everyone had to speak in code constantly to avoid the censors. It was amazing to me that these people had no idea what IRC was, and had no concept of 'chatting' outside of AOL.

The single worst pox of naivete upon the Internet behind WebTV is AOL.